"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a total duration of 10M hours."

I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themselves.

▲

ks2048 a day ago | parent | next [-]

I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html

They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.

▲

kennyadam a day ago | parent | next [-]

Ha, is it just me or does a “residential proxy network” sound like a fancy way of saying “botnet”?

▲

stymaar a day ago | parent | next [-]

It's the "legal version" of botnets, yes.

▲

MallocVoidstar a day ago | parent | prev | next [-]

Basically all of them are botnets, yes. They generally claim to have consent but I'm pretty sure 99% of it is "some app the user uses has it buried in a 200 page ToS"-style consent.

▲

m00dy a day ago | parent [-]

No, they are not. Just ask any AI.

	▲	MallocVoidstar 19 hours ago \| parent [-]
		The companies say they have consent because saying "yeah we'll give you access to a botnet" is illegal. https://cloud.google.com/blog/topics/threat-intelligence/dis... >These SDKs, which are offered to developers across multiple mobile and desktop platforms, surreptitiously enroll user devices into the IPIDEA network. If I install an app with a big ToS and buried in there is "we use third-party monetization SDKs" and buried in there is "welcome to our botnet" they do not actually have my consent.

▲

m00dy a day ago | parent | prev [-]

Nope, they’re different things. Proxy networks like Proxybase [0] use open-source clients and ask for the user’s consent before allowing them to join the network.

[0] https://proxybase.xyz

▲

kevmo314 a day ago | parent | prev | next [-]

That’s a pretty penny on the bandwidth bill alone!

▲

maybewhenthesun a day ago | parent | prev [-]

> youtube seems to now more aggressively block yt-dlp.

Another thing to thank AI for! /s

▲

ks2048 a day ago | parent | prev [-]

I was wondering if youtube blocked them a lot and they got a lot of videos from other sources - but no, you can see on this HF page that 93.1% of URls are youtube.com, https://huggingface.co/datasets/laion/BVD-URLs