|
| ▲ | jmb99 5 hours ago | parent | next [-] |
| Scraping text across the entire internet is orders of magnitudes easier than scraping YouTube. Even ignoring the sheer volume of data (exabytes), you simply will get blocked at an IP and account level before you make a reasonable dent. Even if you controlled the entire IPv4 space I’m not sure you could scrape all of YouTube without getting every single address banned. IPv6 makes address bans harder, true, but then you’re still left with the problem of actually transferring and then storing that much data. |
| |
| ▲ | earthnail 3 hours ago | parent | next [-] | | For now, you actually get pretty far with Tor. Just reset your connection when you hit an IP ban by sending SIGHUP to the Tor daemon. I did that when I was retraining Stable Audio for fun and it really turned out to be trivial enough to pull of as a little evening side project. | |
| ▲ | tucnak an hour ago | parent | prev [-] | | IPv6 doesn't make it "harder," as they would typically ban whole /48 prefixes. |
|
|
| ▲ | monocasa 9 hours ago | parent | prev | next [-] |
| And we're probably already starting to see that, given the semirecent escalations in game of cat and also cat of youtube and the likes of youtube-dl. Reminds me of Reddit's cracking down on API access after realizing that their data was useful. But I'd expect both youtube to be quicker on the gun knowing about AI data collection, and have more time because of the orders of magnitude greater bandwidth required to scrape video. |
| |
|
| ▲ | awesome_dude 8 hours ago | parent | prev [-] |
| > Seems like all model providers managed to scrape the entire textual internet just fine Google, though, has been doing it for literal decades. That could mean that they have something nobody else (except archive.org) has - a history on how the internet/knowledge has evolved. |
