I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.

I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

▲

andai an hour ago | parent | next [-]

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?

Edit: the article says millions of times per hour? (!?)

The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?

▲

nobodywillobsrv an hour ago | parent | prev | next [-]

Feels like it would be a good time for freenet and the like to catch on.

▲

jay_kyburz 3 hours ago | parent | prev [-]

I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.

You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.

▲

jay_kyburz 3 hours ago | parent [-]

How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.

▲

mips_avatar 2 hours ago | parent [-]

that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.

▲

jay_kyburz 2 hours ago | parent [-]

I was thinking you scrape your own website every day in the middle of the night when traffic is low, and make that available. They can come and collect it every day if they want to.

	▲	mips_avatar an hour ago \| parent [-]
		Yeah. Though I guess the point I thought of was like a deals site. That would have infinite pages and content