It's because they want to restrict AI companies from stealing content, but they can't do it if internet archive proxies it all for them.

All of the LLMs would be massively less useful if it wasn't for scraping the latest news.

▲

stephen_g 2 hours ago | parent | next [-]

LLMs have other ways of accessing the content, they don’t need the Web Archive.

Every LLM company can afford to spin up a new subscriber account every day, proxying to appear different IPs from all sorts of ASNs, do some crawling until the account gets banned, and then do it again, and again, and again.

	▲	overfeed 2 hours ago \| parent \| next [-]
		> LLMs have other ways of accessing the content, they don’t need the Web Archive. What's the conclusion from this train if thought? Just because some burglars can pick locks doesn't mean you should leave your front door unlocked. Locking a door (or robots.txt) is how one can establish mens rea for those who bypass the barrier.
	▲	Gigachad an hour ago \| parent \| prev [-]
		The legal implications would be different vs scraping publicly available content.

▲

userbinator 31 minutes ago | parent | prev [-]

"stealing" is BS because the original still exists. Copyright infringement is more correct.

	▲	Gigachad 30 minutes ago \| parent [-]
		[dead]