> The vast, vast, VAST majority of the original datasets were from pirated books and the like

And there's been significant legal consequences as a result

> Also, arguably a robots.txt is the exact mechanism to follow to do the mass GET-ing

You're free to argue this of course, but the courts have largely rejected it already pre LLMs. See for example hiQ Labs v. LinkedIn

▲

fooblaster 2 hours ago | parent [-]

Yes, anthropic and openai have really been brought to their knees and ipos cancelled because of the legal consequences of obtaining their training data.

	▲	gpm 2 hours ago \| parent [-]
		This would have been a problem but it turns out that Anthropic is actually valued multiple orders of magnitude more than a copy of all the books in the world. So they survived the significant legal consequences.