I'm not an expert in LLM training, but I think we can all agree that the writing on the Internet is generally very low-quality and surface level compared to the depth of books. Most books in the past were even edited by a separate person from the writer!

Humans manage to be pretty intelligent with only reading perhaps a few thousand books in their lifetime. It seems unlikely that AGI will appear but only once it has read that last out of print 1983 book on knitting patterns.

Granted that that is indeed the case, surely the quantity of high-quality writing available online (including things like Project Gutenberg) still dwarfs the quantity remaining in obscure unscanned books.