They're not trained on a raw feed of the internet. They are given curated and synthetic data. The curation and synthesis of new data is done by existing LLMs.

Even if you're given the perfect textbook to read, it still helps you to take notes. Notes serve multiple purposes -- they help add clarity where it is needed, and more importantly, they help integrate new info (the current batch) with prior info (previous batches).