This AGI silliness is predicated on a flawed "Platonic" view of epistemology. The notion that there is some well-defined "idea space" and therefore superintelligence can explore that space faster than any human. In fact, there is no such space. There is a "token space" that can be explored, but that has only fleeting overlaps with reality.

It also completely ignores what we know about evolution. Our brains are the result of billions of years of natural selection. The amount of “training data” that resulted in our neural structures is on a completely different scale than the training data used for today’s LLMs. And this isn’t even up for debate.

The assumption that this process can be “distilled” from written word is completely insane. I’m not sure how people trick themselves into thinking it’s even remotely possible.