Typical thought experiment - go back to the early years of humanity, even before we've created language itself. Communication is basic grunts and other such things, cutting edge knowledge is 'poke him with the pointy end'. Compile all of this knowledge into the training set for an LLM, let it run with infinite processing power and infinite time. It'll just continually remix the knowledge of that snapshot of time. You're certainly not getting to the Moon, discovering the secrets of the atom, or even discovering mathematics.

It sounds increasingly dumb to bring this up given how advanced models have become, but it remains true - they are still fundamentally token prediction algorithms. And that imposes hard constraints on their potential. These constraints are relatively academic in our current world because there's just so absurdly much information and no human can hold more than a negligible amount in his mind at once. So they'll certainly be able to connect many 'obvious' dots that we don't see simply for a lack of carrying capacity, but they're not going to discover concepts that do not clearly follow from the comprehensive knowledge of an era.