Sure, a given LLM can only produce so much (though it might be a lot), and then you'll need fresh inputs. But you contend that those have to be human, while I am arguing that they could be

- random,

- from the environment,

- from other AIs.

That increases the space considerably.

In particular, note that your argument also applies to humans: A human, being finite, can only produce finitely many things. But you add other humans, the environment, and pure chance to the mix, and it'll be pretty infinite for all practical purposes.

model collapse seems to say that "AI" cannot train on itself.

I would be curious to see the end result of letting it endlessly update its inputs from its own outputs. I was under the impression that eventually this would cause delusions and if you let it do this long enough and it is solving problems that no humans can verify as correct that it isn't useful.

Maybe that is the limits of my imagination, but it seems unwise to take humans completely out of the loop. Having mathematicians around that can understand what it is doing so that we can utilize that knowledge in a way that we can have some confidence in seems important.