That is not true - a model trained in the internet can both build verifiers to remove false/poor quality data from the next training, and build synthetic datasets that will supplement its training.

Similar to a human that wants to learn something and invents exercises to practice.

1-2 years ago it was a theory, but new models are trained, successfully, on synthetic datasets.

This feels like a bit of a semantic debate, but maybe a few useful perspectives, as interpret current bespoke work as not so rosy wrt collapse:

- Synthetic datasets are typically human-steered today, which points to model collapse wrt learning from the internet. (Edit: or even simpler, model x data tapped out for cost/benefit even before collapse.) I don't think standard practice is (yet) AI looking at the internet and deciding to build its own gyms to go further. When it does, model collapse may happen again, and be even more expensive

- Distillation attacks are getting interesting here. There seem to be 2 kinds: intentionally querying other models, and maybe not so intentionally, learning from reasoning traces going through shared routers, esp. coding ones

- A lot of neolabs are trying to go where the big labs might not look as directly to avoid being squashed, which suggests they aren't ready to bet on being smarter, and that means the general AI is more about collapse / $