If they really want to be spiteful they can rewrite all the news articles they post to Meta using Muse. Then we can finally see if the theories of statistical model collapse hold up at the trillion param level model scale.

I'd really be curious to see how much of Muse and similar 2026 models are actually from scraping, especially from this hypothetical unsorted random sites.

My understanding is that most of the gains after the early GPT-3 "add more weights" era have come from post-training from synthetic data, especially through reinforcement learning.

I couldn't quickly find any public statements about the actual breakdown of training data, other than almost every lab saying that post-training size was "scaling" or growing "by an order of magnitude" from their previous major release.