Very large models trained with sgd variants don't tend to over fit, especially when using few epochs. See double descent.

The coding/math breakthroughs were caused by training models on the subset of their own outputs which pass a verifier, see reinforcement learning from verified rewards.

More data or parameters can break double descent; it's not a magic free ride, e.g.,

https://arxiv.org/pdf/1912.02292

There's plenty more. Read Google scholar