But that doesn't really matter and it shows how confused people really are about how a coding agent like Claude or OSS models are actually created -- the system can learn on its own without simply mimicking existing codebases even though scraped/licensed/commissioned code traces are part of the training cycle.
Training looks like:
- Pretraining (all data, non-code, etc, include everything including garbage)
- Specialized pre-training (high quality curated codebases, long context -- synthetic etc)
- Supervised Fine Tuning (SFT) -- these are things like curated prompt + patch pairs, curated Q/A (like stack overflow, people are often cynical that this is done unethically but all of the major players are in fact very risk adverse and will simply license and ensure they have legal rights),
- Then more SFT for tool use -- actual curated agentic and human traces that are verified to be correct or at least produce the correct output.
- Then synthetic generation / improvement loops -- where you generate a bunch of data and filter the generations that pass unit tests and other spec requirements, followed by RL using verifiable rewards + possibly preference data to shape the vibes
- Then additional steps for e.g. safety, etc
So synthetic data is not a problem and is actually what explains the success coding models are having and why people are so focused on them and why "we're running out of data" is just a misunderstanding of how things work. It's why you don't see the same amount of focus on other areas (e.g. creative writing, art etc) that don't have verifiable rewards.
The
Agent --> Synthetic data --> filtering --> new agent --> better synthetic data --> filtering --> even better agent
flywheel is what you're seeing today so we definitely don't have any reason to suspect there is some sort of limit to this because there is in principle infinite data
