| ▲ | mkarrmann 3 hours ago | |
I'm curious what exactly each "base-model" here is. We've generally believed that most model upgrades aren't new pre trainings, but instead are better post trainings of a common pre trained model. But this graph suggests every release had a better base model. Have the public narratives around how these models are trained been misleading? Does each model release usually always include an extended, if not completely fresh, model pretrain? | ||
| ▲ | SsgMshdPotatoes 2 hours ago | parent | next [-] | |
The figure indicates that some dots (base models) correspond to two production models (GPT-5 / 5.1). So I guess that tells you at least which models are built on the same pre-train checkpoint, but not whether some pre-trains are, say, very minimal extensions of others. | ||
| ▲ | mcfig 3 hours ago | parent | prev [-] | |
I can believe the “not completely fresh” for some minor version bumps. I’d imagine every minor version includes more than $0 on additional pretraining for the next release. (Though sometimes, maybe they found no gains and reverted to the previous base model). If nothing else to move the training cutoff date forward. Whenever they change architecture or even just number of layers, I’d imagine they start from scratch but I could be wrong. Also, I imagine there’s an incentive to start fresh whenever there’s an innovation in curation of the training dataset. | ||