Remix.run Logo
▲ mcfig 3 hours ago

I can believe the “not completely fresh” for some minor version bumps. I’d imagine every minor version includes more than $0 on additional pretraining for the next release. (Though sometimes, maybe they found no gains and reverted to the previous base model). If nothing else to move the training cutoff date forward.

Whenever they change architecture or even just number of layers, I’d imagine they start from scratch but I could be wrong.

Also, I imagine there’s an incentive to start fresh whenever there’s an innovation in curation of the training dataset.