Remix.run Logo
▲ COAGULOPATH a day ago

We have scaled on both axes and many others, I'd say.

Inference can be increased vertically (a single model thinking for longer), or horizontally (NS was solved by a 10,000-agent swarm).

And pretraining has two inputs that can be scaled independently - training data and model size. (Things like the Chinchilla 20:1 law are meant to establish the optimal way to train with a limited budget - you could just keep making either term bigger and bigger on its own if you had infinite money.)

I think creating and curating high-quality datasets is also part of scaling. Datasets like FineWeb are quite a bit better than the barely-filtered CommonCrawl dumps GPT-3 trained on, but someone had to invest in making them. (To say nothing of whatever exotic bespoke robotics/physics/etc datasets OpenAI and Anthropic are investing in.)