| ▲ | COAGULOPATH a day ago | |
We have scaled on both axes and many others, I'd say. Inference can be increased vertically (a single model thinking for longer), or horizontally (NS was solved by a 10,000-agent swarm). And pretraining has two inputs that can be scaled independently - training data and model size. (Things like the Chinchilla 20:1 law are meant to establish the optimal way to train with a limited budget - you could just keep making either term bigger and bigger on its own if you had infinite money.) I think creating and curating high-quality datasets is also part of scaling. Datasets like FineWeb are quite a bit better than the barely-filtered CommonCrawl dumps GPT-3 trained on, but someone had to invest in making them. (To say nothing of whatever exotic bespoke robotics/physics/etc datasets OpenAI and Anthropic are investing in.) | ||