Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models.

The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.

And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.

On narrow domains, it is very common for small models to match or outperform larger ones at a fraction of the parameter count.

For example in language, this is called the “curse of multilinguality”. Small models that handle a single translation direction can easily outperform big ones that try to handle them all.

https://arxiv.org/pdf/2311.09205

In any case, for most tasks the question is not “how many tasks can this model kind of do well” but “given time/cost constraints, what is the maximum level of quality we can achieve”. And for that, small models are usually very competitive.

I think the context here is that small models run locally, not rented from a cloud.