General purpose models are always more robust and generally better than smaller narrower models. My bet is that compute will catch up and any “small” model will still be generally capable, just smaller than sota, rather than intentionally narrow. The exception would be for very well defined tasks where the data distribution never varies, but these are rare and don’t really need “AI” anyway when they do exist.

▲

swiftcoder a day ago | parent | next [-]

> General purpose models are always more robust and generally better than smaller narrower models

I feel like this is just the marketing conflation of AI=LLM, versus regular old ML? We're never going to need to deploy a full reasoning model on a low-power device just to do some fancy image recognition in the field. Specialised ML models are just intrinsically able to be a lot more efficient than their generalist equivalents

▲

plastic-enjoyer a day ago | parent | prev | next [-]

> General purpose models are always more robust and generally better than smaller narrower models.

What do you mean with more robust?

▲

ACCount37 a day ago | parent | next [-]

Less weird unexpected failures, more innate ability to handle edge cases gracefully. Quite important when you're running high on automation and low on oversight.

▲

plastic-enjoyer 3 hours ago | parent [-]

This may be speculative, but couldn't robustness emerge by having a number of specialized models, that are interconnected and feed into each other? Are there any arguments from ML that would speak against this?

	▲	ACCount37 2 hours ago \| parent [-]
		That does work. Even if you drop the "specialized" part. Ensembles of the same architecture at the same scale trained on the same data do outperform a singular model of the same line - especially on corner cases. Successes of an ensemble correlate stronger than failures do. The usual argument against is that if you have "a number of specialized models" that perform well in ensemble, you can take that ensemble, and distill it into a single larger model (dense or integrated sparse, like MoE), and get the same improvement in performance with an efficiency win. This works because having those "specialized models" duplicates a lot of the highly conserved "low level" wiring that's required for a model to function at all. As such, you end up running a small scale version of the same "backbone" computational processes many times. "Merging" those models into a larger, denser model allows for a singular strong "backbone" to be used for everything.

▲

a day ago | parent | prev [-]

[deleted]

▲

ACCount37 a day ago | parent | prev [-]

You're getting downvoted, but you're completely right. There are very few cases in which narrowing a model down is buying you anything worthwhile.

It seems like for LLMs, "general intelligence" is expensive, but "one more domain" is fairly cheap.

▲

Danox 18 hours ago | parent [-]

The demand for smaller models and single purpose AI agents will only get bigger in the future. It’s no different than going for mainframe computers back in the day to personal computers. It’s inevitable…..

	▲	andy99 16 hours ago \| parent [-]
		Has nothing to do with demand, it’s a question of the most efficient way to train a good model. Narrower models are in general worse than general purpose ones, so you’re better using the largest general purpose model that fits in your compute budget vs trying to somehow remove capability or knowledge and assume the model stays as capable for the task you want.