It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model.

There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.

▲

LPisGood 6 hours ago | parent | next [-]

Small amounts of world knowledge seems like it would inherently be tied to more hallucinations.

▲

TJTorola 6 hours ago | parent | next [-]

Perhaps we'll get to a point where believing any un-sourced information from an LLM will feel crazy. I don't want my model to know more than it needs to perform logic and use tools. Once it is capable of using tools I would much rather it looked up information or sourced it from existing context rather than just divine it from it's weights.

	▲	pinkmuffinere 14 minutes ago \| parent [-]
		I empathize, and I have the same preference, but I wonder how this interacts with other people (many of them being our coworkers) using LLMs. There is no authoritative source for the models to pull info from, so either people will have to exercise good judgement and double check important claims, or they will trust too blindly and fall close to the level of whatever LLM they use. In that case, I prefer my coworkers to use an LLM that does have world-knowledge -- I will still hear them spout ridiculous claims, but at least it should be less frequent. It strikes me there's a sort of prisoners dilemma here, where if nobody trusts others to critically evaluate info, it's in our interest to make the tooling do it instead, to whatever degree that is possible. Maybe I'm too cynical about working with others though.

▲

jbstack an hour ago | parent | prev | next [-]

Doesn't matter if you aren't asking the type of questions where hallucinations are relevant e.g. you're seeking pure reasoning rather than factual information.

▲

DennisP 5 hours ago | parent | prev | next [-]

Only if we require the knowledge to be built into the weights. Give it access to a search engine and a big library of ebooks, and it might do better.

▲

Zambyte 6 hours ago | parent | prev [-]

Probably. You can solve it with either some grounding context, or spending hundreds or thousands a month extra on a model that has more knowledge baked in. With modern harnesses, the choices is obvious.

▲

giraffe_lady 6 hours ago | parent | prev | next [-]

Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models.

The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.

And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.

▲

janalsncm 3 hours ago | parent | next [-]

On narrow domains, it is very common for small models to match or outperform larger ones at a fraction of the parameter count.

For example in language, this is called the “curse of multilinguality”. Small models that handle a single translation direction can easily outperform big ones that try to handle them all.

https://arxiv.org/pdf/2311.09205

In any case, for most tasks the question is not “how many tasks can this model kind of do well” but “given time/cost constraints, what is the maximum level of quality we can achieve”. And for that, small models are usually very competitive.

▲

wredcoll 5 hours ago | parent | prev [-]

I think the context here is that small models run locally, not rented from a cloud.

	▲	giraffe_lady 5 hours ago \| parent [-]
		Yes small models are and will be useful for lots of stuff for several reasons. But the idea they’d be better than a bigger model is cope, you’re pretty much always better off running the biggest one you can bring to bear within your constraints.

▲

cpill 3 hours ago | parent | prev | next [-]

yeah, I think they will get smaller so they can be run everywhere, and really just be an interface to various non AI systems.

▲

cyanydeez 5 hours ago | parent | prev [-]

[flagged]