> I love the idea of an on-device model

My impression is that most people here haven't tried similar small models and don't have first hand experience with them. They are, to be honest, terrible. They may be good for certain tasks, but are much weaker than something like GPT4. I don't feel excited about these small models that are not fast yet hallucinate all the time.

▲ theshrike79 5 days ago | parent [-]

Weaker by what metric? Are you asking them to explain the fall of Rome to you?

The point of a small model isn't to be an interactive Wikipedia. It's there to call tools, get more data, aggregate the data and return a natural language result.

It does not "hallucinate", because it only uses what the tools provide.

▲ rs186 4 days ago | parent [-]

If you just look ar similar discussions on HN, you'll see that these models often don't even answer the specific question you ask but just give you random nonsense. I'd rather see an honest "I don't know" instead of complete gibberish.

I'm not going to use any small model that has a chance of dialing 911 when asked to send a text message to a friend.

	▲	theshrike79 4 days ago \| parent [-]
		This is what tool calling is for. It's just the model being trained to produce a specific type of JSON when it needs an external call After that it's plain code. `if phone_number == "911" return false` Done. Won't be calling the cops.