Fair, we gotta do a better job at explaining this properly!

So an 8-27B on a LAN box wins for generic tasks on hardware that can hold it. Needle is for hardware that can't, like plain ARMv7, MIPS32 (the Ingenic chips in cheap IP cameras), RISC-V and watches. Also, we found cost to not really be the lever for on-device models, but availability and latency.

▲

ash_091 6 hours ago | parent | next [-]

So it will run on those devices, but when/why would you do that?

All of the demo setups (smart home, robot vacuum, watch, etc) could easily have access to a bigger model running on a more capable device either locally or via the internet.

▲

yorwba 8 hours ago | parent | prev [-]

A list of hardware platforms doesn't make a use case. Do you have an active deployment of Needle that is noticeably useful, and if so, what do you have it do?

▲

stymaar 6 hours ago | parent [-]

At my current company we're evaluating small models embedded directly in the web app to provide a natural language interface to the app without spending money on inference (and ideally avoid a ChatChipotle situation where people end up having free token going through our interface).

And needle is one of the most promising model due to its original architecture (but we still need to finish building the actual eval dataset before making out final call).

	▲	HenryNdubuaku 5 hours ago \| parent [-]
		Thanks for considering needle. Keep in mind that you can also fine-tune the model to fit your use case more. I think this illustrates the intended deployment pretty well, where both computational resources and compute credits can both be issues for deployment.