Optimizing speed is really the way to go. Yet 24GB is not what everyone can afford. Maybe we could take some of those 56tk/s and transfer into some free RAM space using MoE loading ? I'd be glad with a less than 10GB and more than 6tk/s model.

▲

lisplist 3 hours ago | parent | next [-]

Unfortunately this is just the entry price for LLMs. With the exception of the Qwen 27B models, I personally haven’t found a ton of use cases for models less than 200B. With the right setup, fine tuning, etc, you can make small models do cool things, but hard to please everyone given the insane hardware costs at the moment and the comparably cheap API costs.

▲

dannyw 40 minutes ago | parent | next [-]

Small models are still great for lots of “simple intelligence” use cases, like annotating or summarising files and media; or even just basic chat when given web search tools.

My local NAS is private and I’m not going to send it off to APIs for captioning or metadata; but even Qwen3VL 8B does an excellent job at this, despite being quite old.

They are also really excellent for fine tuning. Unsloth and Tinker (from Mira’s TML) are great places to start.

If your use case is narrower than “coding agent for everything”, you can probably match frontier performances on that narrow domain with ~30b and exceed it with ~100b+.

▲

dist-epoch 2 hours ago | parent | prev [-]

Gemma4-E4B (4B params) works pretty well as a local wiki, or when you don't have connectivity.

	▲	dannyw 42 minutes ago \| parent [-]
		Nitpick: Gemma4-E4B is actually a 8 billion param model, but only 4.5B params worth of memory bandwidth needed per decode.

▲

Manfrednotfunny 3 hours ago | parent | prev [-]

I don't thinnk just MoE will solve it. If you hit constantly different expert layers, you can't outsource layers efficently and have to swap it in.

MoE will be faster because it will read less memory for sure, you still have to have it though.