Genuinely curious: what are people using these smaller local models for? They are getting decently capable, but they are still small enough that I don't trust them for "real" work outside of a handful of fun toy projects.

Are people actually using them in coding agents? Or are they mostly using them for other things?

I use a gemma4 model locally to extract content from messages to a personal agent I'm building for its memory graph (to break the message up into the topic, source (assistant or owner), facts, entities, etc. in the message content (all getting thrown into a magma-esque graph using NLEmbeddings for memory search). This is for a custom personal agent that targets deepseek-v4 flash. The local model is too slow in my setup for a chat agent, but for memory extraction it works pretty well, saving API usage on every chat turn.

We've shipped some code generated by Qwen3.6 27B to production (under OpenCode). It lacks the breadth of knowledge of models like Opus, but if a change is fully inferable from the prompt and the surrounding code, it works very well. It won't be able to write something from scratch that requires niche knowledge (say, a performant inference engine tailored to Blackwell GPUs), but if it's just a PR adding a new use case to an existing project (which is usually just "load from the DB, do some invariant checks, modify the entities, store them back"), it works as well as Sonnet (provided you have the correct configuration, like recommended temperature and top-p settings, the model isn't over-quantized, you have at least 150k tokens of context available, etc.).

I don't use them as coding agents, but they can be very useful for things like text transformation, summarizing, or text extraction.

That said, if you have a subscription to a paid model already, you're not necessarily winning out on anything except perhaps privacy, which isn't nothing.

there is plenty of grunt work these smaller models can do. update dependencies, fix merge conflicts, write --help, markdown, or readme files for existing code. etc.

sometimes they fail but undo is just a "git restore" or if automated, rejecting a PR and having a better model take a crack at it.

Qwen35ba3b can do a huge amount of data cleaning work on pretty modest hardware. Already have run about 100 billion tokens on it using 2x3090 gpus.

They're great at helping me look up web dev stuff when I don't have internet access.

[flagged]