There's an emerging practice of using Q4 quants and Q8 KV cache for local inference. At that point you can run both Qwen3.5-122B-A10B (my personal choice on Framework Desktop 128gb) and Laguna-S-2.1.

Now whether that's good enough for one's use-case remains to be determined. You can also get more out of those (local models and quantizations) if you further tweak the harness you use them with, but tbh this is where it gets too much work (at least for me and the time I have available).

▲

zargon 5 hours ago | parent | next [-]

> emerging practice of using Q4 quants and Q8 KV cache for local inference

That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.

▲

walrus01 5 hours ago | parent [-]

I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.

▲

trollbridge 2 hours ago | parent [-]

Gemma's QAT is surprisingly good (although Gemma isn't that great to begin with).

	▲	KerrAvon 24 minutes ago \| parent [-]
		IME: Gemma is not great for programming, but it is fantastic at following directions compared to anything else in its size class.

▲

apitman 19 minutes ago | parent | prev [-]

Artificial Analysis ranks qwen3.6-27b higher than qwen3.5-122b-a10b on both intelligence and coding. Does that run counter to your experience?