> emerging practice of using Q4 quants and Q8 KV cache for local inference

That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.

▲

walrus01 5 hours ago | parent [-]

I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.

▲

trollbridge 2 hours ago | parent [-]

Gemma's QAT is surprisingly good (although Gemma isn't that great to begin with).

	▲	KerrAvon an hour ago \| parent [-]
		IME: Gemma is not great for programming, but it is fantastic at following directions compared to anything else in its size class.