This is not fully true. 4 over 6 paper(https://arxiv.org/pdf/2512.02010) for example showed that the initial and last layers of transformers that expand into the high-dimensional vocabulary are hurt much more than the center ones. These layers are kept in fp16 and are hurt much more than others by quantization.

If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.

With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.

▲

moffkalast an hour ago | parent [-]

Afaik this is already a part of regular quants, Q4 just means the average reduction in accuracy is at that level for the whole model, some weights will be fp16, some int8, some int4, some even lower. 4KM is slightly larger than 4_0 for example.

▲

augment_me an hour ago | parent [-]

Really? I didn't know, where does it spec this usually?

I have never seen it neither on the link above nor on HF.

	▲	julianlam 30 minutes ago \| parent \| next [-]
		Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.
	▲	kanbankaren 28 minutes ago \| parent \| prev [-]
		What do you mean never seen? On HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.