Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.

Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.

[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855

▲

Youden 4 hours ago | parent | next [-]

I think your leading statement is a bit more authoritative than what the linked paper actually demonstrates. It demonstrates your claimed qualities for specific algorithms, methodologies, and models.

It explicitly says that Quantization-Aware Training is out of scope, may reveal different trends and has potential for extreme quantization. Heavily-quantized model releases today generally use QAT or something similar, so I don't think the paper can be universally applied to modern quants (e.g. [0]).

That said, I don't see any evidence that TFA does any of that so the paper's claims may be quite applicable there.

[0]: https://huggingface.co/vcruz305/DSV4.1-Flash-1017-SAGE-REXL3...

▲

fho 2 hours ago | parent | next [-]

I recently did some toying around with (very) small language models (<200k). I used QAT from the beginning and was surprised that I got byte identical results from the int8 model vs the float32 one.

If that scales with parameters I don't see why QAT shouldn't be the norm.

(As I was only comparing the QAT f32 model vs the quantization its possible that a not QAT f32 would have performed even better)

	▲	lostmsu 2 hours ago \| parent [-]
		Do you think QAT isn't the norm yet? I expected it to be the norm since GPT-OSS

▲

bitexploder 2 hours ago | parent | prev [-]

And things like GSQ-RCO optimization from ISTA labs, EXL that you linked. Preserving bits for certain (large) domains like agentic coding. Pruning others. It really works and creates functional models.

▲

augment_me 36 minutes ago | parent | prev | next [-]

This is not fully true. 4 over 6 paper(https://arxiv.org/pdf/2512.02010) for example showed that the initial and last layers of transformers that expand into the high-dimensional vocabulary are hurt much more than the center ones. These layers are kept in fp16 and are hurt much more than others by quantization.

If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.

With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.

▲

moffkalast 9 minutes ago | parent [-]

Afaik this is already a part of regular quants, Q4 just means the average reduction in accuracy is at that level for the whole model, some weights will be fp16, some int8, some int4, some even lower. 4KM is slightly larger than 4_0 for example.

	▲	augment_me 4 minutes ago \| parent [-]
		Really? I didn't know, where does it spec this usually? I have never seen it neither on the link above nor on HF.

▲

pu_pe an hour ago | parent | prev | next [-]

There is always quality degradation when quantizing, but some models keep surprisingly good performance at aggressive levels of quantization too. DeepSeek v4.1 flash at 2-bit is still a beast for example. In my own experience I would easily trade off accuracy/intelligence for a vast speedup, especially if your setup simply wouldn't run something at 4-bits.

▲

onlyrealcuzzo 5 hours ago | parent | prev [-]

I imagine that it's just the non-linearity of audio makes it easily apparent how much quantization really matters, and that benchmaxxing is so ingrained in LLMs that if you simply run them through benchmarks, they can do well under quantization, when in reality, they have a similar drop off that they do in audio, it just doesn't slam you in the face at every turn as obviously.