Remix.run Logo
▲ augment_me an hour ago

Really? I didn't know, where does it spec this usually?

I have never seen it neither on the link above nor on HF.

▲julianlam 29 minutes ago | parent | next [-]

Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw

Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.

▲kanbankaren 28 minutes ago | parent | prev [-]

What do you mean never seen?

On HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.