Remix.run Logo
▲ julianlam an hour ago

Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw

Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.