| ▲ | julianlam an hour ago | |
Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers. | ||