Remix.run Logo
▲ Tepix an hour ago

Q2 quantization. Not interested.

▲tcdent 43 minutes ago | parent [-]

All of these projects targeting low spec systems and "100 tok/s" are the same 2 bit quant without much else. Conveniently none of them include any mention of accuracy in their published numbers. 4 bit is the floor.