Everything lower than 4/5 is really bad

latentsea 3 hours ago | parent | next [-]

This mentality is really pervasive and yet I've found when a lower than 4-bit quant of a given model was all I could use, then using that was often better than using a 4-bit quant of a model that I could. YMMV. It just really made me rethink my stance on the whole thing. Most people likely see something like 2-bit and just feel like the want to puke without ever actually learning whether or not it can handle their tasks and how well relative to other models they run. I've been running Qwen3.8-Flash-Next at Q4, but was crazy surprised at the performance of Unsloth's IQ1_M of DeepSeek v4.1 Flash.

	▲	pyrolistical an hour ago \| parent [-]
		You don’t know what you don’t know

▲

rpdillon 2 hours ago | parent | prev | next [-]

Deepseek v4 flash 0731 in a 2-bit quant has been useful for me. Other models don't seem to hold up nearly as well at 2-bit. Most of my other models are 5-6 bit quants because of this.

▲

smallerize 5 hours ago | parent | prev | next [-]

Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?

▲

serf 5 hours ago | parent | next [-]

probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.

haven't tried it yet though.

	▲	cyanydeez 3 hours ago \| parent [-]
		Qwen3.8-Flash-Next fits in 80GB on strix halo. That's when I started real "vibe" coding with them. Speed is still an issue if I want to move fast and break things, but I've built a little universe for development (I assume many people have also) and it's cruising. But yeah, 32-64 is the gateway I think. If Qwen4 can make a leap in the flast-next MoE and hit 64 GB on VRAM w/context, that's definitely the cloud killer to me.

▲

Tepix 5 hours ago | parent | prev | next [-]

I’m wondering myself: Do bigger models suffer less from quantization? Are there papers?

▲

ricardobeat 4 hours ago | parent [-]

They seem to degrade a lot less.

You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.

	▲	alfiedotwtf 4 hours ago \| parent [-]
		Always check the label and if pain persists, see your doctor! The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.

▲

pitchlatte 4 hours ago | parent | prev [-]

you can get a lot done with models well under that - the google and qwen 20-30B class models are fantastic

▲

LeBit 4 hours ago | parent | prev | next [-]

This but also the kv quantization shouldn’t be lower than 8 bits.

▲

mt_ 4 hours ago | parent | prev | next [-]

Next frontier is achieving good output with that kind of quant dimension

▲

ranger_danger 3 hours ago | parent | prev [-]

There are solutions for that: https://prismml.com/news/bonsai-2-27b

(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)