new | show | ask | jobs Github

akie 5 hours ago

At which quantization? That's the only thing I want to know.

EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.

▲

madduci 5 hours ago | parent | next [-]

Everything lower than 4/5 is really bad

▲

latentsea 3 hours ago | parent | next [-]

This mentality is really pervasive and yet I've found when a lower than 4-bit quant of a given model was all I could use, then using that was often better than using a 4-bit quant of a model that I could. YMMV. It just really made me rethink my stance on the whole thing. Most people likely see something like 2-bit and just feel like the want to puke without ever actually learning whether or not it can handle their tasks and how well relative to other models they run. I've been running Qwen3.8-Flash-Next at Q4, but was crazy surprised at the performance of Unsloth's IQ1_M of DeepSeek v4.1 Flash.

	▲	pyrolistical an hour ago \| parent [-]
		You don’t know what you don’t know

▲

rpdillon 2 hours ago | parent | prev | next [-]

Deepseek v4 flash 0731 in a 2-bit quant has been useful for me. Other models don't seem to hold up nearly as well at 2-bit. Most of my other models are 5-6 bit quants because of this.

▲

smallerize 5 hours ago | parent | prev | next [-]

Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?

▲

serf 5 hours ago | parent | next [-]

probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.

haven't tried it yet though.

	▲	cyanydeez 3 hours ago \| parent [-]
		Qwen3.8-Flash-Next fits in 80GB on strix halo. That's when I started real "vibe" coding with them. Speed is still an issue if I want to move fast and break things, but I've built a little universe for development (I assume many people have also) and it's cruising. But yeah, 32-64 is the gateway I think. If Qwen4 can make a leap in the flast-next MoE and hit 64 GB on VRAM w/context, that's definitely the cloud killer to me.

▲

Tepix 5 hours ago | parent | prev | next [-]

I’m wondering myself: Do bigger models suffer less from quantization? Are there papers?

▲

ricardobeat 4 hours ago | parent [-]

They seem to degrade a lot less.

You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.

	▲	alfiedotwtf 4 hours ago \| parent [-]
		Always check the label and if pain persists, see your doctor! The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.

▲

pitchlatte 4 hours ago | parent | prev [-]

you can get a lot done with models well under that - the google and qwen 20-30B class models are fantastic

▲

LeBit 4 hours ago | parent | prev | next [-]

This but also the kv quantization shouldn’t be lower than 8 bits.

▲

mt_ 4 hours ago | parent | prev | next [-]

Next frontier is achieving good output with that kind of quant dimension

▲

ranger_danger 3 hours ago | parent | prev [-]

There are solutions for that: https://prismml.com/news/bonsai-2-27b

(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)

▲

adrian17 4 hours ago | parent | prev | next [-]

I’m confused too and the slop readme doesn’t help at all.

> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB

And later

> Standard IQ2_XXS (76.87 GiB, Unpatched)

Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.

Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.

	▲	adrian17 13 minutes ago \| parent [-]
		PS: only now noticed that 76.87GiB _is_ 82.5GB, which happens to be size of the unsloth IQ1_S. The “technical” paper also mentions IQ1_S at some point. Sounds to me like the author’s model or the author mixed IQ2_XXS with IQ1_S at some point. Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality; they (somehow) shrink the (already IQ1) FFN weights, and shrink some attention weights from Q5 to Q2. I’m surprised it was even able to generate the text in the demo. Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”. Still doesn’t explain how they compressed the actual model, but I have no interest digging that info from that slop paper.

▲

Tepix 5 hours ago | parent | prev [-]

This 100%

Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.