| ▲ | adrian17 4 hours ago | |
I’m confused too and the slop readme doesn’t help at all. > Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB And later > Standard IQ2_XXS (76.87 GiB, Unpatched) Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to. Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants. | ||
| ▲ | adrian17 11 minutes ago | parent [-] | |
PS: only now noticed that 76.87GiB _is_ 82.5GB, which happens to be size of the unsloth IQ1_S. The “technical” paper also mentions IQ1_S at some point. Sounds to me like the author’s model or the author mixed IQ2_XXS with IQ1_S at some point. Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality. Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”. Still doesn’t explain how they compressed the actual model, but I have no interest digging that info from that slop paper. | ||