| ▲ | spmurrayzzz an hour ago | |
I think its good advice to test both on your own evals for sure, but the MoE parameters are already natively FP4 in ds4. Dropping to 2bpw isn't as big of a loss as it seems (and as corroborated by antirez's work). Its also only 13B active, so your decode speed would be nearly 2x that of Qwen3.6-27B. So there are other latent benefits as well. | ||
| ▲ | verdverm 10 minutes ago | parent [-] | |
z-lab has been dropping dflash addons for a lot of models https://huggingface.co/collections/z-lab/dflash I'm running the qwen3.6-27B + dflash on a spark and tgen is way up, but keep the number low, acceptance rate is terrible beyond half a dozen and it requires more memory | ||