| ▲ | tssge 4 days ago | |
The GPU has INT4, INT8, BF16 and FP16. Notably no FP8 or FP4.The official GPTQ-Int4 release from Qwen is a great quant for this but custom kernels are still rare for this hardware. | ||
| ▲ | moffkalast 4 days ago | parent [-] | |
Must be a case of the hardware being there and the software not actually supporting it then. | ||