Remix.run Logo
tredre3 2 days ago

Do you have any reasons to believe that granite is more immune to the effects of quantization than other tiny models? Otherwise it seems odd to judge a tiny model true capabilities by using its 4bit quant.

simonw 2 days ago | parent [-]

This model is small enough that it might be sensible to try the same prompts against all of the quant sizes to try and spot any differences.

simonw 2 days ago | parent [-]

This inspired me to give that a go: https://simonw.github.io/granite-4.1-3b-gguf-pelicans/

archerx 19 hours ago | parent [-]

That was interesting almost like a weird little modern art gallery. I’m surprised that the BF16 one looks so bad…