QFN Q4 K XL Unsloth is basically identical to the bf16 model. It’s KL divergence is very very small. It is competitive with the full model. Also I benchmarked iq3 xxs on Strata for 3 days against deepswe and it scored better than the published benchnarks for the model. If the gsq rco quant prettain tasks fit well it is a great model quant. My data shows it takes 20% more tokens basically than q4 xl. A4 xl is within noise on all my repeated task benchmarks over the public model serving and my own runs of q8. I published the deeswe results. You have to be very specific about model choice and know these models and their quirks.

This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.

https://stealthgoat.com/fnfl/pelicans.html

Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)