> QFN Q4 K XL Unsloth is basically identical to the bf16 model.

Yes, but like original poster I have 16GB of VRAM and 64GB of RAM, I was referring to non-Strata benchmarks :)

I did quickly run a couple things on Strata but haven't gotten around yet to benchmark bigger models/customize them with Strata, but will do rather soon. So far Strata/Swift IQ2_XS performed really well compared to non-Strata/Qwen3.8 quantized variants.

> This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.

Yep, totally get it, and to be honest getting to run ablated models sometimes can also be useful.

> Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)

My rig is... ~1.5k, maybe 3k if we count the CPU and GPU upgrade since 2019 :-D I really wanted to grab one of the latest mini-PCs (Halo, Spark, and other minor assemblers) for local setup, but to be honest I can't justify a 6k+ purchase for me and my wife alone. Perhaps in 3 years these mini-PCs will get to the ~2k ballpark (perhaps used), and although older by then, I might be able to run things 24/7 locally as well.

With 32 GB more of RAM you can run ISTA GSQ RCO IQ3 XXS fast. You can make it work now but it is tight. You can stream experts from disk with builds now. You will get an okay token rate. Try it out if you have some disk space. It is a very good local model. I benched it at that quant. Sonnet 5 class at coding benchmarks.

Get a couple or V100S GPU. Figure out cooling. They will run it /fast/ with the right engine. :)