Remix.run Logo
alessandroberna 4 hours ago

It's just a wrapper around llama-quantize, i don't see why it wouldn't work.

Yet it's only useful if one wants to have both the F16 original model plus all the other GGUF quants at a time and doesn't mind spending compute every time they need a specific quant again.

It doesn't do much for a consumer who only needs one or a few specific quants that work on their hardware and I'd wager it's not something that multi billion dollar companies need. At least as far as I'm aware major llm providers aren't using gguf and llama.cpp to serve their inference