Llama.cpp can quantize without special training, but I'm not sure if any special model architecture support is needed to read it in the first place. If it can be converted to gguf at all and you know what tensors to target, it can get the full ternary bonsai treatment today.

▲

NitpickLawyer 38 minutes ago | parent | next [-]

Sure, but that's for "personal" serving. I meant for 3rd party providers. Usually we get a good indication on what it costs to host this, as the prices settle on open router. That's why I said it's tougher to serve than kimi k3 on launch. As a provider you'd do fp8 if the model creator didn't do QAT on q4, or until someone does a good calibrated nvfp4. And that's usually nvda :)

▲

MrDrMcCoy 28 minutes ago | parent [-]

That makes sense, but your specific phrasing precluded the possibility of non-QAT quantization.

	▲	NitpickLawyer 23 minutes ago \| parent [-]
		Should have worded that better, my bad.

▲

binary132 an hour ago | parent | prev [-]

QAT is an optimizing quantization algorithm, not naive quant.

	▲	MrDrMcCoy 42 minutes ago \| parent [-]
		Right, but the way they phrased it suggested that without QAT it could not be quanted at all.