This looks very interesting. Possible to get those rates without exotic hardware.

But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison.

We need to see the comparison with this framework and useful models, which at present seems to mean ~30 B.

▲

gaeld 4 hours ago | parent | next [-]

Great points.

We strived to be fair as possible in the benchmark, but it's indeed not perfect. Taalas should have been added in the dedicated hardware section, even though they use 3-bit quantization when we are on FP16 (to be fair in both directions) and they burn the model directly on the card.

Our tech preview is about the speed (hence the small dense model, it was easier to implement).

The math checks out though to allow support for large frontier MoE models at similar speeds: - At batch size 1, GPT-OSS-120B has 5.1B active parameters - in FP8, it's in the same size ballpark than our 2B model in FP16 (5.1 GB vs 4GB). - DeepSeek V4 Flash has 13B in mixed FP4/FP8, so let's say ballpark around 3x bigger than 4GB - so in theory we could reach >1,000 tok/s on it with MI300X/H200 and up to 4k on next generation GPUs.

Check out the math at the end of our blog post:

https://blog.kog.ai/real-time-llm-inference-on-standard-gpus...

▲

Imustaskforhelp 2 hours ago | parent [-]

Your playground/write-up is very interesting and I would be really interested when you can have something like Deepseek V4 Flash model (49B) running as you are suggesting.

I haven't read the article at the moment and I will try to read them hopefully but I wish to ask a question regarding, can this approach be done for say trillion or large parameter models as well or is there some wall which gets hit that makes it valuable for only smaller parameter model.

That being said, its still really incredible because in future, because these small models are really getting good for many use cases and speed becomes their bottleneck, with greater speeds at consumer hardware, I think its gonna be amazing work!

	▲	gaeld 2 hours ago \| parent [-]
		Thanks for the comment and the question! The last section of the article lays out the scaling laws that apply when porting this approach to another model. In a nutshell, DeepSeek V4 Pro with 49B active params is close to the upper bound. Also worth noting that our results are currently for standard datacenter GPUs. On consumer hardware, though the same low-level optimization approach applies, the bandwidth limitations will cap the achievable speed.

▲

kirtivr 4 hours ago | parent | prev | next [-]

They got 1K tok/s with Deepseek v4 Pro. That's kinda cool..

	▲	gaeld 4 hours ago \| parent [-]
		Thanks. To be fair, this number is what we expect to get once we port DeepSeek V4 in our engine on the upcoming generation of GPUs!

▲

hirako2000 3 hours ago | parent | prev | next [-]

Fallacies look interesting ? Like if we aren't getting dubious claims every day ?

▲

cyanydeez 4 hours ago | parent | prev [-]

likely the small model makes whatever fuzzer they designed to poke the gpus much faster optimizations.

they seem to think it scales up because theyre shortening the stack.