My self hosted LLM runs at 60-ish tok/s and does great at agentic tasks. I think that’s about the threshold where it stops feeling annoyingly slow.

I have a couple of V100 and am working on making Flash Next really fast on them. So far I am at like 193 t/s decode on a specialist 3-bit quant. On shorter prompts. Around 109 t/s on a ggml/llama fork at depth before I gave up and started writing the engine from scratch. I guess I will end at about 220-250 t/s at 128k context size with about 3K prefill on these cards.

https://stealthgoat.com/deepswe-flash-next.html (Total AI pile). I benched this quant for 3 days on DeepSWE in a 16GB VRAM gaming PC. It held 90 t/s on avg I think. It’s solid. The authors claims of minimal intelligence loss hold. Sorry it renders bad in mobile will fix it soon.

It feels great to have a local model ripping at these speeds. I now use Flash NeXT as an active agent managing part of the engine build process and let it do some coding on easier chores :)