| ▲ | Philpax an hour ago | |
Using a RTX 5090 ($3~4k), I can run Qwen 3.8 27B at ~180 TPS with ninfer [0]. With its thinking maxed out, I can confirm that the quality of output is roughly on par with Opus 4.5~4.6 - that is, this 20GB file really can write software by itself, but the amount of thinking required makes it strictly slower than larger models, even at 180 TPS. Still, it's largely replaced the cheap tier of the frontiers that I would otherwise be using. It can be run with older GPUs (a 3090 is ~1k), but the time spent thinking will become a fairly noticeable impediment for staying in the flow. The next step up would be to run DeepSeek V4 Flash 0731 on two DGX Sparks (~$10k), which serve at 60 TPS and sit somewhere around Opus 4.7 level without 3.8-tier thinking. However, it is worth noting that, if you are buying this hardware just to serve LLMs, it is not cost-effective. It would take over a decade of continuous use to make back the cost of the DGX Spark setup in 0731 tokens. I'm running this setup because I happen to have a 5090, and the two 3090s in my server were cheap enough when amortised over several years. | ||