Remix.run Logo
▲ jakswa a day ago

I had astra patch in both required llama.cpp PRs and it's passing a few tests locally on my 7900XTX so far, but ~500 tok/s prefill and ~26 tok/s decode are sobering: The new three-time-looping might be cutting speeds. Maybe there's something here still though, maybe it needs less reasoning output for results to be usable/good? More testing needed!

edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE

edit2: trying to see if this dspark drafter is usable and will give a good speed boost, didn't load up on first attempt. I'm probably too early [5-10min later] ... found an unsloth llama.cpp build with vulkan dspark support and it actually slowed down decode :(

▲enlyth a day ago | parent | next [-]

I'm getting 700 tok/s prefill and 52tok/s generation on a 5090 RTX, not bad

▲shadowpho a day ago | parent | next [-]

That seems slow, should be much much faster hmmm

▲jakswa a day ago | parent | prev | next [-]

5090 RTX is like 1800GB/s? I bet you were seeing much higher speeds on Qwen 3.8 27B right?

▲DeathArrow 8 hours ago | parent | prev [-]

It's bad! I got 150 tok/s on RTX 3090.

▲rsolva a day ago | parent | prev [-]

Hm, that seems strangely slow. Qwen3.8-27B/DaisyRose-27B Q4 currently run with ~66 t/s and ~800 t/s prefill with 3 slots and about 190k kv cache. It performs really well so far.

▲jakswa a day ago | parent [-]

Yeah I've had 27B (dense model) up to similar speeds on this setup as well. I'll hold off to compare my speeds to others, maybe it's too early or it's particular to my setup.