| ▲ | jakswa a day ago | |||||||||||||||||||
I had astra patch in both required llama.cpp PRs and it's passing a few tests locally on my 7900XTX so far, but ~500 tok/s prefill and ~26 tok/s decode are sobering: The new three-time-looping might be cutting speeds. Maybe there's something here still though, maybe it needs less reasoning output for results to be usable/good? More testing needed! edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE edit2: trying to see if this dspark drafter is usable and will give a good speed boost, didn't load up on first attempt. I'm probably too early [5-10min later] ... found an unsloth llama.cpp build with vulkan dspark support and it actually slowed down decode :( | ||||||||||||||||||||
| ▲ | enlyth a day ago | parent | next [-] | |||||||||||||||||||
I'm getting 700 tok/s prefill and 52tok/s generation on a 5090 RTX, not bad | ||||||||||||||||||||
| ||||||||||||||||||||
| ▲ | rsolva a day ago | parent | prev [-] | |||||||||||||||||||
Hm, that seems strangely slow. Qwen3.8-27B/DaisyRose-27B Q4 currently run with ~66 t/s and ~800 t/s prefill with 3 slots and about 190k kv cache. It performs really well so far. | ||||||||||||||||||||
| ||||||||||||||||||||