| ▲ | rsolva an hour ago |
| Going to test this right away on my AMD 7900XTX and see how it does compared to Qwen 3.8 27B (or DaisyRose 27B that I'm currently running). Given that Inkling 2 Mini is a MoE, it should provide way higher decode speed, but with comparable performance. Exiting! |
|
| ▲ | IAmGraydon 3 minutes ago | parent | next [-] |
| >Exiting! OK, but please come back and let us know your results! |
|
| ▲ | AndrewKemendo 37 minutes ago | parent | prev [-] |
| Please reply when you have tested it I’m curious. Also do you know what the best local setup is? |
| |
| ▲ | jakswa 25 minutes ago | parent | next [-] | | I had astra patch in both required llama.cpp PRs and it's passing a few tests locally on my 7900XTX so far, but ~500 tok/s prefill and ~26 tok/s decode are sobering: The new three-time-looping might be cutting speeds. Maybe there's something here still though, maybe it needs less reasoning output for results to be usable/good? More testing needed! edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE | | |
| ▲ | enlyth a minute ago | parent | next [-] | | I'm getting 700 tok/s prefill and 52tok/s generation on a 5090 RTX, not bad | |
| ▲ | rsolva 20 minutes ago | parent | prev [-] | | Hm, that seems strangely slow. Qwen3.8-27B/DaisyRose-27B Q4 currently run with ~66 t/s and ~800 t/s prefill with 3 slots and about 190k kv cache. It performs really well so far. | | |
| ▲ | jakswa 18 minutes ago | parent [-] | | Yeah I've had 27B (dense model) up to similar speeds on this setup as well. I'll hold off to compare my speeds to others, maybe it's too early or it's particular to my setup. |
|
| |
| ▲ | rsolva 28 minutes ago | parent | prev [-] | | For 7900XTX you mean? I run Fedora Silverblue 44 at the moment and could share my current containerized setup with DaisyRose 27B. |
|