| ▲ | AndrewKemendo a day ago | |||||||||||||||||||||||||||||||||||||||||||
Please reply when you have tested it I’m curious. Also do you know what the best local setup is? | ||||||||||||||||||||||||||||||||||||||||||||
| ▲ | jakswa a day ago | parent | next [-] | |||||||||||||||||||||||||||||||||||||||||||
I had astra patch in both required llama.cpp PRs and it's passing a few tests locally on my 7900XTX so far, but ~500 tok/s prefill and ~26 tok/s decode are sobering: The new three-time-looping might be cutting speeds. Maybe there's something here still though, maybe it needs less reasoning output for results to be usable/good? More testing needed! edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE edit2: trying to see if this dspark drafter is usable and will give a good speed boost, didn't load up on first attempt. I'm probably too early [5-10min later] ... found an unsloth llama.cpp build with vulkan dspark support and it actually slowed down decode :( | ||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||
| ▲ | baoooooooooooo 20 hours ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||
I'm always on the lookout for reports from my 3090 friends Looking a bit oof on reddit rn, I'm seeing reports of 10 tok/s via llama.cpp :| https://www.reddit.com/r/LocalLLaMA/comments/1y1g9cr/inkling... Edit: Hopeful that this gets the ninfer treatment, if the benchmarks are to be believed it feels like there is probably both amazing speed and real performance to be unlocked by the community here on the x090 cards | ||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||
| ▲ | rsolva a day ago | parent | prev [-] | |||||||||||||||||||||||||||||||||||||||||||
For 7900XTX you mean? I run Fedora Silverblue 44 at the moment and could share my current containerized setup with DaisyRose 27B. | ||||||||||||||||||||||||||||||||||||||||||||