Remix.run Logo
▲ baoooooooooooo 21 hours ago

I'm always on the lookout for reports from my 3090 friends

Looking a bit oof on reddit rn, I'm seeing reports of 10 tok/s via llama.cpp :|

https://www.reddit.com/r/LocalLLaMA/comments/1y1g9cr/inkling...

Edit: Hopeful that this gets the ninfer treatment, if the benchmarks are to be believed it feels like there is probably both amazing speed and real performance to be unlocked by the community here on the x090 cards

▲baoooooooooooo 5 hours ago | parent [-]

Back like 12 hours later to say much progress has been made, folks are already hitting >280tok/s on 5090 configs

These really low speed early results had compilation issues that were causing CPU fallback