llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit.

Two examples:

- https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost a month. Fortunately there's workaround (switching to Vulkan rather than ROCm devices), but both the way the bug was introduced and the way it is not fixed just doesn't give much confidencen

- LM Studio is using llama.cpp internally for GGUF, they ship their own build with their closed source system as "runtimes". Their ROCm runtime does not enable the the AMD GPU inside the Framework, even thought the llama.cpp version would support it. So their runtime keeps telling me that there's no supported AMD GPU -- again, the solution is to use the GPU with the Vulkan devices. Not fixed since Jan at least https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1...

I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan, though in some cases I don't have enough knowledge, or I don't have access to fix things, and that feels like a bummer...

▲

bjackman 34 minutes ago | parent | next [-]

I think ROCm is just a total second class citizen in the space TBH.

It's a shame coz it's not even really what we want, we would obviously all be better served if we could use Vulkan or something. But I guess it's inevitable that a generic framework lags behind here.

If I was AMD I'd hire a whole ecosystem team to sit next to the ROCm people and just support big users like llama.cpp to work better on their HW, e.g. giving OSS maintainers access to their board farms. Maybe they have already done that, in which case I guess I should say I'd double the size of that team.

▲

d3Xt3r 3 hours ago | parent | prev | next [-]

So are there any alternatives which do actually work well with ROCm OOTB?

	▲	imrehg 3 hours ago \| parent [-]
		I just switched to Vulkan, and be done with it. :) As much as I can tell, the ROCm version of llama.cpp would be a bit faster on prompt processing, but about the same on the token generation as Vulkan. Real life benchmarks don't seem to give any "ROCm or nothing" sort of vibes. And the difference between the performance of different models are way bigger than the difference between the llama.cpp versions (and versus different runtimes like the llama.cpp/GGUF and the MLX runtimes on Mac for the same models)... I've tinkered enough with the serving, that I'd rather do something with them with, say 10% slower speed, than spending hours on seting things up again... YMMV

▲

etdznots 3 hours ago | parent | prev [-]

The first one multiple contributors highlighted the PR as urgent andits had lots of review but it appears to be waiting for another review and/or someone that owns the affected hardware to test that the PR fixes the issue, it wpuld be easy for you to test and report whether or not it does, and the second thing is not related to llama.cpp at all

Yes ideally there would be testing every hardware + software combo but this costs engineering time and $$$ money, and you are running on master branch, no master branch of any software is stable, inherently, if you run into issues, just stick to the old hash where stuff worked, why are you insistent on both being at the bleeding edge and experience 0 breakage!

	▲	imrehg 3 hours ago \| parent [-]
		I did report my test results on the first one. :) The second I didn't say it's any of llama.cpp's "fault", but it is _related_ to llama.cpp since it's being shipped in another system, aye? Can't stick to the old hash either, because older version have different bugs. E.g. on older versions the same Qwen3.6 model reliably fails to call specific tools due to template issues, while just having the newer llama.cpp version has that fixed. So different versions - different bugs, rather than no bugs. Why the beating you are trying to gimme, mate? :)