> The issue I have run into is that for running llama.cpp, which is a very actively developed bleeding-edge software, it seems like experimenting with different versions/configs/patches etc. was fighting with the Nix philosophy of immutable software

May be. I'd guess it probably does it in a high-friction way compared to what you expect.

e.g. in a typical linux distro, you can just replace the (globally installed) version of the package and use that.

With Nix, you've got a declaration of how to build a package, including its dependencies; and each of those dependencies is also a declaration of how to build that package, and so on. -- And so, "just change this version" results in rebuilding the full chain of dependencies.

If you want multiple readily-available versions of llama.cpp each with different configs/patches, then using Nix would make this easier I'd think. But for "I just wanna try this, then try that", it's going to add overhead.

It requires rebuilding the nodes whose inputs have been affected by the change, not the entire tree.