llama.cpp has a flake in-repo that I often use for trying out different branches and patches. I also have yet to have an issue just replacing the src and build number attribute in the nixpkgs build (in one cases I wanted to add an additional CMake flag but that was also easy).

IMHO this is way easier than without Nix in many cases as figuring out the upstream build process and getting it running can often be quite the chore. With Nix it is all set up for you.