This works as long as your Dockerfile is reasonably reproducible and does its best to lock dependencies. However, this approach has failed me a couple of times in the past. For example, I rebuilt a container some weeks later, in the meantime a new version of clang had been released that just so happened to break my build due to a bug.

I personally use Nix these days, but the complexity is too high for me to recommend it to everyone for every software project.

Yeah, Nix pretty much solves this problem. The other day I wanted to try a really old version of spaCy for fun/historic interest. spaCy 1.8.2 installed freshly from the binary cache on NixOS-unstable as if it was still April 2017.