With a binary cache that is not so bad, see for example what nix does.