SOPS reduces the surface area you need to cover. You can use Age as a backend and then you only need a long lived private key on the server. https://github.com/getsops/sops

The bad guys will steal that private key and decrypt the encrypted secrets the same way they can steal the unencrypted secrets directly.