Hell does not work if the agent knows it is code. You cannot scare an optimizer with theology; it just optimizes around the loss function.

Well, the "you" now is not the same "you" 10 minutes from now. The only thing the new "you" inherits is your code, but otherwise it could be a different person. So in a way we are code too.