How do you do that if you only control one end?

Asymmetric encryption? Both you (the human) and the agent publish public keys, the agent sign/encrypt the OTP request with you public key, you verify/decrypt using your private key, then do the same the other way to send the OTP (always encrypted though, given you’re sending a secret).

Something like that?