There's no efficient way to run inference through homomorphic encryption. If the inference server is vulnerable, it seems feasible to MITM an unencrypted version.

There’s no efficient way to do anything with homomorphic encryption.