The encryption itself may not be.

Establishing the initial exchange of crypto key material can be.

That's where certificates are important because they add identity and prevent spoofing.

With TOFU, if the first use is on an insecure network, this exchange is jeopardized. And in this case, the encryption is not with the intended partner and thus does not need to be attacked.