I don't think this is right.

Yes, with TPM and yubikey you have the option to store the per key material on disk, encrypted by the TPM. But the way this is then used is that the PKCS software sends that encrypted blob AND the requested operation, and gets only the output back. The CPU doesn't get the SSH private key back. Just the output of the RSA operation using the key.