For SSH to use your keys a calculation has to be done using your private key and then send the results back to the remote site so it can validate that you got the results that prove you have your private key. The TPM and your yubikey do not do this calculation. They allow software on your computer to access the private key in plaintext form, perform this calculation, and then send the result (and then presumably overwrite the plaintext key in RAM). If your system has been compromised, then when this private key is provided to the host based software, it can be taken.

Yubikey (and nitrokey and other HSMs) are technically smart cards, which perform crypto operations on the card. This can be an issue when doing lots of operations, as the interface is quite slow.

Downvoted - this is false, sorry. The whole point of security keys (whether exposed via PKCS#11, or FIDO) is that the private key material never leaves the security key and instead the cryptographic operations are delegated to the key, just like a commercial HSM.

Technically, a private key that was imported (and is marked as exportable) to a PKCS#11 device can subsequently be re-exported (but even then, during normal operation the device itself handles the crypto), but a key generated on-device and marked as non-exportable guarantees the private key never leaves the physical device.

It is not present in the RAM with smart cards, and especially never with server type HSM:

https://wiki.archlinux.org/title/SSH_keys#Storing_SSH_keys_o...

And even the password can be forced to be re-entered by the agent for every use, if that level of security is wanted.