What do you authenticate against? Your shadow file is in the unencrypted area leaving it susceptible to offline attack.

With the TPM you can fully disable password auth over SSH.

My Raspberry Pi some time ago had a setup where only public key auth was enabled for LUKS unlock, so I only had to have an authorized_keys file unencrypted.

Correct, someone with physical access could run a MitM attack and steal your passphrase. I just find this extremely unlikely, so I honestly don't care.