>I mean, we do TOFU for SSH server certificates and nobody really seems to bat an eye at that.

Mostly because ssh isn't something most people (eg. your aunt) uses, and unlike with https certificates, you're not connecting to a bunch of random servers on a regular basis.

I'm not arguing for replacing existing uses of HTTPS here, just cases where you would today use self-signed certificates or plaintext.