Why remove ssh from the LAN? Brute-forcing a cert-based login is unrealistic, and passwords should of course be disabled. You can add a passphrase to your ssh key to make it useless when stolen.

What am I missing?

Yes, what you describe (or storing ssh key on yubikey) would be a more secure setup. I like the web ui though, so having that be available from any device including phone in a reasonably secure way would be great IMO. You can do pretty much anything via uci so when using web ui I see no reason to leave ssh running.