Now anybody with root/sudo/physical access to the remote machine has full R/W access to your entire home directory.

Well, what if it's a separate directory meant exclusively for remote systems alone? And what if the remote mount is read-only, perhaps with a writable layer on top using overlayfs that can be discarded on logout?