Yes, what you describe (or storing ssh key on yubikey) would be a more secure setup. I like the web ui though, so having that be available from any device including phone in a reasonably secure way would be great IMO. You can do pretty much anything via uci so when using web ui I see no reason to leave ssh running.