> It quiets down the logs nicely, people aren't always knocking on the door.

Q: If you've still done the right things - "disable[d] password auth, disable[d] root login, and only use ssh keys for access" - why do you care about how 'quiet' your logs are?

Whatever you've done, you should keep an eye on your logs for anything suspicious. A quieter log is easier to monitor.