They probably mean leaving ssh open to all ips. Take a look at your auth failure logs to see the thousands of daily attempts to compromise your server using default passwords. Most of those are low effort and low risk. Sometimes the bots will try password stuffing. Disabling password auth in sshd config is good practice. Fail2ban also helps block repeated attempts like that.

There’s also the risk of a zero day RCE vulnerability in ssh (though I’ve not seen one in the 20 years I’ve been paying attention )

I tend to not expose ssh to the world and log in with some other method to pass the perimeter (VPN, IP whitelist, tailscale) and the ssh from inside.

fail2ban seems like security theater for a keys-only SSH server, and it won't help against zero days either (unless it happens to be one that requires many attempts).

The only thing it helps with is log spam, but then why not just configure SSH to not log login failures?