> My gut feeling is that this is way harder than most people think

I've had this feeling for a while too; partially due to the screeching of "putting your ssh server on a random port isn't security!" over the years.

But I've had one on a random port running fail2ban and a variety of other defenses, and the # of _ATTEMPTS_ I've had on it in 15 years I can't even count on one hand, because that number is 0. (Granted the arguability of that's 1-hand countable or not.)

So yes this is a different thing, but there is always a difference between possible and probable, and sometimes that difference is large.