Well you kinda said it yourself? If men are generally unattractive then a man should be less likely to lose out to a better-looking man, simply because there's fewer of these to go around.

Yes but an average man would lose to most of women in term of looks, style, clothes choice, skin care, makeup, hair, etc. If employers cared only about looks then they would prefer to hire women. Of course, it is possible that only women get judged by looks and men are not.