Is this not the case with men?

To an extent, yes.

However, men's clothing is generally rather shapeless, whereas women's clothing is usually quite figure-fitting.

Women also have a lot more variety in body shape: breasts are a thing, for starters, and there's a wide variety in which areas fat is primarily stored.

Is sizing an issue for men as well? Yeah, probably. But it is significantly worse for women.