I have competed in reasonably high level sports, and my wife was US Masters duathlete of the year a few years ago (with me as her coach). I think you're wrong, though it's easy to see why.

Currently, with sex-based categories, a woman can be declared "the best in the world" and most people won't waste much time on the question "yeah, but could she beat the best men?" (granted, some will). They will accept that, e.g. she has the fastest time over 26.2 miles in the world right now, even though a few hundred or a few thousand men worldwide are faster.

If you use performance based metrics to create the categories (the way that road cycling does, for example, though still within gender divisions), that "title" would go away, and likely a woman would only be "the best in the world in division X", other than in (as you noted) some endurance, climbing and gymnastics sports where an elite subset of women could potentially be the best of "top" category.

It isn't completely obvious that this is a negative - how much of a change it would be would depend on a lot of other changes (or lack thereof) in how sport was organized. Certainly if it continued to focus on only the top division, then women would be shut out of most opportunities to be professional. But that's not inherent in the design. I do concede, however, that it is quite a likely outcome of such a category structure.

If we are talking about amateur sports where the stakes are low, the concept of slotting athletes into divisions makes sense.

In elite sports, no one wants to see "best in division X". They want to see the best hockey players, the best golfer, the best skier, etc. The money incentives are considerable.

This would destroy women's professional sports.