Why not? Aren't you mixing gender and sex?

If you try to define athletic divisions by sex, you're going to have a hard time. The world's best athletes probably all have some unusual biology. About 1.7% of people are not completely biological male or female. I would expect that proportion to be higher for Olympic athletes. Excluding 1.7% or 2% or 10% of the world's best athletes from competing would not be fair.