I know of one person who was born physically a woman, but has XY chromosomes. It is only due to modern medicine that we know that there is anything "unusual" with her gender. Otherwise, she is physically a woman with no observable clues to her condition.

(IE, in the past, she would have been infertile, and probably died young due to her situation.)

I'm not comfortable with saying that people like her need to compete with men.

Complete androgen insensitivity syndrome, I imagine? That's probably the one category of XY people who have undergone no hormonal masculinization throughout their lives, and the one case where I'd agree with them competing with women. Wikipedia says it's estimated to be "1 in 20,400 to 1 in 99,000".