Sounds plausible at least, but I think the question isn't necessarily making a valid assumption. Why do men have to have nipples? Why is our retina installed backwards? Why do sinuses drain upwards? It's just a path evolution took, it doesn't jump to some optimal design.

Asking “why” questions about our body / evolution often (not always) gives informative answers. As in the example you gave: https://www.scientificamerican.com/article/the-purpose-of-ou...

male nipples are developmental vestiges, the male condition is derived from response to embryonic testosterone, and is a developmental variation from default.

early stage embryos of both sexes are not easily distinguishable by genitalia, they look morphologically similar. later developmental events culminate in morphological rearrangement to male form.

lack of response to testosterone during development results n a curious state of affairs, where a person is genetically male, having x, and y chromosomes, develops according to a female plan. external appearances are female, with loss of secondary sex development in puberty.

Androgen insensitivity syndrome:

https://en.wikipedia.org/wiki/Androgen_insensitivity_syndrom...

Correct, though interestingly apropos to the discussion is that sex is one of the ways evolution is able to get around local maximums.