Why wouldn't you also need an inverted X axis to truly mirror your head movements?

That's true. I used to think I was mentally modeling the input as dragging a joystick on the back of the character's head, but it doesn't quite work if you are looking left and right. Now I wonder if it's a lot more subconscious and the "model" explanation is just a rationalization that feels right.

Maybe it's like how some people feel more natural goofy foot on a skateboard/snowboard than the regular way, regardless of their handedness.