Why do I always invert the Y-axis but never the X-axis?

Because motion along the x-axis (left-right) is a body rotation about the gravity vector. y-axis motion is not.

We are upright beings in a gravitational field, so if we see a berry to the left of our visual field, we turn our whole body to face it. Then we walk towards it. We do this from a first person vantage only. We don't see our own backs - just the world itself.

But if a berry is above our visual field, we can't rotate our body that way. That would make us fall over. We instead remain vertical to gravity and rotate a third-person thing. We tilt something else like an arm or stick in the direction. We see this from a third-person vantage only. We see the back of the arm, or back of the stick. If the berry is up high, the part of the stick closer to us is down low. We see the inverted end moving, so it becomes intuitive. Of course, you can focus on the far end of the stick and get a non-inverted intuition too. But this is only possible from a third-person view which we don't often get when our bodies so easily rotate about the gravity vector.