This is how my brain works too. The joystick is the head and my thumb is on top. I pull back to make the little joystick man look up, and push forward to look down.

Edit to add:

>It turns out the most predictive out of all the factors we measured was how quickly gamers could mentally rotate things and overcome the Simon effect. The faster they were, the less likely they were to invert. People who said they sometimes inverted were by far the slowest on these tasks.

This tracks with me. I feel like games that require quick multi-dimensional movements (FPS includes) I'm dreadfully slow at. Especially if the game doesn't have the one control setup that my brain prefers, which many don't.