That works for up and down. But what about side to side? I don't know anyone who inverts left/right. If the lever was at the back of the head, won't left/right be inverted too?

Put your finger on top of the character's head. Left and right are still left and right, but forward (up on the controls because of how they're positioned) makes the character look down and backward makes the character look up.