I think a big part of it, historically, is that this control scheme provides negative feedback, which may help stabilize the controls.

Think about the inertia of the pilot and their limbs inside the plane, acting on the controls. A sudden acceleration/jerk in the direction of the control signal will bias the operator's body to input the opposite control signal unless they are tensed up and prepared to maintain it in spite of the forces they experience.

If the nose pitches up suddenly, you're likely to push the yoke forward. If it pitches down suddenly, you're likely to pull back a bit. Similarly, if the plane (or boat) jerks forward, you are more likely to pull back on the throttle than push it forward. A sudden airplane roll will bias you to input the opposite aileron signal.

Even in a car, if you are holding the top half of the wheel as in the classic 10-and-2 grip, a sudden turn will cause you to counter steer a bit as you experience the centripetal force effect pulling you towards the outside of the turn.

If the controls were inverted, all these default inputs would instead cause positive feedback and seem more likely to send a vehicle out of control.