And given how both mental models are reasonable, I think a lot of the preference is going to come down to what you're used to.

For me it seems to be tied to muscle memory too? Because I've noticed that when I play using a Gamecube controller I prefer the camera's x-axis to be inverted, but when I play using a modern controller I prefer not inverting it.