I mean I think we are all clearly prejudiced, given that our brains just immediately pre-judge such a map as upside-down. Nobody, or at least certainly not the OP, is calling us bad for doing so.

It's the same as logical fallacies: you're not a bad person for falling prey to them, but they ARE something you should be aware of if you're trying to make logical arguments.

Following a convention is not prejudice. Conventions are generally a good thing.