I have a personal unscientific theory that some of it, domestically within Japan, is as simple as a heightened degree of racism towards people with darker skin color.

From research I've learned that placing a high value on blending in is a cultural thing. There's the resulting good, like the focus on harmony which promotes good relationships (and is a major contributor to the low crime rate), and there's the bad, like spurning those who look/speak different (I've heard stories from a few who went to visit seeing signs that gaijin aren't allowed in certain establishments, etc).