IMO: every countries and localities have the concept of "but where are you FROM", to varying degrees. People just don't expect Japan to be a big enough place that could have such a thing, thinking it's like a tiny dying village at the end of a small branch near the tail end of road to Rome, distracted by its Westernized facades and microscopic presence in the world. It doesn't register as a nation with population size of France and Germany combined. So there's often a bit of divergence between perception and reality coming from there.