They don’t want the working class southern Europeans around, they don’t have problem with the rich southern Europeans.

I wont call this xenophobia. Its just rich people annoyed by the poor hanging around outside the working hours. They often even like their poor people that do their things, they are actually annoyed by the other rich peoples employees or the kids of those employees who are seen as not as well behaved as theirs.