This seems like a sarcastic/unserious comment, but based on my interactions with people who are supposedly anti-immigration - yes, it's entirely based on skin colour.

Someone from India, China etc whose family immigrated in the 1800s to work in gold mines/railroads etc and probably has deeper roots in the country than the person criticising them = immigrant, bad, shouldn't be here. Somehow simultaneously taking all the jobs and living off the state and not contributing.

Someone from Europe/America/Canada with white skin who either came here as a child or was born here to immigrant parents = not a problem at all, they "don't count" for some reason.

I agree the people who lump children of Chinese railroad workers in with illegal immigrants are racist. But it’s the pro-immigration folks that do that pervasively, under the label “people of color.” Meanwhile, you have to look at pretty fringy parts of the right to find that.