Many (not all) of the people labeling this position as racist are themselves the immigrants. If you legally prevent someone from immigrating to your country, then the ability of the would-be immigrant to call you racist in a way that matters is severely curtailed (since they're not currently in your country, they might well not even be doing so in a language you speak).