You don't need to carry water for racists and invent more palatable explanations for what they said. There's nothing in there about not enough British people - only about too many foreigners (whom he can tell by looking at them).