That argument only works if society distributes wealth more equitably.

Current setup simply brings in foreign labor so that capitalists can reduce wages and they pocket the profit, while Americans pocket the costs. Not to mention migrating for purely economic reasons is obviously not going to make the locals like you very much.