> Who actually benefits from such limits?

People who work. It's just an economic argument. The US (and any other targets of net immigration, really) has a higher wage level than the regions from which the labor is arriving, more or less by definition. That's why the labor force is moving.

So if you allow completely unlimited population motion at zero cost, the system will seek to a state where all wages for a given job are the same, everywhere. And that means that we highly-paid Americans end up poorer.

Now, is such a world more just? More fair? Maybe! But it's worse for us, and that makes it a politically infeasible solution to argue for. You'll never convince people to live poorer for the benefit of others.

A feasible/reasonable/moderate/boring immigration policy would simply ask the question "How much immigration is needed to fill existing jobs (good for growth) without depressing wages (bad for workers)?", and allow that much.