Is anyone disputing that?

What people are arguing against is that making having a passport mandatory to participate in society is an unreasonable burden, under the current structures. If you wanted to mail a passport to every American in the mail, at no cost, no questions asked, that would be a very different proposition to what is being discussed.