If anything they should be banned in private spaces, like if someone wearing them enters someone's home etc.

There is no expectation of privacy in public.

The owner of the private space generally has authority to deny this already, there's no need for an additional law.

In the US at least, any private homeowner/renter can deny entry to their property, barring legal warrants and exceptional circumstances. A business can have a policy, and is generally legally protected as long as the policy is 1) equally applied, and 2) does not violate ADA... A court would have to weigh in if glasses are allowed or not for ADA... but I suspect there's already a case where a movie theater banned such glasses and they would probably(?) win, since such individuals could be expected to have non-recording glasses.