Open source Local LLMs are already a thing. That pandora’s box is way way open already.

If a big company does that they wouldn't be able to hide it, this is just as easy to enforce as any other regulation.

black market guns are also a thing, and they're realtivly easy to manufacture using untracked machines and materials using skills you can develop in under a year. That doesn't mean that regulating the sale / supply / ownership of guns isn't useful.