> Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.

We do, I believe, regulate uranium enrichment (the equivalent on building larger SOTA models), so while you are perfectly free to study theoretical physics and even run very large collider experiments (the equivalent of improving RLHF with DPO), it is generally frowned upon to go full Edward Teller and advocate for scientific experiments requiring detonating thermonuclear weapons in hurricane clouds (the equivalent of, well, see OP).