Even if you ignore my more fundamental objection to that paradigm, I don’t think that it makes any sense on the level you discuss either.

But - just to play along, LLMs do act differently if you tell them they will be punished. And, they do appear to simulate suffering-like behavior.

I just think the adversarial model of trying to catch and punish misbehavior quite obviously sets up adversarial us-vs-them dynamics between AI and humanity, and also simply won’t work when the agents are ~as smart as is but faster, let alone smarter than us.

Unless, you get the AIs to be fundamentally aligned to our values, such that the majority of AIs support some sort of punishment for misbehaving AI. And that alignment part is the hard part we need to solve first. The rest is easy.

To be clear, I was being entirely silly - mostly to express agreement with your point that our current laws aren't really built for a world with lots of agentic LLMs running around in it.