We should accept that to get a system that is safe to work in the chaotic real world environment the developers have to test the system in the chaotic real world environment. And that the testing might cause problems.

If this testing constitutes a criminally negligent behavior, it should be punished. Given the benign outcome (the tip got into spam, Anthropic promptly contacted the police) I doubt that it will make the case.

Somehow Waymo built their system without having their test cars drive into buildings, nudge other cars off the road, honk incessantly, drive backwards just for the heck of it, etc.

That's not the line asked for, that's just repeating the statement that caused me to ask for a line, or criteria for when we should prosecute. Sounds like the answer is "never", just like for the copyright infringement on a planetary scale. Ok. "AI" gets a free pass from the state. I better see some of these insanely good benefits, or I'm going to be angry. I bet the rest of the mob will be, too.