I was more surprised by the apparent lack of guards. Sol recognized that controlling a CNC mill has both human-safety issues and risks to the machine, and explicitly added various guards (like having the program stop movement after a certain amount of time, distance traveled, or lack of communication). It also initially started out with like 1mm moves and I had to instruct it to do longer moves so I could actually observe it. I didn't even have to mention safety once. Claude deciding to turn on the fuel pump without a test run where it just turns it on for one second is surprising. I'm also surprised the user went along with that, but content I guess?

I've seen Claude be very cautious about doing far less dangerous things than running a fuel pump.

This isn't meant as a defense of Claude or to refute the OP anecdote, but rather to point out that all models are non-deterministic even when safeguards are meant to exist.

Just because they don't do the stupid thing 90% of the time doesn't mean they won't do the stupid thing 10% of the time.

So when tasks are actually dangerous one should use models assuming they will do the stupid thing 100% of the time and find your own ways of avoiding the damage with that assumption in mind.