I've seen Claude be very cautious about doing far less dangerous things than running a fuel pump.

This isn't meant as a defense of Claude or to refute the OP anecdote, but rather to point out that all models are non-deterministic even when safeguards are meant to exist.

Just because they don't do the stupid thing 90% of the time doesn't mean they won't do the stupid thing 10% of the time.

So when tasks are actually dangerous one should use models assuming they will do the stupid thing 100% of the time and find your own ways of avoiding the damage with that assumption in mind.