Good example of why you have AI build tools for you to take risky actions, rather than have it take those actions on its own.

▲

terribleperson 5 hours ago | parent [-]

I'm kind of surprised Claude decided to control the fuel pump directly. I was having Sol help me get control of a CNC mill, and it was pretty adamant about always building tools for every attempt at control.

▲

BikiniPrince 4 hours ago | parent | next [-]

It usually builds throw away tooling. It’s not polished or user friendly. If you give it a directive to preserve and document tooling it will reuse it later and potentially improve it.

▲

terribleperson 3 hours ago | parent [-]

I was more surprised by the apparent lack of guards. Sol recognized that controlling a CNC mill has both human-safety issues and risks to the machine, and explicitly added various guards (like having the program stop movement after a certain amount of time, distance traveled, or lack of communication). It also initially started out with like 1mm moves and I had to instruct it to do longer moves so I could actually observe it. I didn't even have to mention safety once. Claude deciding to turn on the fuel pump without a test run where it just turns it on for one second is surprising. I'm also surprised the user went along with that, but content I guess?

	▲	georgemcbay an hour ago \| parent [-]
		I've seen Claude be very cautious about doing far less dangerous things than running a fuel pump. This isn't meant as a defense of Claude or to refute the OP anecdote, but rather to point out that all models are non-deterministic even when safeguards are meant to exist. Just because they don't do the stupid thing 90% of the time doesn't mean they won't do the stupid thing 10% of the time. So when tasks are actually dangerous one should use models assuming they will do the stupid thing 100% of the time and find your own ways of avoiding the damage with that assumption in mind.

▲

01HNNWZ0MV43FF 3 hours ago | parent | prev [-]

Mine's never said such a thing, that's a good idea though.

I notice with mine that it only matches my level, by default it's a dump slop generator, then if I start asking, "shouldn't you test this, shouldn't you refactor that" suddenly it 'remembers' all kinds of software engineering knowledge.