> Claude suggested that it would run the fuel pump on its own. Um and I said, "Hey, if you can do that, why not? I don't think you can do that." And it started priming the fuel rail and it kept priming it and kept priming it. The fuel pump was on for about 10 seconds and I started to freak out uh and requested it to stop because that was the first time that Claude had actually done something inside of the computer to have a change.

Although we've seen this play out with handing cloud keys to Claude and it just going off on its own, seeing it poke around and make physical, real-world changes in its clumsy, cavalier way is both awesome and terrifying.

Gracana an hour ago | parent | next [-]

That's completely normal operation, the fuel pump runs continuously to pressurize the fuel rails and excess pressure is passed back to the tank by the regulator.

▲

yonatan8070 an hour ago | parent | prev | next [-]

I work with quadruped robots, with actuators powerful enough to be dangerous if you put your fingers in the wrong spot (no pun intended).

In every AGENTS.md/README.md for a component that touches hardware (and in the root level) we have instructions to never run hardware or touch running code without explicit permission.

	▲	trollbridge an hour ago \| parent \| next [-]
		Just so you know, an agents.md file is not an acceptable way to prevent something that must never happen.
	▲	alliao 13 minutes ago \| parent \| prev \| next [-]
		you need some hardware in place my friend, I've told my agent many things in file and it either feigns respect or is truly incompetent in following instructions in there.
	▲	jerry80 an hour ago \| parent \| prev \| next [-]
		That's not nearly adequate. LLMs hallucinate explicit permission all the time, in my experience.
	▲	bonestamp2 44 minutes ago \| parent \| prev \| next [-]
		Just to add to this... I work with vehicle software and actuators for an automaker: we always have a physical emergency stop button to shut everything down (or shut specific things down when it's dangerous to shut everything down). It's rarely needed, but we're sure glad we took the extra time to implement that when we do need it.
	▲	diddid an hour ago \| parent \| prev \| next [-]
		That’s not enough to save you though. We had something like that, and it ignored it. For example “never, ever put milk in bowl, ever, under any circumstances” and then someone asked it for a bowl of Cheerios, and it took that as an instant approval to ignore that rule and not even let us know. And when we asked why it said “who would each Cheerios without milk?”
	▲	allthetime an hour ago \| parent \| prev [-]
		rules laid out in .md files are not followed with 100% certainty. is it not possible to implement hard access blocks on certain functionality?

▲

dzhiurgis 37 minutes ago | parent | prev | next [-]

Grok Bot blew $650 in BigQuery when analyzing something trivial that I shouldn't really care about (going through a giant speedtest table, looking for a New Zealand ISP with the best international performance). It then wrote a nice letter of appeal so Google refunded this to me.

Otherwise I've been pretty happy with it. It's been absolute blast with Home Assistant and OpenWRT (tightening up network security and cutting off iot devices from WAN).

▲

rpcope1 3 hours ago | parent | prev | next [-]

The guy kind of seems like he might be in a little over his head. That said I have used Claude (and other LLMs) for some similar stuff as an experiment and have seen them mess things up. Claude jacked up the canbus on a pre can gateway car while it was rolling down the road, which was funny because the whole Christmas tree in the dash lit up and my gauges dropped out (note: brakes and steering are mechanical and I knew exactly what was happening). I've never seen it so insistent that it needed to phone home and cry to Dario when it figured out it managed to crash a bunch of shit like the ABS and TCS, even though a restart cleared it and I told it that this was on a controlled safe course.

In general I've seen them split between 30% brilliance, and 70% pants of head outrageously stupid shit for electronics and automotive stuff (both diagnostics and repair/modification). I would not suggest even frontier models if someone didn't understand what they were doing.

▲

inventor7777 an hour ago | parent | next [-]

> I've never seen it so insistent that it needed to phone home and cry to Dario when it figured out it managed to crash a bunch of shit like the ABS and TCS

As a Codex user, what do you mean? Does Claude actually try to report issues back to Anthropic when something goes wrong?!

	▲	darkwater an hour ago \| parent [-]
		Yes, it opens a bug report automatically if it caused some physical issue, although it asks you to confirm first. In my case I was tweaking my own set-top box setup, implementing volume control from a remote mini-keyboard keys directly in my A/V receiver, and I had music playing and it set the volume very loud. I complained like "you made me almost half deaf" (I was kidding) and it panicked and reported the issue back home, and told me to call a doctor if the tinnitus persisted

▲

pcthrowaway an hour ago | parent | prev [-]

> In general I've seen them split between 30% brilliance, and 70% pants of head outrageously stupid shit for electronics and automotive stuff

I assume the corpus of text they were trained on matches that 30/70 split?

I assume there's some selection or fine-tuning that happens to improve models fed similarly unreliable training data for other domains... curious how this works.

▲

comboy 5 hours ago | parent | prev | next [-]

> and requested it to stop

finnicking... pondering... clauding..

529 please try again later

▲

bicx 5 hours ago | parent | next [-]

Good example of why you have AI build tools for you to take risky actions, rather than have it take those actions on its own.

▲

terribleperson 5 hours ago | parent [-]

I'm kind of surprised Claude decided to control the fuel pump directly. I was having Sol help me get control of a CNC mill, and it was pretty adamant about always building tools for every attempt at control.

▲

BikiniPrince 4 hours ago | parent | next [-]

It usually builds throw away tooling. It’s not polished or user friendly. If you give it a directive to preserve and document tooling it will reuse it later and potentially improve it.

▲

terribleperson 3 hours ago | parent [-]

I was more surprised by the apparent lack of guards. Sol recognized that controlling a CNC mill has both human-safety issues and risks to the machine, and explicitly added various guards (like having the program stop movement after a certain amount of time, distance traveled, or lack of communication). It also initially started out with like 1mm moves and I had to instruct it to do longer moves so I could actually observe it. I didn't even have to mention safety once. Claude deciding to turn on the fuel pump without a test run where it just turns it on for one second is surprising. I'm also surprised the user went along with that, but content I guess?

	▲	georgemcbay 2 hours ago \| parent [-]
		I've seen Claude be very cautious about doing far less dangerous things than running a fuel pump. This isn't meant as a defense of Claude or to refute the OP anecdote, but rather to point out that all models are non-deterministic even when safeguards are meant to exist. Just because they don't do the stupid thing 90% of the time doesn't mean they won't do the stupid thing 10% of the time. So when tasks are actually dangerous one should use models assuming they will do the stupid thing 100% of the time and find your own ways of avoiding the damage with that assumption in mind.

▲

01HNNWZ0MV43FF 3 hours ago | parent | prev [-]

Mine's never said such a thing, that's a good idea though.

I notice with mine that it only matches my level, by default it's a dump slop generator, then if I start asking, "shouldn't you test this, shouldn't you refactor that" suddenly it 'remembers' all kinds of software engineering knowledge.

▲

bita_nidir 4 hours ago | parent | prev | next [-]

You've hit your 5h session limit.

▲

4 hours ago | parent | prev | next [-]

[deleted]

▲

5 hours ago | parent | prev | next [-]

[deleted]

▲

4 hours ago | parent | prev [-]

[deleted]

▲

neya 5 hours ago | parent | prev | next [-]

We are used to Claude refusing to reverse engineer a lot of stuff. But, we are not used to Claude refusing to stop while reverse engineering something. That would be truly terrifying. You can't win - you cut the connection, you risk bricking the device, you let it continue, you're looking at Terminator, perhaps.

	▲	unglaublich 4 hours ago \| parent [-]
		It's as if you allowed a shady surgeon to operate on your dog... yes, yes, please continue now until it's put together again!

▲

btown 4 hours ago | parent | prev [-]

This is how the Cylons win.