The way I read it was that he expects large industrial systems with AI to have safeguards to "fail closed."

Which: it seems like we're already getting to the point of capability vs. unexpected behavior that having the ability to "trip" a complex system may be an important design criterion.

I mean, there's plenty of harm these systems can do besides trying to turn us all into paperclips.

▲

AndrewKemendo 2 hours ago | parent [-]

Correct interpretation

▲

bigyabai an hour ago | parent [-]

So, which do we choose?

A) Let AI override safety mechanisms (increases uptime while endangering humans)

B) Let AI impose additional safeties (decreases uptime with questionable safety uplift)

If you intend to go with option B) then there's basically zero benefit to investing in that kind of automation. AGI will be fallible, and even with all the safeties and safeguards it's hard to see what tangible benefit AGI would add to the system. It'll be a hard sell for critical infrastructure of any kind.

I understand that it's hard to get a man to understand something when his AGI Society executive credentials depend on it, but it feels glaringly obvious why AI isn't masterminding these sorts of systems. Even accounting for unprecedented advancement in AI, it's hard to see what AI could contribute to managing these things.

	▲	mlyle an hour ago \| parent \| next [-]
		> It'll be a hard sell for critical infrastructure of any kind Remember, critical infrastructure attached SCADA systems to the internet and has had trouble removing them all despite 30 years of warnings and audits. There's definite advantages to having a LLM in the loop. The best we can hope for is that it's architected in a way that it fails safe and damages are limited, so we get some of the advantages but less of the downside.
	▲	AndrewKemendo 16 minutes ago \| parent \| prev [-]
		There are more than two options