So, which do we choose?

A) Let AI override safety mechanisms (increases uptime while endangering humans)

B) Let AI impose additional safeties (decreases uptime with questionable safety uplift)

If you intend to go with option B) then there's basically zero benefit to investing in that kind of automation. AGI will be fallible, and even with all the safeties and safeguards it's hard to see what tangible benefit AGI would add to the system. It'll be a hard sell for critical infrastructure of any kind.

I understand that it's hard to get a man to understand something when his AGI Society executive credentials depend on it, but it feels glaringly obvious why AI isn't masterminding these sorts of systems. Even accounting for unprecedented advancement in AI, it's hard to see what AI could contribute to managing these things.

> It'll be a hard sell for critical infrastructure of any kind

Remember, critical infrastructure attached SCADA systems to the internet and has had trouble removing them all despite 30 years of warnings and audits.

There's definite advantages to having a LLM in the loop. The best we can hope for is that it's architected in a way that it fails safe and damages are limited, so we get some of the advantages but less of the downside.

There are more than two options