Not even the singularity can fix php.

But all jokes aside, we're entering a new era where there's a paradox of pro-AI people going AI is really good, but not that good... when it kind of is that good.

We'll probably have ai monitors as a service, but the bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.

In Mike Pondsmith's Cyberpunk universe (the same one as Cyberpunk 2077), there's this point of in-universe history that the Old Net got overrun with rogue, nigh-on sentient AIs that made safe use of the internet as it was known then impossible.

Society's response to that was to build a new Net, walling off the rogue AIs of the Old Net behind the Blackwall and carrying on in a corporate hellscape.

I wonder if we'll see life imitating art on this one.

Network behaviour anomaly detection has been a thing for decades by now. But it's not AI, because it's mere neural networks and machine learning.

> humans refuse to spend effort on writing more formal specifications.

Code is formal specifications. It's just that programming language semantics has no domain related value of correctness. So you overflow an unsigned integer variable and got a lesser number which does violate addition rules. Or increase account A without decreasing account B, which violates accounting rules (and all kind of closed systems).

So it's up to the programmer to encode all of those rules. While we do have some tooling like type checking and memory safety, they are nowhere near enough. And encoding such thing in a language like Lean or TLA is brittle in terms of maintenance (which you do need to do as the real world evolves).

But AI "is really good, but not that good..."

AI's spottiness is exactly what describes this era. AI can create amazing code/image/video/article exception XYZ really annoying qualities that somewhat hobbles its ability to be autonomously.

And sure, maybe those problems will go away once we're at month X of the madness but we're still at month Y so are going to going remain for some moment.

The bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.

But that part is the key point. We've had many years of networks building inherently insecure structures with security tacked as feature and the primary security process being more ad-hoc stopgaps.

Even if you write a formal specification the operating system the code runs on is almost certainly not formally specified so you're just moving the trust boundary slightly further out to the OS & associated libraries.