It's like a variant of the halting problem though: given N agents in a review loop reviewing a codebase, will it ever terminate and say the thing is done and bug free?

It seems to me just from @codex review, given a codebase of any appreciable size, if you ask the agent to find things to fix, it'll find things to fix. And despite N agents agreeing that some code is ready to ship, I've still sat down to try it and nothing actually works as advertised.

So the need for better verification tools that are not AI is very urgent. Because the AI can be told to write a thing without mistakes, it does write code that compiles, N-agent AI review process fixes bugs and eventually the code is approved, and then run against an extensive suite of tests to prove certain functionality which all pass... and yet it still can be the case that nothing actually works in production.

> will it ever terminate and say the thing is done and bug free?

Will human reviewers, if you ask them to find more bugs? Will it be because there really are no bugs, or because they got tired and just can't see any?

How would you even figure out when the thing is done and is bug free? There have been bugs that have been in codebases for years and have only been discovered recently.

Would you trust static analysis tools that give you an all clear, when they themselves cannot encode the above desired state fully?

Seems like you'd need formal proofs or something, but at that point also for the frameworks and libraries, database, networking stack and the whole damn OS.

I'm all for good tools, just saying that we'll need a whole bunch of those.