Hey, Jeff I'm one of the co-founders. Totally agree for a small side project, having a single sub-agent that focuses on vibe testing is enough.

But I would say even for medium sized features, it's the same difference between asking your Codex to "review the code make no mistakes" and using a paid AI code review tool.

As your product surface area increases, you need fundamentally different agent architecture to maintain context over time and across bug bash sessions. The benchmark apps are probably among the simplest apps we test on, and even those show a massive perf. gain. For large companies with 1000s of important user flows, have to support different viewports, and need to work across mobile/web/desktop, asking Codex to test it all before a big launch is not scalable.

Even basic constraints like the # of browsers/devices you can run concurrently, the amount of reliable TPM you need, and how many agents can run on a single box become problematic. You also have to think about ktlo: how do you evaluate new model releases in a data driven way, how does agent become more efficient itself over time, etc.