What size project an AI can complete on its own is IMHO not an interesting question. The AIs and harnesses are getting good enough that quite large things can be built, primarily as a function of cost.

The interesting question in my view is what is the largest system an AI can maintain over a long time horizon -- while keeping a coherent architecture that adapts and responds well to change. All the while remaining performant, scalable, and understandable to humans. I am unaware of what is the currently longest-lived vibe coded project but it cannot be more than a few years old.

So to my mind, we don't have very much real data yet.

I think reproducing existing software and building a new product are quite different tasks.

In a benchmark, there are usually clear tests and a correct reference. In actual product development, requirements are often unclear, and we do not always know what the correct result is.

Passing tests is also not enough to confirm security, maintainability, or operability. I would like to see a benchmark where an AI continues changing the same product for several months. It would be interesting to see whether the architecture remains understandable and safe after many changes.

isn't this a harness engineering problem at its core (and the human equivalent being a product org problem)?