Same experience. I've spent hours tweaking prompts, harnesses, skills and what not to have a simple (but non trivial) project completely managed by an agent.

The key constraint was keeping the quality bar high.

If I lower the quality bar and _only_ care about a working solution, this works great. And I dare not look at the implementation details if this is the outcome I care about.

But if I want a working solution _and_ have it done the right way, then I need to be in the loop at key points. Newer models, unfortunately, have made this problem worse. I will inevitably come back to some form of bad decisions, deviations or just slop. Left unattended, it snowballs into a pile of mess. At this point, I decide if I spend time fixing it or switch my mental model to not care about the quality.

The problem is worse when there's a team involved here. Fragmented tooling , prompts, harness, model causes more divergence. Especially in quality. Senior engineers become bottleneck as the quality enforcers.

I do like the revert rate metric that Posthog tracks. This can help incrementally build the tooling to a point it can be trusted and scales.