To me, this comes down to verifiability. How do we measure the quality of what an agent is doing on our codebase rather than simply measuring task accomplishment?

> Verifying quality is orders of magnitude harder than "did the tests pass"

Agree that agentic grading is the future here. Cognition's Frontier Code is probably the best large public benchmark at this. You attribute agent quality issues to RLVR's binary pass/fail, however I wouldn't be surprised if labs are already supplementing that with rubrics as rewards to train more 'tasteful' models like Fable.

What can a practitioner do? I think there's promise in turning the optimization machine to the harness itself - building out a representative dataset of tasks on your repo, grading agent quality on them across various configurations, and optimizing [AGENTS.md / SKILLS.md / workflow / model / harness / tools] on that signal. High quality grading is still very hard, but it's more tractable at smaller, repo-level scale, and you can afford slower, more expensive verification for each task. You only need it to be right about your codebase's standards.

> In fact, it's not hard to imagine that if a model could reliably tell good code from bad, it might have written the good version to begin with

Pushing back slightly - detecting slop and discriminating quality is easier than generating it (why code review is so effective), and why grading is viable at repo eval scale even if it's much harder at RL scale.

Everyone is flying blind. For example, I am genuinely interested in trying HumanLayer, but would likely want some harder evidence (beyond anecdotes) that it's actually making my agents more effective before rolling out to an enterprise team.

I'm building this harness optimization loop @ https://stet.sh if curious