Is anyone exploring the (imo more practically useful today) space of using agents to put together better changes vs "more commits"?

Yes, I am, although not really in public yet. I use the pi harness, which is really easy to extend. I’m basically driving a deterministic state machine for each code ticket, which starts with refining a short ticket into a full problem description by interviewing me one question at a time, then converts that into a detailed plan with individual steps. Then it implements each step one by one using TDD, and each bit gets reviewed by an agent in a fresh context. So first tests are written, and they’re reviewed to ensure they completely cover the initial problem, and any problems are addressed. That goes round a loop till the review agent is happy, then it moves to implementation. Same thing, implementation is written, loop until the tests pass, then review and fix until the reviewer is happy. Each sub task gets its own commit. Then when all the tasks are done, there’s an overall review that I look at. Then if everyone is happy the commits get squashed and we move to manual testing. The agent comes up with a full list of manual tests to cover the change, sets up the test scenarios and tells me where to debug in the code while working through each test case so I understand what’s been implemented. So this is semi automated - I’m heavily involved at the initial refine stage, then I check the plan. The various implementation and review loops are mostly hands off, then I check the final review and do the manual testing obviously.

This is definitely much slower than something like Gas Town, but all the components are individually simple, the driver is a deterministic program, not an agent, and I end up carefully reviewing everything. The final code quality is very good. I generally have 2-4 changes like this ongoing at any one time in tmux sessions, and I just switch between them. At some point I might make a single dashboard with summaries of where the process is up to on each, and whether it needs my input, but right now I like the semi manual process.

I have a code quality analysis tool that I use to "un-slopify" AI code. It doesn't handle algorithms and code semantics, which are still the programmer's domain, but it does a pretty good job of forcing agents to dry out code, separate concerns, group code more intelligently and generally write decoupled quasi-functional code. It works quite well with the raph loop to deeply restructure codebases.

https://github.com/sibyllinesoft/valknut

> Is anyone exploring the (imo more practically useful today) space of using agents to put together better changes vs "more commits"?

That’s what I’ve been focused on the last few weeks with my own agent orchestrator. The actual orchestration bit was the easy part but the key is to make it self improving via “workflow reviewer” agents that can create new reviewers specializing in catching a specific set of antipatterns, like swallowing errors. Unfortunately I've found that what decides acceptable code quality is very dependent on project, organization, and even module (tests vs internal utilities vs production services) so prompt instructions like "don't swallow errors or use unwrap" make one part of the code better while another gets worse, creating a conflict for the LLM.

The problem is that model eval was already the hardest part of using LLMs and evaluating agents is even harder if not practically impossible. The toy benchmarks the AI companies have been using are laughably inadequate.

So far the best I’ve got is “reimplement MINPACK from scratch using their test suite” which can take days and has to be manually evaluated.

I think you massively underestimate the number of useful apps that are crud and a bit of business logic and styling. They’re useful, can genuinely take time to build, can be unique every time, and yet not brand new research projects.

What would be an example of something you think wouldn’t work with 5 or higher? Is there something about GPU programming that LLMs can’t handle?

Could you perhaps replace the VMs with bubblewrap instead?