I've been trying codex and claude code for the past month or so. Here's the workflow that I've ended up with for making significant changes.

- Define the data structures in the code yourself. Add comments on what each struct/enum/field does.

- Write the definitions of any classes/traits/functions/interfaces that you will add or change. Either leave the implementations empty or write them yourself if they end up being small or important enough to write by hand (or with AI/IDE autocompletion).

- Write the signatures of the tests with a comment on what it's verifying. Ideally you would write the tests yourself, specially if they are short, but you can leave them empty.

- Then at this point you involve the agent and tell it to plan how to complete the changes without barely having to specify anything in the prompt. Then execute the plan and ask the agent to iterate until all tests and lints are green.

- Go through the agent's changes and perform clean up. Usually it's just nitpicks and changes to conform to my specific style.

If the change is small enough, I find that I can complete this with just copilot in about the same amount of time it would take to write an ambiguous prompt. If the change is bigger, I can either have the agent do it all or do the fun stuff myself and task the agent with finishing the boring stuff.

So I would agree with the title and the gist of the post but for different reasons.

Example of a large change using that strategy: https://github.com/trane-project/trane/commit/d5d95cfd331c30...

Don't you also need to specify the error-cases at each stage and at what level of the system you would like to handle them (Log away, throw ever more up, Inform others, create Tasks, etc.)?

I found that to be really vital for good code. https://fsharpforfunandprofit.com/rop/

My twist on this is to first vibe code the solution with the aim of immediately replacing it.

I’ve found that two to three iterations with various prompts or different models will often yield a surprising solution or some aspect I hadn’t thought of or didn’t know about.

Then I throw away most or all of the code and follow your process, but with care to keep the good ideas from the LLMs, if any.

I've been working with coding LLMs for almost a year. Here's what I've found works best:

- Do a brainstorming session with the LLM about your idea. Flesh out the major points of the product, who the stakeholders are, what their motivations and goals are, what their pain points are. Research potential competitors. Find out what people are saying about them, especially the complaints.

- Build a high level design document with the LLM. Go through user workflows and scenarios to discern what kinds of data are needed, and at what scale.

- Do more market research to see how novel this approach is. Figure out what other approaches are used, and how successful they are. Get user pain points with each approach if you can. Then revisit your high level design.

- Start a technical design document with the LLM. Figure out who the actors of the system are, what their roles are in the system, and what kinds of data they'll need in order to do their job.

- Research the technologies that could help you build the system. Find out how popular they are, how reliable they are, how friendly they are (documentation, error messaging, support, etc), their long-term track record, etc. These go into a research document.

- Decide based on the research which technologies match your system best. Start a technical document with the LLM. Go through the user scenarios and see how the technologies fit.

- Decide on the data structures and flows through the system. Caching, load balancing, reliability, throughput requirements at the scale you plan to reach for your MVP and slightly beyond. Some UX requirements at this point are good as well.

- Start to flesh out your interfaces, both user and machine. Prototype some ideas and see how well they work.

- Circle back to research and design based on your findings. Iterate a few times and update the documents as you go using your LLM. Try to find ways to break it.

- Once you're happy with your design, build an architecture document that shows how the whole system will concretely fit together.

- Build an implementation plan. Run it through multiple critique rounds. Try to find ways to break it.

- Now you're at the threshold where changes get harder. Build the implementation piece by piece, checking to make sure they work as expected. This can be done quickly with multiple LLMs in parallel. Expect that the pieces won't fit and you'll need to rethink a lot of your assumptions. Code will change a LOT, so don't waste much time making it nice. You should have unit and integration tests and possibly e2e tests, which are cheap for the LLM to maintain, even if a lot of them suck.

- Depending on how the initial implementation went, decide whether to keep the codebase and refine it, or start the implementation over using the old codebase for lessons learned.

Basically, more of the same of what we've been doing for decades, just with faster tools.