Using a coding agent over days on a personal project. It has made me think

1. These llms are smart and dumb at the same time. They make a phenomenal contribution in such a short time and also do a really dumb change that no one asked for. They break working code in irrational ways. I’ve been asking them to add so many tests for all the functions I care about. This acts as a first guard rail when they trip over themselves. Excessive tests.

2. Having a compiler like Rust’s helps to catch all sorts of mines that the llms are happy to leave.

3. The LLMs don’t have a proper working memory. Their context is often cluttered. I find that curating that context (what is being done, what was tried, what is the technical goal, specific requests etc) in concise yet “relevant for the time” manner helps to get them to not mess up.

Perhaps important open source projects that choose to accept AI generated PRs can have such excessive test suites, and run the PRs through them first as a idiotic filter before manually reviewing what the change does.

Questions. I want to get into coding agents, so, out of curiosity: which one(s) did you use and how much money has it costed you? (Any metric is fine)