This resonates with my experience of using LLMs to build tooling.

I have a repo with several libraries where i need error codes to be globally unique, as well as adhere to a set of prefixes attributed to each library. This was enforced by carefully reviewing any commits that touched the error code headers.

I’ve had a ticket open for years to write a tool to do this and the general idea of the tool’s architecture but never got around to implementing it.

I used the LLMs to research design alternatives (clang tools, tree sitter, etc) and eventually implement a tree sitter based python tool that: given a json config of the library prefixes, checks they all adhere and that there are no duplicate error codes within a library.

This would probably have taken me at least a few days to do on my own (or probably would just sit in the backlog forever), took about 3 hours.

The ROI on those 3 hours is immense. Runs in milliseconds. No capitalized instructions in AGENTS.md begging models to behave. And you can refine it anytime to cover more cases!