I'm about to release some tooling that's been very effective for me.

Essentially, there's a few ways LLMs write "bad" code that is different from how people write "bad" code.

We've got pretty good tooling to catch the ways people write bad code - it just happens to be much easier to do with static analysis (and is less noise prone).

The ways LLMs write bad code is typically 1) bad architecture - hard to detect in the ways that are really important, 2) unnecessary state and control flow (and decisions based on state), 3) bad / inadequate tests.

Methods to detect these problems have existed for ages, but they've never caught on because it's typically too difficult to tune them to have high signal / noise for humans, and AFAIK - no one else tried putting them all together and seeing how LLMs work with it.

LLMs are great at sorting through signal / noise -> so you can help surface potential issues with metrics that would be too noisy for humans, but seems to work pretty well for LLMs to find the source of architectural problems and design better solutions (from my experience - may be biased, I built the tooling to literally solve this problem for the main project I'm working on).

Can you elaborate on the tests thing? I'm new to testing, and AI agents recently voluntarily added an large number of tests to one of my projects. I've been learning a lot by reading them, and it seems like a great habit to develop.

But they're also writing some very strange code and some very strange tests. I don't know what good practices look like here, so when something looks strange to me I can't trust my own judgment, whether it's actually smelly or just a pattern I'm not used to yet.

--

On a side note, I recently had an agent implement a major architectural change. It turned out to have done it completely backwards, in a way that was pointless. (Improved nothing and actively made things worse.) However it had supplied generous tests for the new code, and of course all the tests passed...

So it had "proven the correctness" of something which was completely incorrect.

I later realized that even formal verification would not have prevented this. It would have just written a mathematical proof that the wrong code was correct.

You forgot to include link to your tool.