I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them.

I respectfully disagree. IME writing a good test suite is one of the areas where current LLMs perform worst. They certainly like to write plenty of tests but those tests aren't necessarily adding much value and they don't necessarily cover the important functionality.

Given how heavily LLMs and specifically agentic workflows still rely on producing something vaguely correct and then iteratively applying guard rails to reach something that actually works I humbly suggest that trusting a very substandard test suite to be one of those guard rails might be unwise.

▲

flir 2 hours ago | parent [-]

IMO it works better if you split the tests into the ones you care about, and the ones the LLM writes. I assume its tests help it write code with fewer errors, so I'm happy for it to keep doing that, but they're akin to CoT - an artifact of the process that I don't need to inspect in detail.

I realise this is heresy, but my conclusion is that unit tests aren't important any more. E2E testing on the other hand...

(I'm agreeing with you about the quality of LLM tests, I'm just trying to make lemonade out of all these LLMons).

	▲	Silhouette an hour ago \| parent [-]
		I admire your optimistic approach. :) Unfortunately I don't really share your optimism on this one. All those extra tests are churning through tokens and CI resources. These are far from free in a direct financial sense. I've also noticed some of the more recent models will try to do a lot of experimentation and ad-hoc testing by spinning up shells or running something like Python scripts direct from a command line. Every time they do that you either have to review whatever they propose to run carefully - which is slow and tedious work - or stick the tools in YOLO mode and hope that your agents don't accidentally wipe your entire filesystem, open your firewall for malware to take over your network, and perhaps launch a few nuclear missiles. I would respectfully disagree about the value of unit tests as well. The reason unit tests appear to have less value in the LLM/agent era is because current LLMs suck at decomposition and other architecture-level work. They just see a mass of code and change it as a mass of code. This probably results in orders of magnitude more code being written/changed than would have been necessary for a skilled developer working on a well-maintained code base to solve the original problem and that automated work also takes longer in wall time. Again this gets brushed under the carpet because a lot of people trust that LLMs are so good that their costs are negligible regardless of how much work they do. People used to say that about the Cloud transition as well because not needing so many in-house staff to manage your resources was going to be a huge saving. Today we have companies bringing in consultants because they literally can't even work out how much they're spending on AWS by themselves. Instead of having a few greybeards who could make a Linux server do anything we have a few increasinglygreybeards who know which combination of 17 AWS resources they need Terraform to spin up so they can write the web server equivalent of hello-world.c. Meet the new boss - same as the old boss. :)