> Don’t be tempted to skip the manual test because you think the automated test has you covered already! Almost every time I’ve done this myself I’ve quickly regretted it.

Seriously, this cannot be emphasized enough. Before LLMS when we were writing tests completely manually, manual testing made sense to me as the second step. However after playing around a lot with coding agents and LLMs, I fully agree this has flipped. Test it manually first! When you generate the tests it is extremely wise to ensure that the tests fail without the new code, and pass with it. You definitely need to review the test though, because it's remarkably easy to have the agent put something in there that makes it not a good test.

Just a couple days ago for example, Claude made a test pass by skipping authentication and leaving a brief comment informing that the authentication made the test flaky. It even threw a quick variable in there that enabled running or disabling flaky tests, and flaky tests were disabled by default! Had I not been doing a good review, I definitely would have missed it because it was cleverly subtle. I've also seen it test the wrong endpoint!