There’s a trap here though: the LLMs if you let them will spend all their time maintaining that test suite, they can become quite rigid in the way they design them. So it’ll write some code, break a test, which breaks a contract so then it updates the test and the contract but updating the test also broke another contract… soon enough you’re spending all your tokens fixing contracts instead of writing code.

In a sense the LLM having a fast inner loop is its blessing and curse. A blessing because it gets feedback quickly, but a curse because it becomes naval gazing and cannot see the forest for the trees.

At least this is my experience with Sol maby other models behave differently.

Software engineering also applies to the test suite. So its management requires supervision.

Sounds like one of the teams I left before project completion 13 years ago.