Regression tests start to play a different role with LLMs.

On one hand, they give an LLM a short feedback loop to correct itself, and iterate fast when writing code. A human also uses it as a feedback loop, but we don't iterate as fast and don't handle big walls of conditions, so its effect is not as big.

On the other hand, LLM's ability to handle a big wall of if-conditions can backfire if it starts taking shortcuts and taking the tests-as-a-spec too literally, overfitting the solution, overly focusing on the given datapoints (conditions checked by tests) and missing the overall behavior shape that the tests intend to pin down. For humans, this is less of a concern because we are bad at big walls of if-conditions, and we'd rather try to see the original shape that the tests are pinning down than monkey-patch the solution to fit the individual points.

It's interesting to see how one balanced these two. In this case particularly. Maybe you could play around with separating the data you give an LLM into "training set" and "validation set", training set can be seen fully, but validation set is hidden and is only queried when the solution is deemed ready. Say, training set = original source code + half of the tests; LLM uses that for quick feedback loop. And validation set = the remaining half of the tests; test code is not shown to the LLM and run only when the LLM says it's done to catch potential overfitting of the resulting solution over training set.

To me, the credibility of a solution like that would depend on what methodology the authors used. If they just let the LLM see all tests, I'd be skeptical (albeit unable to point out specific bugs due to the volume of work and LLM's ability to make bad things look trustworthy). The good thing is, real-life use will add new, unseen before datapoints for testing — so validation set will build up with time. Really curious to see how it will work.