Validation is really hard and requires a great understanding of the entire relevant code. So to answer your question, they don’t know or asked the AI to do that too which has very obvious problems.

I find it hard to believe they would spend the effort and money making an AI do tasks like that, but are ALSO willing to spend the effort and time to properly validate the changes. I’m sure many do… but probably not a large percentage.

In my case it's usually a video game in question, not a life support system, so I'm not poring over every line of code.

I mainly look at:

- The overall architecture to make sure it's in line with what I want

- The points where it interacts with certain other systems I'm concerned about

And I test the feature myself in addition to the test cases. Sol has proven to be outstanding at good test cases and having a coherent view on overall architecture integration. I basically never think to myself "this entire implementation is slop, I need to rip it out and refactor it" anymore.

I suspect the model's capacity to translate requirements into contextually appropriate test cases, and insistence on thoroughly covering functionality in tests, is a large part of its strength.