Why couldn't you express all those as test cases rather than instructions?

In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..."

I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc

You can express them as tests, but you also need a feedback mechanism that creates the rule that when the LLM generates some net new code or performs some refactor, that there are these CAPAs that it needs to cover with test cases.

The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures.

In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.

> Why couldn't you express all those as test cases rather than instructions?

How would prevent removing the test case and the code that it tests? That's a legitimate thing to do when you're modifying a codebase. My intuition is "corrective and preventative actions (CAPA)" is a level or two up from test cases.