Your example is super amenable to vibing some tests. As an example, I’ve been able to ban `number` from representing a duration by walking the AST in a linter to fail if var or param names that look like the end in millis or ms or sec appear. This is largely good enough. If you see that “drifting” behavior appear more than once, you have enough to stop and force the agent to write some static verifiers that reject all but the option you want. For a closer example, we did this with zod schemes and their corresponding inferred types to be universally ZPascalCase and PascalCase instead of camelCaseSchema and CamelCase

Yes that's a great example! I think this type of "structural" harness engineering is what people need to be focusing on at this point in the journey. I think in almost all cases "agentic engineering" really is just "good software engineering practices enforced adversarially". Without guards, even the best models still behave like that Patrick Not My Wallet meme.

And to address your broader question, yes this is a form of RSI and to me a vastly superior approach to fine tuning since it allows adopting new model releases without throwing anything away while still having the same effect on improving adherence to local acceptance criteria.