Wait; So you say you outsource development to a LLM fully knowing that it will not write sound/deterministic code and your quality requirement are your test cases, right?

I further suppose you are not writing the test cases in its whole by hand. But you try to specify them before you hand out the development task to the LLM, right?

It sounds like you just introduces a new team member which is not trustworthy yet. Normally you would review each change of him and explain how to improve hisself and the code. But that's not possible to a fixed-state LLM. That sounds exhausting.

If your Markdown Files should aim at improving the LLM contributions, you are again stuck with the fact that it did not follow in the first place.

So I conclude: You pay for an Intern who is not trustworthy and pay additional input token on Markdown Files to still no be certain about future contributions.

I just don't grasp how we as engineers are accepting this and integrate it into our craft. And: Everybody using External LLM Services, possibly providing the entire project as context, is allowed to let the source code of your company be leaked to some third party. I hope that party is trustworthy and does not have a track record of copyright infridgement. Because this would be fairly naive and reason to be fired. So we as Engineers knowing the implications should therefore point to these issues at the correct management level.

At least that's what I am doing shrugs