At the risk of being obvious, this seems set up for failure in the same way expecting a human to catch an automated car's mistakes is. Although I assume mistakes here probably don't matter very much.

This reminds me the issue with the old windows access control system.

If those prompts pop up constantly asking for elevated privileges, this is actually worse because it trains people to just reflexively allow elevation.

yes, mistakes are not a huge problem. they will become evident farther down the process and they happen now with the human only system. worst case is the LLM fails and they just have to do the manual work that they are doing now