I've done something like that too, but I find it restricting.

I want back-and-forth, very approximately like when I do pair programming. The dividing line between what I do and what the AI does varies according to task and sometimes during the task, and is seldom clear at the start.

Then there's the work that wants a human to click buttons and decide whether something is a good and correct user experience. The AI does not have access to my display if I can avoid it.

Overall, the model you describe is one that's worked very well for me, but for some problems. An unsatisfyingly small set.