I wrote a thing about this (https://www.klio.org/theory-of-llm-dev-skepticism/) back at the beginning of the year, in a totally different era of AI dev tooling. What you're describing is what I called there "bottom-up" programming.

Unfortunately, I don't have a lot of first-person advice for you; as I say in that essay, I was also a bottom-up programmer for a long time and was deeply skeptical of top-down approaches. What changed my opinion and my practice was just doing things for long enough that I started having a sense of where things were going to go before I did the exploratory work to find it out, and eventually the exploratory work was unnecessary and I could create a basically-correct top-down design early on. I don't know how you build that experience without actually getting in the reps (which has some implications for future generations of developers, if we have any).

That said, my second-person advice would be to use the agents for the design part of things, too. "Here's the problem, how would you design it?" and then have a conversation with them, push back on the things that seem wrong, imagine reviewing this as a finished design from a coworker and seeing where it looks right and where it doesn't. Ask why it designed things a certain way; suggest alternatives for it to evaluate and think about what it's weighing. Basically borrow the robot's experience to augment and build up yours.