You are doing something wrong.

I am routinely using swarms of agents, with complex sequencing between them and agents watching other agents, and then triggering forks and/or new design cycles. My normal flow is to have a few design tasks open per codebase during the day, with each design agent aware of the other sessions, and having an unattended nightly build which sometimes takes 8hrs and also requires coordination for tests, merges, and so on.