I like the idea of llms collaborating like this a lot; planning, critiquing, verifying, coding etc. I think that’s a very general and powerful approach. How did you end up with that structure and what did you try first? What are the downsides? How do the component agents communicate, just json?

The agents communicate through different paths. First, there's a "big boss" orchestrator that decides who speaks next. The outputs from all agents (including the code from the coding agent) is put into a shared context that each agent can draw from. Practically speaking, to make this happen we use AutoGen framework.

We slowly started building more and more agents. Everything we tried just worked (kinda amazing). We first started by trying to incorporate visual understanding via VLMs. Then we slowly added more and more agents, and the BlenderRAG gave a huge boost.