I found a study a while ago that measured the effect of defining personas, and the effect was significant, but not very big. I like defining roles because I think it makes setting boundaries a bit easier. When I assign the role of architect for brainstorming, I expect the model to be a bit less eager to immediately jump into implementation. I'll still tell it explicitly to not do that, so the effect is probably extremely small.

So far, I find it much more important to define task scope and boundaries. If I want to implement a non-trivial feature, I'll have one role for analyzing the problem and coming up with a high-level plan, and then another role for breaking that plan down into very small atomic steps. I'll then pass each step to an implementation role and give it both the high-level plan and the whole list of individual steps as context, while making it clear that the scope is only to implement that one specific step.

I've had very good results with this so far, and once the two main documents are done, I can automate this with a small orchestration script (that does not depend on an LLM and is completely deterministic) going through the list and passing each item to an implementation agent sequentially, even letting the agent create a commit message after every step so I can trace its work afterwards. I've had very clean long-running tasks this way with minimal need for fixing things afterwards. I can go to bed in the evening and launch it and wake up to a long list of commits.

With the new 6 dollar subscription by Z.ai which includes 120 prompts (around 2000 requests) every 5 hours, I can pretty much let this run without having to worry about exceeding my limits.