I'd like to see this problem solved in such a way that your group memberships are transitively applied to your access to agents. In other words: you shouldn’t be allowed to talk to an agent at all if, at a given point in time, that agent is or was in a group that you are not or were not in at that time. Does that make sense? I don’t know how practical this restriction is, but letting the AI manage such permissions on its own seems reckless to me.

These kinds of rules never survive real world tests. For example if I create a private group just for myself and auto invite every agent in the company into it, does that mean no one can now use any of them?