There is nothing preventing you from expanding your group chat roster. It is just that random strangers can't drop in; you have to add them.

You would have to sacrifice the privacy of your group if you wanted to support serendipitous membership growth. Do you want to be constantly reviewing membership requests? That's what Facebook groups look like. And you have little information to judge the requests by, since the profiles can be fake, especially today. And when complete strangers can join the group, the dynamics change.