Discord VC works like this

Yep, this is the exact use case that voice channels are used for in the community-oriented Discord servers I'm in. With the important note that voice channels support not just voice but video, multi-person screensharing, text chat, and even built-in games, though I've never actually seen the last of those get used in practice. Voice channels are just passively there in the channel sidebar with a list of people currently hanging out there, and you can drop in and drop out as you like. You can also send individual invites to join you in VC to server members.