I don't think they were ever popular. It was always TeamSpeak.

vent was massively popular. basshunter even has a chart-topping song about it. "Vi sitter i Ventrilo och spelar DotA" or "We're sitting in Ventrilo, playing DotA"

mumble, admittedly, not so much.

While I disagree, it's beside your point, because Mumble and Ventrilo also used the term "server." All three also required installing a client application, which Discord did not (my recollection is that Discord was the first major WebRTC application, and that made it a killer feature in itself at the time).