The thing is that that same argument can be used to justify just about anything. If the scum of the earth is in power, they will ignore whatever rights you thought you had put into the constitution anyway. We are seeing that now. And I am already cursing the day that we decided on the restrictions we currently have. The Trump administration is declaring with complete legality that Trump won the 2020 election and is punishing people who believe that. Right now they're not taking the direct route, but it's abundantly clear that government power is being used to punish people who say things that Trump doesn't like.

There is no way of listing rights on paper that can protect you if truly evil people get into power. But there are ways of listing rights on paper that can allow good people who believe in those rights to defend them in ways that involve preventing evil people from getting into power. Free speech is not a magic bullet in either direction.