Yeah, we don't ban harmful speech, and we shouldn't. Harming people can be a legitimate and valuable consequence of speech.

We ban a subset of harmful speech with a direct link to clear harms, usually criminal.