If your ideology involves hurting people, to a point, you should be left alone by the government. but by that same principle, others should be left alone too when they show similar levels of hostility or worse.

You shouldn't lose your fundamental human rights, up to a point, that point is when you start believing and spreading a belief that other people should lose their fundamental human rights.

If only it were that simple. People seem to have to constantly argue about what human rights actually are. They use free speech to argue for all kinds of censorship for example. Censoring and harassing people who think differently will rarely do more than make them angry. Philosophers even centuries ago figured out that free speech is a most essential freedom and must be nearly unlimited in a free society, but it's a constant struggle to maintain this ideal in the face of the fearful and easily offended.