I would also add that, once they do lay their hands on someone (or actively conspire to), I think it's perfectly fine to respond with heftier penalties if the motivation for it is Nazi ideology or other similar stuff. It's not a free speech matter at that point.

Absolutely. The situation where a couple of guys get into a relatively normal fight, and one where a guy seeks somebody out because of any reason are obviously very different - and should be treated differently under the law.

In this regard I also think hate crimes miss the mark, because is something like 'the knockout game' a hate crime? Probably not, but I think it should be treated in a similar fashion because it's essentially the same problem - of some ideology driving somebody to violence, as opposed to a normal personal conflict.