That is the precise mechanism how this works: when you see a weak man, first apply a label, then express disgust against the label.

When challenged, make it a discussion about the label, why the label actually deserves the hate. (Not about why the weak man automatically deserves the label.)