The mistake is to direct your empathy only one way

Absolutely not. The value of empathy is not "have empathy because empathy is good", it is in the pro-social outcomes. Empathy is wasted, and counterproductively so, on the non-empathetic, especially on the powerful and destructive. It is a peace treaty, much like tolerance.