How should one judge a writer if not by the body of their work?

You didn't judge the writer by their body of work. You judged the author by which works are famous.

If I write a cautionary tale about the seductive evils of fascism,

with an unreliable narrator who's a cog in the evil machine, and obviously deluded about it,

and deeply unpleasant detailed descriptions of the awful cruelty perpetuated by the nazi regime,

and some fascists really like my book, because detailed descriptions of awful nazi cruelty are their jam, and they really identify with my evil, unreliable, deluded narrator

and a lot of people haven't read my book, but they know the kind of person who like my book - fascists

should I be judged by the popular reception of my work?