The argument that I keep hearing that he was just a guy talking does not quite fly.

The most horrible people in history did not do any physical harm to other people themselves. Many were also very nice to hang out with and had lovely families. But they definitely inspired and ordered others to do unimaginably horrible acts.