If you made a list of all the people you hate with their pictures, should government regulators be involved with where you keep it and how you safeguard it? I don't feel like I would care even if I was on the list.

In the EU, that list would be regulated under GDPR. Just as any other list with people on it. You would be allowed to have it, but storing it insecurely or giving people who don't need it access to it would be a violation