Indeed using the term "neurodiverse" only makes sense if you also want to include, for example, psychopaths (another form of neurodiversity) in the group that you want to describe.

I don’t see any reason not to include psychopathy. It’s not a synonym for evil: https://www.smithsonianmag.com/science-nature/the-neuroscien...