It seems a poor analogy, since it's impossible not to be on the spectrum somewhere, even if it's #000000.

This is a misconception I see pop up frequently online. In terms of the color spectrum, there are plenty of things—even things that have qualities in common with color—that aren't on the color spectrum. And while there are colors outside of what humans can see, we generally use it not to refer to the entire electromagnetic spectrum, but only to the subset that makes up light visible to human eyes.

Likewise, when we talk about the "autism spectrum," we're not including every exhibition of traits associated with autism. You can have some traits associated with autism without being "on the spectrum."

Also, perhaps as importantly, "spectrum" isn't a term that generally applies only to color, or even electromagnetism.