I agree with you. "Dark Matter" (and "Dark Energy") are colorful (colorless?) names that I think helped these theories diffuse into the popular consciousness at a time when popular interest in science was at a high-water mark (remember when "chaos theory" was fashionable?). As I mentioned in another comment recently (it feels like a "Dark Matter" or "Dark Energy" headline trends on HN almost every day), this coded these theories as "exotic" or "weird" as you say, and invited speculation about Dark Matter and even an urge to overturn it among laypeople who equated "exotic" with "tendentious." But, as you suggest, personally I don't regard Dark Matter as all that exotic. We already know about some species of "dark matter": the neutrino is one, and before that there was the neutron. Oh, well. I suppose there will be another episode on HN in a day or so.

Had we named it "invisible matter", perhaps not as much controversy would surround it.