Addendum: "I could care less" is a perfectly natural and recognizable idiom in some circles. To someone unfamiliar, it can seem strange, but that's true of many idioms.

The objections to it, though, fall broadly into two categories: ignorance, and bigotry. The former becomes the latter when someone refuses to recognize their ignorance, and doubles down on it.