And yet, colors themselves are arbitrarily chosen partitions of a spectrum.

Not exactly - there are very clear areas where everyone agrees the dividing line exists when you look a full spectrum map. Even most colorblind will agree with the areas in general (there are lots of specific color blind types but most will agree what area of the map is which colors even if you don't put any scale indications on the map)