In UK English ginger isn't exactly a color either. It's not like yellow or orange. It's a state of being.

I don't know why this is being downvoted. Does "ginger-colored" sound right to you? It doesn't quite work.

It's true. A person is ginger, but nobody ever says things are ginger coloured. Ginger isn't even really one colour - it's a spectrum of orange-containing hair colours.