It’s fine as an occasional stylistic choice, but using it repeatedly as a regular synonym for brown is a pragmatic and collocational error. The meaning is clear, but the wording is marked, and overuse makes the speech sound odd in everyday contexts.

coffee color won't be synonym for brown. it will be distinct color, just like strawberry, raspberry, straw, ruby, etc colors.