"кофейного цвета брюки" is acceptable too.

I admit that. I also realize that tguvot is actually arguing in my favor, as he said that coffee color is distinct from brown, and therefore the inference is that they aren't synonymous. I would summarize that they are conceptually different, as "brown" is a real color, whereas "coffee color" is a marketing color.