I've never had members of other religions get upset about me wishing them a Merry Christmas, so no I'm not worried about it. If Jewish people started whining to me about it or something then the equation might change, but in fact those that I know are usually more perfunctory about wishing me a Merry Christmas than I am to them.

As to privilege.. whatever man. The simple fact is that Christians are the predominant religion where I live and they are touchy about this matter in a way other people aren't, so why should I go against the flow? Just to spite people I feel no real animosity against? To undermine a holiday I enjoy? Literally why.