I can't deprive someone of their right to use them, but I can refuse to interact with someone who's wearing them. This seems like a fair natural consequence. Feel free to wear them, but I won't speak to you when you do.