I think it's more that it's implicitly derogatory to other men. People get annoyed when you speak for them, sort of deal.

I'm a male, I don't write notes on my clothing, and I found the whole comment hilarious

It's possible some folks have thin skin.