We do say what we mean, it's just either carefully coded in mutual assured understatement, or buried under expletive-laden exaggeration.

Any native knows that "Interesting, but perhaps we should reconsider" means "You're an idiot and I don't understand how you ever learned to breathe."

The pinnacle is "Not bad", which can mean either deep approval or blistering contempt, depending on tone of voice.

It drives foreigners insane. But of course it's not our fault if they never learned English.