Why do we care that much about the SNR of spoken words? Language is inherently quite redundant.

Redundancy is still "signal"

I don't think it's an argument of efficiency but rather the avoidance of noise.

The "ums" isn't redundant, it's not repeating or decorating the conversation. It's filler like static. Stops people from filling the gaps with their own thoughts