This comment chain is talking about people using speakerphone, though, meaning they hear both sides of the conversation

In theory yes, but in practice they usually have the speaker up far higher than they are speaking themselves so we do only hear one side clearly.

I think the high distractability is a trifecta of volume, non-naturallness of the sound (compression etc: feeling out of place in the space) and this point.