In theory yes, but in practice they usually have the speaker up far higher than they are speaking themselves so we do only hear one side clearly.

I think the high distractability is a trifecta of volume, non-naturallness of the sound (compression etc: feeling out of place in the space) and this point.