There is a substantial difference between people talking amongst themselves and one person on a phone.

Humans are social animals, we tune out conversations easily. Half conversations are just one interrupting, attention-grabbing … jarring start … … after … … … … another. It’s a series of unpredictable spontaneous one-sided outbursts, behaviours that otherwise belong to disturbed individuals.

Listening to people in the phone is inherently more annoying, backed by decent research IIRC.