> Seriously, can someone explain to me the actual experienced difference between 2 people having a conversation, and 1 person having a conversation on loudspeaker?

Loudspeaker/speaker phone is a harsh, artificial sound which can be grating to hear. Two people having a conversation while physically present can also automatically adjust their volume, tone, and subject matter depending on the area around them. This often happens unconsciously and is affected by cultural and social norms i.e. some cultures (famously "latin" ones) are louder than others.

If the conversation is happening over loud speaker, the above does not occur. The person on the phone can not adjust their volume/conversation topics to react to the surroundings of the conversation, is solely dependent on the person holding the phone to modulate their volume or change subject matter. The person speaking also can not modulate their volume properly since they need to talk in a certain way to be intelligible over the phone.

I'd assume most people would get annoyed at 2 people loudly talking and arguing in an area where it's expected to be quieter, or even in public at all since the human brain is good at picking out speech among other sounds, since it would be distracting if not second hand embarrassing. But this happens significantly less frequently then the loudspeaker problem due to the aforementioned automatic speech adjustments.