It's similar to the distinction between a driver having a conversation with a passenger in a vehicle vs. the same driver having a phone call, even in a hands-free / speakerphone mode.

The passenger will be far more aware of context and circumstances, including traffic or other hazards, and will generally adapt to those surroundings. The remote party simply has no access to those cues.

(And yes, some passengers may be oblivious, for various reasons, including but not limited to children. I'm discussing the general case.)