Raw milk isn't dangerous. People drank it for centuries before pasteurization, and some still drink it today. They were fine. I myself grew up drinking raw milk (grew up on a dairy farm), as did a ton of other people I know, and not one ever got sick.

You don't need to drink it if you don't want to, but there's no reason to deny others the same choice.

Prior to pasteurization, people died or became ill on a regular basis: that’s one reason why Louis Pasteur is a household name two centuries after his birth. Even as a relatively niche product it’s still responsible for a significant percentage of the dairy-related deaths and severe illnesses.

The confound is related to what you mentioned: historically few people were shipping milk very far. If you’re getting it shortly after milking your family’s cows, you don’t have the same kind of trust commitment that most modern consumers are making and you don’t have the profit motive conflicting with food safety, either.