Prior to pasteurization, people died or became ill on a regular basis: that’s one reason why Louis Pasteur is a household name two centuries after his birth. Even as a relatively niche product it’s still responsible for a significant percentage of the dairy-related deaths and severe illnesses.

The confound is related to what you mentioned: historically few people were shipping milk very far. If you’re getting it shortly after milking your family’s cows, you don’t have the same kind of trust commitment that most modern consumers are making and you don’t have the profit motive conflicting with food safety, either.