I think the point is they have to follow the rules of the road because they are allowed in the road. Pedestrians, wheelchairs, etc can go on the sidewalk and be safe from traffic (one hopes).

Though it depends on the state and in my experience there are typically some differences, such as bikes are required to share the lane.

Everyone using the road following the same set of rules makes their actions more predictable and thus safer (in theory)