Cost per passenger mile for a bus containing 4 people is much higher than for 4 cars, and since road wear scales with the 4th power of axle loading, a bus causes about 1,000x more damage to the road than a car.

It seems like the best solution would be to implement registration fees to offset externalities like pollution and road wear, and have congestion fees for when the roads are busy (including for vehicles such as busses). Then let the price signal decide the optimal mix of vehicles on the road. I think if such a system were implemented, the biggest change in behavior would be people taking fewer car trips during peak traffic hours.