Most of that wealth is just a tag on physical things that would happen anyway, just with another tag.

This ignores that physical things can happen in many different ways, most of which are decided by those who have the money. The choices of the rich aren't preordained, so there's no reason to absolve them of the responsibility for their decisions.