But why would this ever happen? Why would the owners of land, construction material and machinery give those up for free to the average Joe?

Right now, even an average citizen born in poverty can acquire wealth from his labor. That is basically the only mechanism that prevents limitless accumulation of wealth: rich people still need workers to get things done.

If you replace the workers with AI, there is no remaining incentive for wealth to "trickle down" or get redistributed. This is not desirable.