At least in my area, the price of the land is a huge fraction of the value of a house.

Imagine a house-sized plot of land costs $250,000. That means a builder can offer buyers a $20,000 building for $270,000 or a $150,000 building for $400,000. And in the latter case, the buyer gets 7.5x the building for 1.5x the cost.