That is the wrong way to think about it. Many places have buildings where the structure costs less than the land it's on. You won't get rid of your car because of the land under it. The space required for a box of silverware is similar to a small pack of toilet paper. Shall we throw that out too?

In many such places people buy houses explicitly to tear them down and build something better.

We'd be seeing even more of that if zoning didn't restrict it.