In many such places people buy houses explicitly to tear them down and build something better.

We'd be seeing even more of that if zoning didn't restrict it.