Robert Moses (ab)used eminent domain. If somebody buys a piece of land and can't build on it, that's not the same problem. Most rules now in place encourage far lower and less walkable development. Even a massive chunk of Manhattan has height restrictions that are lower than what's currently built and grandfathered in.