> nobody cares if it went up or down in value

If only this were true, we wouldn't have homeowners constantly fighting against new development or construction. Many homeowners treat their primary residence as an investment.