My theory is that the "house" consists of two things: the land and the building.

The land goes up in value when other people spend money. More retail is constructed nearby, transportation is improved, schools improve, jobs are created, etc. My land captures some of that value even though I paid nothing. That to me explains some of the fairness of property tax: the owner should contribute to the government services, such as schools and police, that help make the land appreciate in the first place.

Then there's the building. It's a wood box that sits out in the rain and rots. Water soaks in from the outside and pipes burst on the inside. Termites eat it and insects and vermin invade. Carpet and walls slowly degrade. HVAC systems wear out. Appliances break. Concrete breaks apart. Even on the land portion, plants die and need maintenance. (Trees are the only thing on a property that get better with time.) This building needs constant maintenance and I'm always spending money and time on it.

So I figure the land might go up slowly in value over time. I figure I'm lucky if the building appreciates at all after I consider the money I sink into it.

On paper my house is worth a lot more than when I bought it. But I don't know how much of that is nominal price change due to inflation.