I can’t help but notice the items of increasing price / cost are not optional / discretionary purchases. Price can increase and not affect demand. Whereas the items decreasing in price are all subject intense competition and price sensitivity.

That's not how it works: Most things that people call non-discretionary still have opportunities for market effects. Take food: You might have spikes for, say, beef, but what is necessary is just enough calories, not necessarily having them come from beef. Therefore, you can switch preferences, eat something else, and not be affected by the price increases. Poeple rarely do though, because the amount of money spent on food is significantly lower than historical, precisely due to agricultural productivity improvements. In the US, we also have to consider the effects of recent tariffs: When supply gets far more expensive, prices go up in a market, regardless of whether people must eat or not.

For housing, there's also significant location effects: One doesn't have to live in, say, Manhattan. People trade time for location, and then select the space they want. A whole lot of the space Americans use is completely optional. Go look at cities in Asia, or in Spain: You can have a city with an average density similar to NYC's Upper East side, but not even NYC comes close. That's not about a limitation of supply, but very specific policy choices.

It's similar in other mandatory things: In healthcare, the amount of things that are actually mandatory isn't that large, training to become a doctor is offered to far fewer people that would want the job, drugs can be handed long monopolies... It's not about non-discretionary, but mostly a regulatory problem. Same with American colleges, which waste an order of magnitude more money in what is shaped like an old luxury good. Anyone that has gone to a public university in continental Europe and to a US college can tell you it's a completely different good, and the American approach isn't all that focused on efficient education, as it's still shaped like a finishing school. And again, it's not necessary.

So I'd argue it's almost always regulation written to help certain incumbents, instead of inability of market forces to keep prices low even when it appears that a good is non-discretionary.

College is optional. Actually, if you're just looking for "a skilled job" trade school is a better bet than college now. But the only people actually saying that are conservative nut-jobs trying to fight a culture war against a balanced education. People send their kids to colleges for the same reason why they demand more car lanes instead of better buses and trains: it's a status symbol.

The thing about status symbols is that you are buying them to feel better than someone else. So they almost have to be scarce - and therefore expensive. That's the basic idea behind cost disease; scarce things in an economy of abundance become more expensive, not less.