>they'd rather pay less than pay more for luxuries they don't actually value very much

Basically. I have used a combination of miles and co-pays to upgrade to business trans-Pacific. But most of the time going from the east coast US to Europe (especially when I can do it without a red-eye to London), I end up thinking of all the nice stuff I could do with $5K at the cost of sort of a miserable flight.

It's not that I couldn't splurge but there are other things I'd generally prefer to splurge on.

One reason flights are more miserable is planes are more heavily loaded. Before deregulation, planes were usually ~50% full. Today in the US, it's about 84%.

This is directly correlated with airfares. Were planes as sparsely loaded now as they were then then fares would be correspondingly higher. But in a deregulated environment there's a very strong incentive for increased economic efficiency to keep the fares competitive.