For some reason I always saw it in reverse, that I had to pay to subsidise a set of channels I'm _not_ interested in for the one I am.

Chances are that's not what was happening unless you were watching the channels nobody else watches.

I haven't looked into cable pricing for a while but i remember a few of the contract disputes that caused some big channels to drop off big cable providers in the 2010s. The price-per-customer those channels were asking the cable companies were significant chunks of what a package would cost the customer (eg upwards for $1).

Meanwhile some of the less common ones were a few cents per customer.

That means that unless you weren't watching any of the $1+ ones, you were mostly actually "paying for what you're watching".