Why should all of society pay for these externalities? If some people manage to improve their energy supply and don't require dirty fuels, why should they be forced to subsidize those who won't?

Taxing the carbon at the source is simply correctly pricing it, and because it makes it impossible to shift the externalities away from the producer it fixes the accounting problem that falsely makes fossil fuels appear cheap.