When the damage accumulates across generations the natural selection has opportunity to weed out particularly harmful instances. You can get a feeling for how important avoiding the mitochondrial damage is and how hard it is to mitigate, by looking at how fiercely the reproductive process protects them from aging.

And with this you get a hundred times the damage (mutation) but still only the same amount of selection.