AFAIK - of those, collision is the only one which could plausibly make earth less habitable than a substantially terraformed mars.

The Sun will get hotter over time and expand, eventually descendants will have to decamp to elsewhere, like Mars.