If attacker would fool me at one website he will get that one account (possibly forever) and that's it. If it is a bank connected account, I can intervene and change email/account by writing a physical request to the bank for example, call the bank, do something. And likely it will be only a single bank account. But it may be even some unrelated account. Maybe it will be my Amazon account and all the attacker gets is some ebooks. Or Steam account. Or some email without important links. Etc.

Point is, the damage will be likely local to a single or a handful of accounts.

If all the accounts are protected by two factor on my phone and I lose it or it bricks, then I'm done. It will be a total mess with no paths to recover, except restarting literally everything from scratch.

I have Google Auth app on my phone and every few months I consider using it, but then reconsider and stay with passwords.