oh dang that's not good. I've had the same phone number since 2006 so I didn't really think about it

But the phone number you have is not 100% in your control. I had AT&T flub something and I lost my number and they assigned me a new one (I was chanting my plan just after they did some merging with someone). Granted its unlikely but I would still use defense in depth and not have password reset be my only login method.