I haven't commonly experienced issues that would cause my router to lose access to ntp for extended periods of time, and in such cases you can just reset using physical button. Of course, TOTP should be optional to use so I am not too worried.