No noteworthy impact at all. The NTP network has hundreds to thousands of redundant servers and hundreds of redundant reference clocks.

The network will route around the damage with no real effects. Maybe a few microseconds of jitter as you have to ask a more distant server for the time.