| |
| ▲ | cornholio 2 days ago | parent | next [-] | | The disater plan is to have a few dozens stratum 1 servers spread around the world, each connected to a distinct primary atomic clock, so that a catastrophic disaster needs to take down the global internet itself for all servers to become unreachable. The failure of a single such server is far from a disaster. | | |
| ▲ | sounds 2 days ago | parent | next [-] | | For those of us near Boulder, it's urgent. But the stratum 1 time servers can shrug and route around the damage. | |
| ▲ | Lammy a day ago | parent | prev [-] | | And the disaster plan for the disaster plan is to realize that it isn't that important at the human-level to have a clock meticulously set to correspond to other meticulously-set clocks, and that every attempt to force rigid timekeeping on humans is to try to make humans work more like machines rather than to make machines work more like humans. | | |
| ▲ | basilgohar a day ago | parent | next [-] | | I really, really can't get behind this sentiment. Having a reliable, accurate time keeping mechanism doesn't seem like an outlandish issue to want to maintain. Timekeeping has been an important mechanism for humans for as long as recorded history. I don't understand the wisdom of shooting down establishing systems to make that better, even if the direct applicability to a single human's life is remote. We are all part of a huge, interconnected system whether we like it or not, and accurate, synchronized timekeeping across the world does not sound nefarious to me. | | |
| ▲ | Lammy a day ago | parent [-] | | > Timekeeping has been an important mechanism for humans for as long as recorded history. And for 99% of that history, Noon was when the sun was half-way through its daily arc at whatever point on Earth one happened to inhabit. The ownership class are the ones who invented things like time zones to stop their trains from running in to each other, and NTP is just the latest and most-pervasive-and-invasive evolution of that same inhuman mindset. From a privacy point of view, constant NTP requests are right up there alongside weather apps and software telemetry for “things which announce everyone's computers to the global spy apparatus”, feeding the Palantirs of the world to be able to directly locate you as an individual if need be. | | |
| ▲ | rogerrogerr a day ago | parent | next [-] | | > The ownership class are the ones who invented things like time zones to stop their trains from running in to each other In a world where this didn't happen, your comment would most likely read: > The ownership class are the ones who had such indifference toward the lives of the lower class passengers that they didn't bother stopping their trains from running into each other. | |
| ▲ | kevinh456 a day ago | parent | prev [-] | | Tell me how you feel about DST. |
|
| |
| ▲ | zettabomb a day ago | parent | prev | next [-] | | Far more things rely on reliable and accurate time-keeping than just being on time to work. Timekeeping is vitally important (even if it's not readily visible) to lots of critical infrastructure worldwide. | | |
| ▲ | flaminHotSpeedo a day ago | parent [-] | | Actually, it's really important to me to have a network of atomic clocks available to verify the times I clock in and out, I want to make sure I get paid for an accurate duration of time down to the nanosecond |
| |
| ▲ | Brian_K_White a day ago | parent | prev | next [-] | | This is like the kid in school who doesn't think they should have to learn algebra since they think they will never use it. | |
| ▲ | alsetmusic a day ago | parent | prev | next [-] | | "Wearing a watch is like being handcuffed to time." -My Friend Andy | |
| ▲ | henning a day ago | parent | prev | next [-] | | And as things fell apart / Nobody paid much attention | |
| ▲ | zzzeek a day ago | parent | prev [-] | | oh....no, not really, no, the world needs GPS, so, yeah. this is not like scrooge mcduck telling you to be at work on time. scrooge still has a windup watch |
|
| |
| ▲ | toast0 2 days ago | parent | prev | next [-] | | If access to the site is unsafe and thus the site is closed; not having access seems reasonable. Time services are available from other locations. That's the disaster plan. I'm sure there will be some negative consequences from this downtime, especially if all the Boulder reference time sources lose power, but disaster plans mitigate negative consequences, they can't eliminate them. Utility power fails, automatic transfer switches fail, backup generators fail, building fires happen, etc. Sometimes the system has to be shut down. | | | |
| ▲ | ssl-3 2 days ago | parent | prev | next [-] | | Maybe this is the disaster plan: There's not a smouldering hole where NIST's Boulder facility used to be, and it will be operational again soon enough. There's no present need for important hard-to-replace sciencey-dudes to go into the shop (which is probably both cold, and dark, and may have other problems that make it unsafe: it's deliberately closed) to futz around with the the time machines. We still have other NTP clocks. Spooky-accurate clocks that the public can get to, even, like just up the road at NIST in Fort Collins (where WWVB lives, and which is currently up), and in Maryland. This is just one set. And beyond that, we've also got clocks in GPS satellites orbiting, and a whole world of low-stratum NTP servers that distribute that time on the network. (I have one such GPS-backed NTP server on the shelf behind me; there's not much to it.) And the orbital GPS clocks are controlled by the US Navy, not NIST. So there's redundancy in distribution, and also control, and some of the clocks aren't even on the Earth. Some people may be bit by this if their systems rely on only one NTP server, or only on the subset of them that are down. And if we're following section 3.2 of RFC 8633 and using multiple diverse NTP sources for our important stuff, then this event (while certainly interesting!) is not presently an issue at all. | | |
| ▲ | Balgair 2 days ago | parent | next [-] | | There are many backup clocks/clusters that NIST uses as redundancies all around Boulder too, no need to even go up to Fort Collins. As in, NIST has fiber to a few at CU and a few commercial companies, last I checked. They're used in cases just like this one. Fun facts about The clock: You can't put anything in the room or take anything out. That's how sensitive the clock is. The room is just filled with asbestos. The actual port for the actual clock, the little metal thingy that is going buzz, buzz, buzz with voltage every second on the dot? Yeah, that little port isn't actually hooked up to anything, as again, it's so sensitive (impedance matching). So they use the other ports on the card for actual data transfer to the rest of the world. They do the adjustments so it's all fine in the end. But you have to define something as the second, and that little unused port is it. You can take a few pictures in the cramped little room, but you can't linger, as again, just your extra mass and gravity affects things fairly quickly. If there are more questions about time and timekeeping in general, go ahead and ask, though I'll probably get back to them a bit later today. | | |
| ▲ | jrronimo 2 days ago | parent | next [-] | | I'm the Manager of the Computing group at JILA at CU, where utcnist*.colorado.edu used to be housed. Those machines were, for years, consistently the highest bandwidth usage computers on campus. Unfortunately, the HP cesium clock that backed the utcnist systems failed a few weeks ago, so they're offline. I believe the plan is to decommission those servers anyway - NIST doesn't even list them on the NTP status page anymore, and Judah Levine has retired (though he still comes in frequently). Judah told me in the past that the typical plan in this situation is that you reference a spare HP clock with the clock at NIST, then drive it over to JILA backed by some sort of battery and put it in the rack, then send in the broken one for refurb (~$20k-$40k; new box is closer to $75k). The same is true for the WWVB station, should its clocks fail. There is fiber that connects NIST to CU (it's part of the BRAN - Boulder Research and Administration Network). Typically that's used when comparing some of the new clocks at JILA (like Jun Ye's strontium clock) to NIST's reference. Fun fact: Some years back the group was noticing loss due to the fiber couplers in various closets between JILA & NIST... so they went to the closets and directly spliced the fibers to each other. It's now one single strand of fiber between JILA & NIST Boulder. That fiber wasn't connected to the clock that backed utcnist though. utcnist's clock was a commercial cesium clock box from HP that was also fed by GPS. This setup was not particularly sensitive to people being in the room or anything. Another fun fact: utcnist3 was an FPGA developed in-house to respond to NTP traffic. Super cool project, though I didn't have anything to do with it, haha. | | |
| ▲ | BrandoElFollito a day ago | parent | next [-] | | I love these comments on HN. Now if the (otherwise very kind) guy in charge of the Bureau international des poids et mesures at Sèvres who did not let me have a look at the refrerence for the kilogram and meter could change his mind, I would appreciate. For a physicist this is kinda like a cathedral. | |
| ▲ | mh- 2 days ago | parent | prev [-] | | This is super cool (and the kind of comment that I love reading on HN!), thanks for sharing. |
| |
| ▲ | Workaccount2 2 days ago | parent | prev | next [-] | | >The actual port for the actual clock, the little metal thingy that is going buzz, buzz, buzz with voltage every second on the dot? Yeah, that little port isn't actually hooked up to anything, as again, it's so sensitive (impedance matching). So they use the other ports on the card for actual data transfer to the rest of the world. Can you restate this part in full technical jargon along with more detail? I'm having a hard time following it | | |
| ▲ | pas 2 days ago | parent | next [-] | | it sounds like there's some kind of coupling, inductive or optoelectronic but yes, I also want the juicy details! so this is the clock https://en.wikipedia.org/wiki/NIST-F1 or this https://en.wikipedia.org/wiki/NIST-F2 or there's already F4 too, but it doesn't have a Wikipedia article yet https://www.nist.gov/news-events/news/2025/04/new-atomic-fou... but maybe they are talking about the new non-microwave clocks that use Ytterbium-based optical combs ... or about the Aluminum ion clock https://www.nist.gov/news-events/news/2025/07/nist-ion-clock... mind blown | |
| ▲ | lisper 2 days ago | parent | prev [-] | | These claims are bullshit. You can get technical details about the clock first-hand at this link: https://www.nist.gov/pml/time-and-frequency-division/time-re... and you can see a photo of the actual installation here: https://www.denver7.com/news/front-range/boulder/new-atomic-... As you can see, the room is clearly not filled with asbestos. Furthermore, the claim is absurd on its face. Asbestos was banned in the U.S. in March 2024 [1] and the clock was commissioned in May 2025. The rest of the claims are equally questionable. For example: > The actual port for the actual clock ... isn't actually hooked up to anything ... they use the other ports on the card for actual data transfer It's hard to make heads or tails of this, but if you read the technical description of the clock you will see that by the time you get to anything in the system that could reasonably be described as a "card" with "ports" you are so far from the business end of the clock that nothing you do could plausibly have an impact on its operation. > You can't put anything in the room or take anything out. That's how sensitive the clock is. This claim is also easily debunked using the formula for gravitational time dilation [2]. The accuracy of the clock is ~10^-16. Calculating the mass of an object 1m away from the clock that would produce this effect is left as an exercise, but it's a lot more than the mass of a human. To get a rough idea, the relativistic time dilation on the surface of the earth is <100 μs/day [3]. That is huge by atomic clock standards, but that is the result of 10^24kg of mass. A human is 20 orders of magnitude lighter. --- [1] https://www.mesotheliomahope.com/legal/legislation/asbestos-... [2] https://en.wikipedia.org/wiki/Gravitational_time_dilation [3] https://tf.nist.gov/general/pdf/3278.pdf | | |
| ▲ | zamadatix 2 days ago | parent [-] | | Agreed the stated claims don't seem to make much sense. Using a point mass 1 meter away and (G*M)/(r*c^2) I'm getting that you'd have to stand next to the clock for ~61 years to cause a time dilation due to gravity exceeding 10^-16 seconds. |
|
| |
| ▲ | ronjakoi 2 days ago | parent | prev [-] | | Will the time it takes you to answer depend on the mass of the person asking? |
| |
| ▲ | mcculley 2 days ago | parent | prev [-] | | > And the orbital GPS clocks are controlled by the US Navy, not NIST. I thought it was US Space Force / Air Force. Was the Navy previously or currently involved? | | |
| |
| ▲ | TylerE 2 days ago | parent | prev [-] | | Step One of most disaster plans is not to create a second emergency. | | |
| ▲ | ronjakoi 2 days ago | parent | next [-] | | Or even just a microsecond emergency. | | | |
| ▲ | amelius 2 days ago | parent | prev [-] | | But can't NTP server downtime cause a disaster? | | |
| ▲ | Vosporos 2 days ago | parent | next [-] | | One (amongst many) NTP server going down creates less issues than an NTP server spreading wrong time. | | |
| ▲ | macintux 2 days ago | parent | next [-] | | General rule of thumb: a misbehaving/slow server in any well-architected distributed system is vastly worse than a dead server. | | |
| ▲ | xeonmc 2 days ago | parent [-] | | i.e. a gaslighting husband is vastly worse than a dead husband. |
| |
| ▲ | PunchyHamster 2 days ago | parent | prev [-] | | technically if you have 3 or more sources that would be caught; NTP protocol was designed for that eventuality | | |
| ▲ | throw0101c 2 days ago | parent | next [-] | | > technically if you have 3 or more sources that would be caught; NTP protocol was designed for that eventuality Either go with one clock in your NTPd/Chrony configuration, or ≥4. Yes, if you have 3 they can triangulate, but if one goes offline now you have 2 with no tie-breaker. If you have (at least) 4 servers, then one can go away and triangulation / sanity-checking can still occur with the 3 remaining. | | | |
| ▲ | da_chicken 2 days ago | parent | prev [-] | | Sure, but not needing a failure to cascade to yet another failsafe is still a good idea. After all, all software has bugs, and all networks have configuration errors. |
|
| |
| ▲ | idiotsecant 2 days ago | parent | prev [-] | | If your application is so critical that NTP timing loss causes disaster and your holdover fails in less than a day and you aren't generating your own via gps, you are incompetent, full stop | | |
| ▲ | geerlingguy 2 days ago | parent [-] | | And if things are that critical, you might have other references besides just GPS... | | |
|
|
|
|
| |
| ▲ | simojo 2 days ago | parent | next [-] | | I guess that explains why they had no qualms shutting down half of Boulder's power with a vague time horizon. After losing everything in my fridge, though, they finally turned it back on today. | | |
| ▲ | ssl-3 a day ago | parent | next [-] | | Indeed. Losing the contents of (lots of) fridges is cheaper, as a whole, than incidentally burning the countryside. We all ultimately pay for the result no matter what, so that seems like a reasonably-sensible bet. On the fridge itself: You may find that the contents are insured against power outages. As an anecdote, my (completely not-special) homeowner's insurance didn't protest at all about writing a check for the contents of my fridge and freezer when I asked about that, after my house was without power for a couple of weeks following the 2008 derecho. This rather small claim didn't affect my rate in any way that I could perceive. And to digress a bit: I have a chest freezer. These days I fill up the extra space in the freezer with water -- with "single-use" plastic containers (water bottles, milk jugs) that would normally be landfilled or recycled. This does a couple of things: On normal days, it increases thermal mass of the freezer, and that improves the cycle times for the compressor in ways that tend to make it happier over time. In the abnormal event of a long power outage, it also provides a source of ice that is chilled to 0F/18C that I can relocate into the fridge (or into a cooler, perhaps for transport), to keep cold stuff cold. It's not a long-term solution, but it'll help ensure that I've got a fairly normal supply of fresh food to eat for a couple of days if the power dips. And it's pretty low-effort on my part. I've probably spent nearly as much effort writing about this system here just now as I have on implementing it. | | |
| ▲ | scheme271 a day ago | parent [-] | | It's probably not worth it to go through your insurance for the loss of food and perishables in the fridge/freezer. It counts as a claim on your home insurance and can result in increased rates or even your insurer dropping coverage at the next renewal. | | |
| ▲ | ssl-3 21 hours ago | parent [-] | | Eh. It worked for me after that wind storm. As previously-stated: There was no rate increase that I could discern. I did neglect to mention that there was also no issue with renewal, but perhaps I should be more careful to always use absolute rote specificity and leave nothing to implication. They also sent over some folks with a tall ladder to have a look at the roof of this property that they insured, which was good since we had no means to visually inspect it from the ground. (The roof was fine.) Anecdotally, that phone call to the insurance company had no downside at all. It provided a roof inspection that I did not have ready means to perform on my own, and a relatively small amount of money (a couple of hundred bucks) that became very useful not just because of lost food, but also due to all of the other storm-related issues that we were not insured against. |
|
| |
| ▲ | bibimsz a day ago | parent | prev [-] | | they gave days of advanced warning they would do this. there was time to prepare. |
| |
| ▲ | tpoindex 2 days ago | parent | prev [-] | | For more background on the Marshal Fire of Dec. 2021: https://en.wikipedia.org/wiki/Marshall_Fire tl;dr - the fire destroyed over 1,000 homes, two deaths. The local electrical utility, Xcel, was found as a contributing cause from sparking power lines during a strong wind storm. As a result, electrical utilities now cut power to affected areas during strong winds. |
|
