Ntp at Nist Boulder Has Lost Power

Awful stuff.

I was just starting off in life (kid, girl, job, apartment, bank, debit card, bills) when I made a 63-cent error with my bank account. Which is to say: If all of the negative debits and all of the positive credits were summed, then the account would have been in the negative by 63 cents.

I screwed up. And in a fair and just world, I'd owe the bank some extra tithing for quite clearly having spent some money that I very definitely did not have. Maybe $20 for the error and $10 per day spent in the red, plus the 63 cents, or something along those lines.

But it wasn't that way. Because the transactions were processed in batches that were re-ordered to be highest-first, I discovered that I owed the bank a little more than $430.

At the time (around 25 years ago, now) that was an absolute mountain of money to me.

The banker at the branch I went into was unapologetic and crass about the fees.

Looking over my transactions, they said "You know how to do math, right? You should have known that the account was overdrawn, but you were just out spending money willy-nilly all over town anyway."

I replied with something like "I made a single error of 63 cents. The rest of what you said is an invention."

This went back and forth for a bit before I successfully managed to leave the building without any handcuffs, fire trucks, or ambulances becoming involved -- and I still owed them more than $430.

The lesson I learned was very simple: Seriously, fuck those guys.

($12 billion in 2024, huh? That's all? Maybe the no-fee fintechs are winning.)

Well, that's _this_ timeline. Other timelines never had an outage.

ugh. I have to replace my time rotors again - I thought regen was supposed to help with the wear while it improved range?

>"you’ll sooner or later end up dead."

What a defeatist attitude, I plan to live forever or die trying! /s

I’m quite certain you can approach it in any convenient manner

cheap microcontrollers use RC oscillators. If they only drift a few seconds a day, that would be an achievement by itself.

RC oscillator is poor enough that early days USB communication would fail if running on RC clock.

If NIST NTP goes down, the internet doesn’t go down. But atomic clocks drifting does upset many scientific experiments, which would effectively go down for the duration of the outage.

Mostly scientific stuff like astronomical observations — e.g. did this event observed at one telescope coincide with neutrinos detected at this other observatory.

Note I didn’t say they are more important than the Internet. That’s a value judgement in any case. I said that NIST level 0 NTp servers are more important to these use cases than they are to the Internet.

I think it wont be affected by this but on the top of my head:

- GPS

- industrial complex that synchronize operations (we could include trains)

- telecoms in general (so a level higher than the internet)

The ability for humankind to communicate across the entire globe at nearly 1/4 of the speed of light has drastically accelerated our technological advancement. There is no doubt that the internet is a HUGE addition to society.

It's not super important when compared to basic needs like plumbing, food, electricity, medical assistance and other silly things we take for granted but are heavily dependent on. We all saw what happened to hospitals during the early stages of the COVID pandemic; we had plenty of internet and electricity but were struggling on the medical part. That was quite bad... I'm not sure if it's any worse if an entire country/continent lost access to the Internet. Quite a lot of our core infrastructure components in society rely on this. And a fair bit of it relies on a common understanding of what time "now" is.

Actually interesting to think about what UTC actually means and there is seems to be no absolute source of truth [0]. I guess the worry is not that much about the NTP servers (for which people anyways should configure fail overs) but the clocks themselves.

[0] https://www.septentrio.com/en/learn-more/insights/how-gps-br...

And more importantly, the tighter the time bound, the higher the performance, so more accurate clocks easily pay for themselves in other saved infrastructure costs to service the same number of users.

I don't know about all hyperscalers, but I have knowledge of one of them that has a large enough fleet of atomic frequency standards to warrant dedicated engineering. Several dozen frequency standards at least, possibly low hundreds. Definitely not one per machine, but also not just one per datacenter.

As you say, the goal is to keep the system clocks on the server fleet tightly aligned, to enable things like TrueTime. But also to have sufficient redundancy and long enough holdover in the absence of GNSS (usually due to hardware or firmware failure on the GNSS receivers) that the likelihood of violating the SLA on global time uncertainty is vanishingly small.

The "global" part is what pushes towards having higher end frequency standards, they want to be able to freewheel for O(days) while maintaining low global uncertainty. Drifting a little from external timescales in that scenario is fine, as long as all their machines drift together as an ensemble.

The deployment I know of was originally rubidium frequency standards disciplined by GNSS, but later that got upgraded to cesium standards to increase accuracy and holdover performance. Likely using an "industrial grade" cesium standard that's fairly readily available, very good but not in the same league as the stuff NIST operates.

> There's a lot of focus in this thread on the atomic clocks but in most datacenters, they're not actually that important and I'm dubious that the hyperscalers actually maintain a "fleet" of them, in the sense that there are hundreds or thousands of these clocks in their datacenters.

I mean, fleets come in all sizes; but if you put one atomic reference in each AZ of each datacenter, there's a fleet. Maybe the references aren't great at distributing time, so you add a few NTP distributors per datacenter too and your fleet is a little bigger. Google's got 42 regions in GCP, so they've got a case for hundreds of machines for time (plus they've invested in spanner which has some pretty strict needs); other clouds are likely similar.

GPS satellites have their own atomic clocks. They're synchronized to clocks at the GPS control center at Schriever Space Force Base, Colorado, formerly Falcon AFB. They in turn synchronize to NIST in Boulder, Colorado. GPS has a lot of ground infrastructure checking on the satellites, and backup control centers. GPS should continue to work fine, even if there's some absolute error vs. NIST. Unless there have been layoffs.

It is also important to realize that an atomic clock will only give you a steady pulse. It will count seconds for you, and do so very accurately, but that is not the same as knowing what time it is.

If you get a rubidium clock for your garage, you can sync it up with GPS to get an accurate-enough clock for your hobby NTP project, but large research institutions and their expensive contraptions are more elaborate to set up.

woah hold on a sec. that's not how these clocks are actually used though.

It's a huge huge huge misconception that you can just plunk down an "atomic clock", discipline an NTP server with it and get perfect wallclock time out of it forever. That is just not how it works. Two hydrogen masers sitting next to each other will drift. Two globally distributed networks of hydrogen masers will drift. They cannot NOT drift. The universe just be that way.

UTC is by definition a consensus; there is no clock in the entire world that one could say is exactly tracking it.

Google probably has the gear and the global distribution that they could probably keep pretty close over 30-60 days, but they are assuredly not trying to keep their own independent time standard. Their goal is to keep events correlated on their own network, and for that they just need good internal distribution and consensus, and they are at the point where doing that internally makes sense. But this is the same problem on any size network.

Honestly for just NTP, I've never really seen evidence that anything better than a good GPS disciplined TCXO even matters. The reason they offer these oscillators in such devices is because they usually do additional duties like running PtP or distributing a local 10mhz reference where their specific performance characteristics are more useful. Rubidium, for instance, is very stable at short timescales but has awful long term stability.

Sure, but F2 is a bit more accurate: "As of February 2016 the IT-CsF2 cesium fountain clock started reporting a uB of 1.7 × 10−16 in the BIPM reports of evaluation of primary frequency standards." ( from https://web.archive.org/web/20220121090046/ftp://ftp2.bipm.o... )

Great list! Just double-checked the CAT timekeeping requirements [1] and the requirement is NIST sync. So a subset of all UTC.

You don’t need to actually sync to NIST. I think most people PTP/PPS to a GPS-connected Grandmaster with high quality crystals.

But one must report deviations from NIST time, so CAT Reporters must track it.

I think you are right — if there is no NIST time signal then there is no properly auditable trading and thus no trading. MFID has similar stuff but I am unfamiliar.

One of my favorite nerd possessions is my hand-signed letter from Judah Levine with my NIST Authenticated NTP key.

[1] https://www.finra.org/rules-guidance/rulebooks/finra-rules/6...

One (amongst many) NTP server going down creates less issues than an NTP server spreading wrong time.

If your application is so critical that NTP timing loss causes disaster and your holdover fails in less than a day and you aren't generating your own via gps, you are incompetent, full stop

Bravo

>The actual port for the actual clock, the little metal thingy that is going buzz, buzz, buzz with voltage every second on the dot? Yeah, that little port isn't actually hooked up to anything, as again, it's so sensitive (impedance matching). So they use the other ports on the card for actual data transfer to the rest of the world.

Can you restate this part in full technical jargon along with more detail? I'm having a hard time following it

I'm the Manager of the Computing group at JILA at CU, where utcnist*.colorado.edu used to be housed. Those machines were, for years, consistently the highest bandwidth usage computers on campus.

Unfortunately, the HP cesium clock that backed the utcnist systems failed a few weeks ago, so they're offline. I believe the plan is to decommission those servers anyway - NIST doesn't even list them on the NTP status page anymore, and Judah Levine has retired (though he still comes in frequently). Judah told me in the past that the typical plan in this situation is that you reference a spare HP clock with the clock at NIST, then drive it over to JILA backed by some sort of battery and put it in the rack, then send in the broken one for refurb (~$20k-$40k; new box is closer to $75k). The same is true for the WWVB station, should its clocks fail.

There is fiber that connects NIST to CU (it's part of the BRAN - Boulder Research and Administration Network). Typically that's used when comparing some of the new clocks at JILA (like Jun Ye's strontium clock) to NIST's reference. Fun fact: Some years back the group was noticing loss due to the fiber couplers in various closets between JILA & NIST... so they went to the closets and directly spliced the fibers to each other. It's now one single strand of fiber between JILA & NIST Boulder.

That fiber wasn't connected to the clock that backed utcnist though. utcnist's clock was a commercial cesium clock box from HP that was also fed by GPS. This setup was not particularly sensitive to people being in the room or anything.

Another fun fact: utcnist3 was an FPGA developed in-house to respond to NTP traffic. Super cool project, though I didn't have anything to do with it, haha.

Will the time it takes you to answer depend on the mass of the person asking?

Direct control is by Space Force. However the US Navy Naval Observatory is responsible for (amongst other things) providing timekeeping for the DoD.

In this context, they feed timing updates to the GPS operators https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-N...

I really, really can't get behind this sentiment. Having a reliable, accurate time keeping mechanism doesn't seem like an outlandish issue to want to maintain. Timekeeping has been an important mechanism for humans for as long as recorded history. I don't understand the wisdom of shooting down establishing systems to make that better, even if the direct applicability to a single human's life is remote. We are all part of a huge, interconnected system whether we like it or not, and accurate, synchronized timekeeping across the world does not sound nefarious to me.

Far more things rely on reliable and accurate time-keeping than just being on time to work. Timekeeping is vitally important (even if it's not readily visible) to lots of critical infrastructure worldwide.

"Wearing a watch is like being handcuffed to time."

-My Friend Andy

This is like the kid in school who doesn't think they should have to learn algebra since they think they will never use it.

And as things fell apart / Nobody paid much attention

oh....no, not really, no, the world needs GPS, so, yeah. this is not like scrooge mcduck telling you to be at work on time. scrooge still has a windup watch

It's probably not worth it to go through your insurance for the loss of food and perishables in the fridge/freezer. It counts as a claim on your home insurance and can result in increased rates or even your insurer dropping coverage at the next renewal.