Nist Was 5 Μs Off Utc After Last Week's Power Cut

> Gah, just when you think you can trust time.nist.gov

You still can...

If you're that considered about 5 microseconds: Build your own Stratum 1 time server https://github.com/geerlingguy/time-pi

or just use ntppool https://www.ntppool.org/en/

I'm more concerned about what you think they did to earn your trust in the first place

Use the other servers as well: https://tf.nist.gov/tf-cgi/servers.cgi

For instance, time-a-wwv.nist.gov.

One should configure a number of different NTP sources instead of just a single host.

Most places that need accurate time get it from GPS. That is 10-100 ns.

Also, you can use multiple NIST servers. They have ones in Fort Collins, CO and Gaithersburg, MD. Most places shouldn't use NIST directly but Stratum 1 name servers.

Finally, NTP isn't accurate enough, 10-100 ms, for microsecond error to matter.

Yes.

Use NTP with ≥4 diverse time sources, just as RFC 5905 suggests doing. And use GPS.

(If you're reliant upon only one source of a thing, and that thing is important to you in some valuable way, then you're doing it wrong. In other words: Backups, backups, backups.)

their handling it responsibly seems like more evidence for trusting them, not less?

The 5us inaccuracy is basically irrelevant to NTP users, from the second update to the Internet Time Service mailing list[1]:

To put a deviation of a few microseconds in context, the NIST time scale usually performs about five thousand times better than this at the nanosecond scale by composing a special statistical average of many clocks. Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet; uncertainties on the order of 1 millisecond (one thousandth of one second) are more typical due to asymmetry and fluctuations in packet delay.

[1] https://groups.google.com/a/list.nist.gov/g/internet-time-se...

It's a good question, and I wondered the same. I don't know, but I'd postulate:

As it stands at the minute, the clocks are a mere 5 microseconds out and will slowly get better over time. This isn't even in the error measurement range and so they know it's not going to have a major effect on anything.

When the event started and they lost power and access to the site, they also lost their management access to the clocks as well. At this point they don't know how wrong the clocks are, or how more wrong they're going to get.

If someone restores power to the campus, the clocks are going to be online (all the switches and routers connecting them to the internet suddenly boot up), before they've had a chance to get admin control back. If something happened when they were offline and the clocks drifted significantly, then when they came online half the world might decide to believe them and suddenly step change to follow them. This could cause absolute havoc.

Potentially safer to scram something than have it come back online in an unknown state, especially if (lots of) other things are are going to react to it.

In the last NIST post, someone linked to The Time Rift of 2100: How We lost the Future --- and Gained the Past. It's a short story that highlights some of the dangers of fractured time in a world that uses high precision timing to let things talk to each other: https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...

> […] where sending inaccurate time was worse than sending no time […]

When you ask a question, it is sometimes better to not get an answer—and know you have not-gotten an answer—then to get the wrong answer. If you know that a 'bad' situation has arisen, you can start contingency measures to deal with it.

If you have a fire alarm: would you rather have it fail in such a way that it gives no answer, or fail in a way where it says "things are okay" even if it doesn't know?

We don't use NTP, but for robotics, stereo camera synchronization we often want the two frames to be within ~10us of eachother. For sensor fusion we then also need a lidar on PTP time to be translated to the same clock domain as cameras, for which we also need <~10us.

We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.

Lots of things do. Shoot, even plain old TDM needs timing precision on the order of picoseconds to nanoseconds.

I believe LTE networks require it to coordinate timeslots between overlapping cells. Of course, they can use whatever reference they want, as long as all the cells are using the same one - it doesn't have to be UTC.

Someone else referenced low power ham radio modes like WSPR, which I also don't know much about, but I can imagine they have timeslots linked to UTC and require accuracy. Those modes have extremely low data rates and narrow bandwidths, requiring accurate synchronization. I don't know if they're designed to self-synchronize, or need an external reference.

You probably want to ask about accuracy. Any random microcontroller from the 90s needs microsecond precision.

I worked at Altera (FPGA supplier) as the Ethernet IP apps engineer for Europe for a few years. All the big telecoms (Nokia, Ericsson, Cisco, etc) use Precision Time Protocol (PTP) in some capacity and all required clocks to be ns levels for accuracy. Sometimes as low a 10ns at the boundary. Any imperfection in the local clock directly converts into timestamp error, and timestamp error is what limits PTP synchronization performance. Timestamps are the fundamental observable in PTP. Quantization and jitter create irreducible timestamp noise. That noise directly limits offset and delay estimation. Errors accumulate across network elements and internal clock error must be much smaller than the system requirement.

I think most people would look at the error and think "what's the big deal" but at all the telecoms customers would be scrambling to find a clock that hasn't fallen out of sync.

[delayed]

We need nanosecond precision for trading - basically timestamping exchange/own/other events and to measure latency.

Lightning detection. You have a couple of ground stations with known positions that wait for certain electromagnetic puses, and which record the timestamps of such pulses. With enough stations you can triangulate the location of the source of each pulse. Also a great way to detect nuclear detonations.

There is a german club that builds and distrubutes such stations (using GPS for location and timing), with a quite impressive global coverage by now:

https://www.blitzortung.org

Nuclear measurements, where the speed of a gamma ray flying across a room vs a neutron is relevant. But that requires at least nanosecond time resolution, and you’re a long way from thinking about NTP.

A database like Google Spanner has higher latency in proportion to the uncertainty about the time. Driving the time uncertainty down into the microsecond range, or lower, keeps latency low.

Yes, but always got it from GPS so presumably they'd be off about the same amount.

Distributed sonar, allows placing receivers willy-nilly and aligning the samples later.

Remote microphone switching - though for this you wouldn't notice 5us jitter, it's just that the system we designed happened to have granularity that good.

I mean, we routinely benchmark things that take microseconds or less. I've seen a 300 picosecond microbenchmark (single cycle at 3GHz). No requirement that absolute time is correct, though.

If you want sample accurate audio transmission you're going to want resolution on the order of 10s of microseconds.

High speed finance is msec and below. Fastest publically known tick to trade is just shy of 14 nanos.

Timekeeping starts to become really hard, often requiring specialized hardware and protocols.

When we collected, correlated, and measured all controlling messages in a whole 4G network. Millisecond precision meant guaranteed out of order message flows.

How do you even get usable microsecond precision sync info from a server thousands of kilometers away? The latency is variable so the information you get can't be verified / will be stale the moment it arrives. I'm quite ignorant on the topic.

(Assuming "precision" really meant "accuracy") The network equipment I work on requires sub microsecond time sync on the network for 5G providers and financial trading customers. Ideally they'd just get it from GPS direct, but that can be difficult to do for a rack full of servers. Most of the other PTP use cases I work with seem to be fine with multiples of microseconds, e.g. Audio/Video over the network or factory floor things like PLCs tend to be find with a few us over the network.

Perhaps a bit more boring than one might assume :).

We use nanosecond precision driven by GPS clocks. That timestamp in conjunction with star tracker systems gives us reliable positioning information for orbital entities.

https://en.wikipedia.org/wiki/Star_tracker

At a previous role, we needed nanosecond precision for a simulcast radio communications system. This was to allow for wider transmission for public safety radio systems without having to configure trunking. We could even adjust the delay in nanoseconds to move the deadzones away from inhabited areas.

We solved this by having GPS clocks at each tower as well as having the app servers NTP with each other. The latter burned me once due to some very dumb ARP stuff, but that's a story for another day.

The high frequency trading guys

Your speakers do so that people's voices match their mouth movements. The speaker clocks need to be in-sync with the cpu clocks and they operate at different frequencies.

Look up PTP White Rabbit.

Yes, the US government is banning all those democrat windmills that conspired to blow over the NTP server.

Spanner

(See https://docs.cloud.google.com/spanner/docs/true-time-externa...)

Not sure but synthetic massive aperture radio telescope would need syncing their local clocks.

I defer to the experts.

Telling people at the bar that I have an atomic clock at home.

GPS

As a very coarse number, 5µs is 1500 meters of radio travel.

If (and it isn't very conceivable) GPS satellites were to get 5µs out of whack, we would be back to Loran-C levels of accuracy for navigation.

I work at a particle accelerator. We use White Rabbit (https://white-rabbit.web.cern.ch/) to synchronize some very sensitive devices, mostly the RF power systems and related data acquisition systems, down to nanosecond accuracy.

Spacecraft state vectors.

It also stands for Universel Temps Coordonné.

Universal Time, Coordinated.

That's an interesting rationale.

That doesn't quite match what the wikipedia page says:

> The official abbreviation for Coordinated Universal Time is UTC. This abbreviation comes as a result of the International Telecommunication Union and the International Astronomical Union wanting to use the same abbreviation in all languages. The compromise that emerged was UTC, which conforms to the pattern for the abbreviations of the variants of Universal Time (UT0, UT1, UT2, UT1R, etc.).

It's also the time in Iceland, conveniently

I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems. Sub-microsecond latency, yes, but when that mattered it was in order to do something as soon as possible rather than as close as possible to Wall Clock Time X.

Still useful for post-trade analysis; perhaps you can determine that a competitor now has a faster connection than you.

The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.

My guess would be scientific experiments where they need to correlate or sequence data over large regions. Things like correlating gravitational waves with radio signals and gamma ray bursts.

SIGINT as a source clock for others in a network doing super accurate TDOA for example.

[delayed]

I think Google uses chrony instead of NTP

[delayed]

> I've never heard of this! Very cool service, presumably for … quant / HFT / finance firms (maybe for compliance with FINRA Rule 4590 [3])?

To start with, probably for scientific stuff, à la:

* https://en.wikipedia.org/wiki/White_Rabbit_Project

But fibre-based time is important in case of GNSS time signal loss:

* https://www.gpsworld.com/china-finishing-high-precision-grou...

> Google/hyperscalers as input to Spanner or other global databases?

Think Google might have rolled their own clock sources and corrections.

Ex: Sundial, https://www.usenix.org/conference/osdi20/presentation/li-yul...

science equipment, distributed radio-telescopes where you need to precisely align data received at different locations

Who does use those top-level servers? Aren’t some of them propagating the error or are all secondary level servers configured to use dispersed top-level servers? And how do they decide who is right when they don’t match?

Is pool.ntp.org dispersed across possible interference and error correlation?

I believe if you use time.nist.gov it round robins dns requests, so there’s a chance you’d have connected to the Boulder server. So for some people they would have experienced NIST 5 μs off.

Chrony is the default already in some distros (RHEL and SLES that I know of), probably for this very reason.

I'll consider it clickbait...

... unless someone with real experience needing those tolerances chimes in and explains why it's true.

You'll never guess why. The answer might shock you.

Can you elaborate on how? Did they consciously exploit this or their systems just had a lucky glitch?