Atomic Time Source Failure at Nist Gaithersburg Campus

Posted24 days agoActive24 days ago

dpcx

19 points

6 comments

groups.google.comTech Discussionstory

informativeneutral

Debate

20/100

TimekeepingNistAtomic Clocks

Key topics

Timekeeping

Nist

Atomic Clocks

Discussion Activity

Light discussion

First comment

58m

Peak period

2-3h

Avg / period

Key moments

01Story posted
Dec 9, 2025 at 9:41 PM EST
24 days ago
Step 01
02First comment
Dec 9, 2025 at 10:39 PM EST
58m after posting
Step 02
03Peak activity
5 comments in 2-3h
Hottest window of the conversation
Step 03
04Latest activity
Dec 10, 2025 at 2:29 PM EST
24 days ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (6 comments)

Showing 10 comments

CaliforniaKarl

24 days ago

3 replies

Huh, I guess it's best to think of each site's NIST NTP servers as 'load-balancers' in front of a single 'application server'.

Fun fact: Per [0], if you provide enough servers, the NTP client can detect a "falseticker" that is not providing accurate time. The number of NTP servers required is `2n+1` where `n≥1`.

Of course, that requires each NTP server use its own time source.

So, note for me: If I want NTP redundancy and I'm using NIST's servers, pick one NTP server from each of NTP's three sites.

[0]: https://support.ntp.org/Support/SelectingOffsiteNTPServers#U...

RossBencina

24 days ago

3 replies

-10ms, no redundant clocks, and they're leaving most of the servers up with that amount of skew. Wow. I am astonished that NIST does not have multiple clocks over multiple distributed sites with robust ability to detect and bypass individual failures.

jedimastert

24 days ago

1 reply

> I am astonished that NIST does not have multiple clocks over multiple distributed sites with robust ability to detect and bypass individual failures.

Is this sarcasm? I can't tell.

Per the email:

> Servers at the Boulder and WWV/Ft. Collins campuses are independent and unaffected.

RossBencina

24 days ago

Sorry, maybe I got carried away with the tone. But it is not sarcasm. I genuinely did not realise that the NTP service level was so low. There are two problems raised in the email: There is no on-site redundant fail-over upstream of the NTP servers. All NTP servers at the site were not automatically taken down immediately upon detection of the fault (because some were still, in some sense, within tolerance). This places all of the fault management onto downstream NTP servers. I honestly expected NIST to be running a robust cross-site timebase upstream of NTP.

ianburrell

24 days ago

They do have multiple clocks and sites. The primary clock is in Boulder. Only the Maryland time servers are affected, the Colorado ones should be fine. They mention switching to another atomic clock, but that probably has to be setup.

The email explains why they haven't shut down, cause haven't hit the threshold. And talks about maybe shutting them down manually.

metaphor

24 days ago

> I am astonished that NIST does not have multiple clocks over multiple distributed sites with robust ability to detect and bypass individual failures.

They may not operate redundant clocks at a single site, but ITS redundancy posture[1] doesn't look bad at all:

>> Servers at the Boulder and WWV/Ft. Collins campuses are independent and unaffected.

[1] https://tf.nist.gov/tf-cgi/servers.cgi

DANmode

24 days ago

This was implied by them taking the time and effort to establish multiple sites.

metaphor

24 days ago

> So, note for me: If I want NTP redundancy and I'm using NIST's servers, pick one NTP server from each of NTP's three sites.

System robustness hazard that won't tolerate just querying time.nist.gov at 4-sec or greater intervals?

From the cow's mouth[1]:

>> The global address time.nist.gov is resolved to all of the server addresses below in a round-robin sequence to equalize the load across all of the servers.

[1] https://tf.nist.gov/tf-cgi/servers.cgi

octaane

24 days ago

Good to know they have multiple backup clocks across the continental US.

floatin

24 days ago

I wonder if it’s a response to this https://www.reuters.com/world/china/china-accuses-us-cyber-b...

View full discussion on Hacker News

ID: 46213406Type: storyLast synced: 12/10/2025, 4:55:26 AM

Want the full context?