Why Time Synchronization Breaks: NTP, PTP and the Leap-Second Problem

TL;DR — Almost nothing on a modern server crashes because the clock is a few milliseconds wrong — until a certificate fails validation, a database writes transactions out of order, or a leap second makes time run backwards and code that assumed monotonicity panics. NTP keeps clocks within milliseconds; PTP gets to sub-microseconds when hardware does the timestamping; leap smearing hides the leap second; and the CGPM has voted to retire the leap second by 2035. Here is how the whole stack works, why it breaks, and how to run it so it doesn’t.

Time is the quietest dependency in your infrastructure. Kerberos refuses authentication if client and server clocks differ by more than five minutes. TLS certificate validation fails outside a cert’s validity window — in both directions. Distributed databases order transactions with timestamps, so clock skew becomes data corruption. Log aggregation, distributed tracing, cron schedules, cache expiry, JWT exp claims: all of them silently assume that “now” means the same thing on every machine. We touched this from the certificate angle in the Kubernetes security fundamentals post; this piece is about the machinery underneath.

The Hidden Dependency Everything Leans On

Every networked computer carries at least two clocks: a real-time clock (RTC), a battery-backed hardware chip that keeps rough time while the machine is off, and the system clock, a software clock maintained by the kernel. Neither is a great timekeeper on its own. Commodity RTCs drift seconds per day; virtual machines inherit hypervisor time and can jump when they migrate. So the system clock is continuously disciplined by an external reference — and everything above the kernel simply trusts it.

That trust is what makes time failures so strange. A monitoring system that shows “no data” for one server may simply have logs timestamped in the future. A database that loses writes may have ordered them by clock. The failure mode isn’t usually “everything crashed” — it’s a small set of systems quietly disagreeing about the order of events.

How NTP Actually Works

The Network Time Protocol (NTP), currently version 4 (RFC 5905), is one of the oldest protocols still running the internet, and its design is beautifully defensive. A client sends a request with its own timestamp, the server replies with its timestamps, and the client computes both the offset between the clocks and the round-trip delay from the four timestamps. Because the server’s processing time is (approximately) symmetric, the client can estimate offset even through variable network delay — accuracy of a few milliseconds over the public internet, tens of microseconds on a LAN.

NTP organizes time sources into strata:

Two protocol details matter operationally. First, NTP timestamps count seconds since 1 January 1900 in a 32-bit field, which wraps in February 2036; NTPv4 handles this with an “era number” so well-maintained implementations roll over cleanly — but embedded devices with frozen NTPv3-era code are a real 2036 concern, the same class of problem as Y2K. Second, NTP deliberately prefers slewing (slightly speeding up or slowing the clock) over stepping (jumping it), because stepping backwards breaks exactly the software we’ll meet in the leap-second section.

What PTP Changes

The Precision Time Protocol (PTP, IEEE 1588 — current revision IEEE 1588-2019) exists because NTP’s accuracy ceiling comes from one thing: where the timestamp is taken. If the host software stamps packets, every layer of jitter — interrupt latency, scheduling, virtualization — pollutes the measurement. PTP’s central move is hardware timestamping: supported network cards and switches stamp PTP frames in silicon at the wire, removing almost all software jitter. With that, and with switches correcting for their own forwarding delay, PTP delivers sub-microsecond accuracy — even tens of nanoseconds in controlled networks.

PTP stopped being niche in this decade. Meta published its production deployment in November 2022, synchronizing its data centers with PTP and open-sourcing the hardware: the OCP Time Appliances Project’s Time Card is an open-source PCIe grandmaster reference design combining a GNSS receiver with PTP hardware timestamping. AWS made the move mainstream the next year: in November 2023 it upgraded the Amazon Time Sync Service to microsecond accuracy, exposing PTP hardware clock (PHC) time from Nitro-based instances via time.aws.com — consumed by a standard chrony configuration. Cloud providers distributing sub-microsecond time as a managed service is the strongest signal yet that clock skew has become a production-scale problem.

Leap Seconds: The Outage Machine

UTC is kept within 0.9 seconds of astronomical time (UT1) by occasionally inserting a leap second — 23:59:60 — at the end of June or December. There have been 27 leap seconds since 1972; the last fell on 31 December 2016; and the offset between continuous atomic time (TAI) and UTC is now exactly 37 seconds. That’s the theory. In practice, a leap second means one of two things on a Linux machine: the kernel replays 23:59:59 (time appears to stand still for a second) or steps the clock backwards — and both break software.

30 June 2012: the leap second triggered a bug in the Linux kernel’s high-resolution timer subsystem that put some CPUs into livelock; Java applications and Cassandra databases spun at 100% CPU, and Reddit, LinkedIn, Mozilla, Yelp and airline reservation systems went down or degraded until services were restarted.

1 January 2017: Cloudflare’s custom RRDNS DNS server assumed elapsed time could only increase. The kernel stepped the clock back one second at the leap, a round-robin computation produced a negative elapsed time, and the code panicked — taking out roughly 0.2% of Cloudflare’s global DNS queries in the first hours of the year. The fix, famously, was a single-character change (a > became >=).

The industry’s main mitigation is leap smearing, pioneered by Google: instead of inserting one dramatic second, the time source stretches time linearly across 24 hours, each second about 11.6 microseconds long, so clients never see a discontinuity. time.google.com serves leap-smeared time; AWS and Cloudflare’s time services do not smear (they follow the leap indicator). This creates the one rule everyone must know: never mix smeared and non-smeared time sources — during a smear window they disagree by up to about a second, and NTP cannot reconcile them.

The end is in sight: at the 27th General Conference on Weights and Measures in November 2022, member states resolved to discontinue leap seconds by or before 2035. Until then, they remain scheduled-capable events that every infrastructure team should have on a calendar.

GNSS Time and the 18-Second Offset

Behind most stratum-0 references is GNSS. GPS satellites broadcast their own timescale, GPS Time — continuous, no leap seconds, epoch 6 January 1980. Because UTC has accumulated 17 leap seconds since 1980 (27 total minus the 10-second TAI−UTC offset at GPS epoch), GPS Time is currently 18 seconds ahead of UTC; receivers apply the offset, which the satellites broadcast. Two operational consequences follow. First, GNSS timing receivers inherit every vulnerability of GNSS itself — we covered how unauthenticated satellite signals can be forged in the ADS-B and GNSS spoofing piece, and a spoofed timing source doesn’t just lie about position, it lies about time. Second, GPS has its own counters: the 10-bit week number wrapped in April 2019 and will wrap again around November 2038 — older firmware that handles the first wrap poorly is a known field failure.

This is why serious timing deployments treat GNSS as one input among several, cross-checked against national-lab NTP services and network-delivered PTP — and why the EFB series‘s theme of data integrity applies to timestamps as much as to charts: the question “how do you know this datum is true?” ends somewhere in a chain of trust that includes physics and protocol design.

Running Time Sync You Can Trust

• Use chrony. It converges in minutes instead of the hours ntpd can take, tolerates intermittent connectivity and virtualized clocks, supports NTS (RFC 8915, authenticated NTP), and is the default on most current enterprise Linux distributions. The chrony project’s own comparison page covers the details; the practical takeaway is that there is almost no workload where ntpd is the better choice today.
• Pick a coherent set of sources. Either your cloud provider’s service (time.aws.com, time.cloudflare.com) plus the NTP Pool — or Google’s smeared time.google.com exclusively. Never both kinds in one configuration.
• Let it slew, but allow a controlled initial step (makestep with a tight limit) so a fresh VM doesn’t serve badly skewed time for hours.
• Monitor the offset, and alert on steps. A chrony offset that grows over weeks is a dying RTC or a flapping network; a sudden step is an event your logs should be able to explain.
• Ask what actually needs PTP. If you’re not running tightly coupled distributed state (financial matching, 5G TDD radio, synchronized measurement), NTP-grade time with good monitoring beats a half-configured PTP network.

Key Takeaways

• Time is a correctness dependency, not a cosmetic one: certificates, auth protocols, databases and logs all fail on skew.
• NTP (RFC 5905) delivers millisecond time over the internet by estimating offset and round-trip together; stratum 16 means “unsynchronized”.
• PTP (IEEE 1588-2019) reaches sub-microsecond accuracy by timestamping in hardware; Meta open-sourced grandmaster hardware (OCP Time Card) and AWS has offered microsecond time via time.aws.com since November 2023.
• Leap seconds broke Reddit (2012, kernel livelock) and Cloudflare (2017, negative elapsed time → panic); smearing fixes it but smeared and non-smeared sources must never be mixed.
• GPS Time runs 18 seconds ahead of UTC and carries no leap seconds; GNSS timing inherits GNSS spoofing risk.
• The leap second is scheduled for retirement by 2035 (CGPM Resolution 4, 2022) — but 2036’s NTP era rollover arrives a year later for unpatched embedded stacks.

FAQ

What is the difference between NTP and PTP?
NTP timestamps packets in software and targets millisecond accuracy across arbitrary networks; PTP timestamps packets in network hardware and targets sub-microsecond accuracy in managed networks. They coexist: NTP for general servers, PTP where timing is part of correctness (finance, telecom, synchronized clusters).

Why do leap seconds cause outages?
The kernel either repeats 23:59:59 or steps the clock backwards. Code that measures elapsed time with the wall clock — instead of a monotonic clock — can see zero or negative durations, and arithmetic that assumes time only moves forward (Cloudflare’s round-robin counter) can panic or loop.

What is leap smearing, and when should I use it?
Serving time with the leap adjustment spread over 24 hours (about 11.6 µs per second) so clients never see a jump. Use it only consistently: an infrastructure that standardizes on Google’s smeared time must not also query non-smeared sources, which disagree by up to ~1 second during the smear.

Why is GPS time 18 seconds ahead of UTC?
GPS Time started aligned with UTC on 6 January 1980 and never inserts leap seconds; UTC has gained 17 since. The 18-second offset is broadcast by the satellites and applied by receivers.

Should I run chrony or ntpd?
chrony, in almost all cases: faster convergence, better behavior on intermittent links and VMs, NTS support, and it’s what the major distributions ship. ntpd remains fine where it’s already working and carefully managed.

What happens to NTP in 2036?
The 32-bit seconds counter since 1900 wraps in February 2036. NTPv4 clients negotiate an era number and handle it cleanly; the risk is unmaintained embedded firmware stuck on old behavior — inventory yours before 2035, ideally alongside the leap-second retirement work.

References

1. Cloudflare — How and why the leap second affected Cloudflare DNS (2017)
2. Wired — Leap second glitch explained (2012 Linux kernel livelock)
3. Google — Leap Smear (Public NTP documentation)
4. AWS — It’s About Time: Microsecond-Accurate Clocks on Amazon EC2 Instances (2023)
5. Meta Engineering — Precision Time Protocol at Meta (2022)
6. OCP Time Appliances Project — Time Card (open-source PTP grandmaster hardware)
7. BIPM — Resolution 4 of the 27th CGPM (2022): discontinuation of leap seconds
8. RFC 5905 — Network Time Protocol Version 4
9. chrony project — Comparison of NTP implementations
10. ntp.org — NTP Timescale and Leap Seconds (D. Mills)

Current as of September 2026. Educational engineering reference — for production timing design, follow your vendor’s current guidance and validate against national metrology services.

Published by Hmmnm

Hands-on cybersecurity tutorials, CVE breakdowns, and guided learning paths — written and lab-tested by the Hmmnm team.

Learning Paths · About Hmmnm · Editorial policy