Hive
fix(infra): NTP-step the macOS runner clock before the job starts
GitHub issue · Closed
What
Forces an NTP clock step on the macOS runner VM immediately before it starts a job, so GitHub step timestamps and machine-metric timestamps land on the same clock. This fixes the runner-job charts where hovering a step highlighted the wrong region and early steps looked missing.
Why
The metrics weren’t actually misaligned — the step timestamps were early, because of a skewed guest clock.
macOS VMs can boot with a clock minutes behind, and timed only syncs it asynchronously over the network. With golden-base provisioning a freshly-booted VM can be handed a job within seconds — before timed catches up. The GitHub runner stamps each step’s started_at/completed_at off that clock, and metrics-poll.sh stamps samples off it too, but timed corrects the clock mid-run. Net result: the step timeline is minutes early at the start and drifts into alignment only by the end, so it sits on a different timeline than the metrics.
Diagnosed end-to-end on job 84477965485 (production):
| Signal | Time |
|---|---|
runner_jobs claimed / started / completed |
07:30:55 / 07:30:56 / 07:33:23 |
| First metrics POST received by the server | 07:30:58 (nothing earlier in Loki) |
| Metric samples (VM clock, ~1–2s from receipt) | 07:30:56 → 07:33:08 |
| GitHub steps | 07:27:12 → 07:33:18 |
Everything that runs on the real (server/accurate) clock agrees the runner started at ~07:30:56. Only GitHub’s step times disagree — 3m44s early at the start, ~5s off at the end. That “early at the start, aligned by the end” signature is the fingerprint of a clock that boots behind and gets corrected mid-run.
How
In dispatch-poll.sh, right after dispatch succeeds (network confirmed up) and before forking metrics-poll.sh / execing ./run.sh:
sudo /usr/bin/sntp -sS -t 5 time.apple.com
-Ssteps a >50ms offset viaclock_settime(2)(our case);-sslews a sub-50ms one.- Best-effort: a failed sync just leaves
timedto converge later, and never blocks the job. - macOS-only — Linux kata VMs get an accurate paravirt clock at boot, and this file is the macOS runner’s boot orchestrator.
Validation
shellcheck/bash -nclean (the two pre-existing warnings on the trap andcdare untouched).sntpis part of the base macOS system (/usr/bin/sntp), and the flag semantics were confirmed againstman sntpon macOS 15.
Rollout / scope
- Takes effect for VMs booted from the next runner-image roll.
- This is the third and final piece of the runner-metrics coverage/quality work, alongside #11595 (step↔chart axis alignment) and #11600 (8h SA-token TTL so long jobs don’t truncate).
🤖 Generated with Claude Code
Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.
Start a Flight and preserve its objective, outcome, and session here.