Hive Hive
Sign in

fix(infra): NTP-step the macOS runner clock before the job starts

GitHub issue · Closed

Metadata
Source
tuist/tuist #11601
Updated
Jul 5, 2026
Domains
Compute
Details

What

Forces an NTP clock step on the macOS runner VM immediately before it starts a job, so GitHub step timestamps and machine-metric timestamps land on the same clock. This fixes the runner-job charts where hovering a step highlighted the wrong region and early steps looked missing.

Why

The metrics weren’t actually misaligned — the step timestamps were early, because of a skewed guest clock.

macOS VMs can boot with a clock minutes behind, and timed only syncs it asynchronously over the network. With golden-base provisioning a freshly-booted VM can be handed a job within seconds — before timed catches up. The GitHub runner stamps each step’s started_at/completed_at off that clock, and metrics-poll.sh stamps samples off it too, but timed corrects the clock mid-run. Net result: the step timeline is minutes early at the start and drifts into alignment only by the end, so it sits on a different timeline than the metrics.

Diagnosed end-to-end on job 84477965485 (production):

Signal Time
runner_jobs claimed / started / completed 07:30:55 / 07:30:56 / 07:33:23
First metrics POST received by the server 07:30:58 (nothing earlier in Loki)
Metric samples (VM clock, ~1–2s from receipt) 07:30:56 → 07:33:08
GitHub steps 07:27:12 → 07:33:18

Everything that runs on the real (server/accurate) clock agrees the runner started at ~07:30:56. Only GitHub’s step times disagree — 3m44s early at the start, ~5s off at the end. That “early at the start, aligned by the end” signature is the fingerprint of a clock that boots behind and gets corrected mid-run.

How

In dispatch-poll.sh, right after dispatch succeeds (network confirmed up) and before forking metrics-poll.sh / execing ./run.sh:

sudo /usr/bin/sntp -sS -t 5 time.apple.com
  • -S steps a >50ms offset via clock_settime(2) (our case); -s slews a sub-50ms one.
  • Best-effort: a failed sync just leaves timed to converge later, and never blocks the job.
  • macOS-only — Linux kata VMs get an accurate paravirt clock at boot, and this file is the macOS runner’s boot orchestrator.

Validation

  • shellcheck / bash -n clean (the two pre-existing warnings on the trap and cd are untouched).
  • sntp is part of the base macOS system (/usr/bin/sntp), and the flag semantics were confirmed against man sntp on macOS 15.

Rollout / scope

  • Takes effect for VMs booted from the next runner-image roll.
  • This is the third and final piece of the runner-metrics coverage/quality work, alongside #11595 (step↔chart axis alignment) and #11600 (8h SA-token TTL so long jobs don’t truncate).

🤖 Generated with Claude Code

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments
F
fortmarek Jul 1, 2026

Folded into #11595 (single combined PR for the runner-metrics fixes). Same commit, cherry-picked there.