Hive Hive
Sign in

fix(server, infra): runner metrics chart alignment + full-job coverage

GitHub issue · Closed

Metadata
Source
tuist/tuist #11595
Updated
Jul 5, 2026
Domains
Compute
Details

What

Three fixes that make the runner-job machine-metrics charts trustworthy — the step↔chart correlation, and metric coverage for the whole job. Originally split across #11595/#11600/#11601; folded here into one.

1. Step ↔ chart axis alignment (server)

fix(server): anchor runner metrics chart axis to the job's step window

Hovering a step highlighted the wrong part of the chart because the axis auto-scaled to the metric-sample extent, which drifts from the job’s step timeline. Anchor the chart x-axis to the job’s [min step start, max step end] window so the step bands and the metric line share one timeline. Covered by a LiveView test.

2. Long-job truncation — SA token TTL (infra / tart-kubelet)

fix(infra): give runner SA tokens an 8h TTL so long-job metrics don't truncate

The macOS sampler reuses the per-Pod dispatch SA token to POST for the whole job, but tart-kubelet mints it once at boot with a 1h TTL and never rotates it. A warm Pod ages the token before claiming, so long jobs expired mid-run → 401 → charts cut off partway (a 17-min job whose metrics stopped 10 min in). Bump ExpirationSeconds 3600 → 28800. The token is Pod-bound, so a longer TTL has no real security cost.

3. Step/metric timeline mismatch — clock skew (infra / runner-image)

fix(infra): NTP-step the macOS runner clock before the job starts

Some jobs’ step timestamps were minutes early relative to the metrics. Root cause: a golden-base VM can be handed a job seconds after boot, before macOS timed has synced a clock that started minutes behind. The GitHub runner and metrics-poll.sh both stamp off that clock while timed corrects it mid-run, so the two land on different timelines. Diagnosed on job 84477965485: the runner truly started ~07:30:56 (claim, first server-received POST, and accurate-clock samples all agree), yet GitHub steps claim 07:27:12 → 07:33:18 — 3m44s early at the start, ~5s at the end (the fingerprint of a boot-skewed clock correcting mid-run). Force sudo sntp -sS -t 5 time.apple.com right after dispatch succeeds (network up) and before the runner + sampler start. macOS-only.

Validation

  • Server: LiveView test for the step-window axis; mix compile --warnings-as-errors / credo clean; formatted.
  • tart-kubelet (Go): go build ./..., go vet, gofmt clean.
  • runner-image (shell): shellcheck / bash -n clean; sntp flags confirmed against man sntp.

Rollout notes

  • Server axis fix ships with the next server deploy.
  • Token TTL takes effect for Pods created after the tart-kubelet roll; the clock sync for VMs booted from the next runner-image roll — both apply as the fleet turns over, not instantly.
  • One assumption to confirm: the macOS VM can reach time.apple.com over UDP/123. It’s fail-open (falls back to timed), but if egress is locked down we’d point it at an in-cluster NTP source.

🤖 Generated with Claude Code

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments

No GitHub comments yet.