Hive
fix(server, infra): runner metrics chart alignment + full-job coverage
GitHub issue · Closed
What
Three fixes that make the runner-job machine-metrics charts trustworthy — the step↔chart correlation, and metric coverage for the whole job. Originally split across #11595/#11600/#11601; folded here into one.
1. Step ↔ chart axis alignment (server)
fix(server): anchor runner metrics chart axis to the job's step window
Hovering a step highlighted the wrong part of the chart because the axis auto-scaled to the metric-sample extent, which drifts from the job’s step timeline. Anchor the chart x-axis to the job’s [min step start, max step end] window so the step bands and the metric line share one timeline. Covered by a LiveView test.
2. Long-job truncation — SA token TTL (infra / tart-kubelet)
fix(infra): give runner SA tokens an 8h TTL so long-job metrics don't truncate
The macOS sampler reuses the per-Pod dispatch SA token to POST for the whole job, but tart-kubelet mints it once at boot with a 1h TTL and never rotates it. A warm Pod ages the token before claiming, so long jobs expired mid-run → 401 → charts cut off partway (a 17-min job whose metrics stopped 10 min in). Bump ExpirationSeconds 3600 → 28800. The token is Pod-bound, so a longer TTL has no real security cost.
3. Step/metric timeline mismatch — clock skew (infra / runner-image)
fix(infra): NTP-step the macOS runner clock before the job starts
Some jobs’ step timestamps were minutes early relative to the metrics. Root cause: a golden-base VM can be handed a job seconds after boot, before macOS timed has synced a clock that started minutes behind. The GitHub runner and metrics-poll.sh both stamp off that clock while timed corrects it mid-run, so the two land on different timelines. Diagnosed on job 84477965485: the runner truly started ~07:30:56 (claim, first server-received POST, and accurate-clock samples all agree), yet GitHub steps claim 07:27:12 → 07:33:18 — 3m44s early at the start, ~5s at the end (the fingerprint of a boot-skewed clock correcting mid-run). Force sudo sntp -sS -t 5 time.apple.com right after dispatch succeeds (network up) and before the runner + sampler start. macOS-only.
Validation
- Server: LiveView test for the step-window axis;
mix compile --warnings-as-errors/credoclean; formatted. - tart-kubelet (Go):
go build ./...,go vet,gofmtclean. - runner-image (shell):
shellcheck/bash -nclean;sntpflags confirmed againstman sntp.
Rollout notes
- Server axis fix ships with the next server deploy.
- Token TTL takes effect for Pods created after the
tart-kubeletroll; the clock sync for VMs booted from the next runner-image roll — both apply as the fleet turns over, not instantly. - One assumption to confirm: the macOS VM can reach
time.apple.comover UDP/123. It’s fail-open (falls back totimed), but if egress is locked down we’d point it at an in-cluster NTP source.
🤖 Generated with Claude Code
Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.
Start a Flight and preserve its objective, outcome, and session here.
No GitHub comments yet.