Hive Hive
Sign in

fix(infra): 8h runner SA token TTL so long-job metrics don’t truncate

GitHub issue · Closed

Metadata
Source
tuist/tuist #11600
Updated
Jul 5, 2026
Domains
Compute
Details

What

Bumps the per-Pod runner ServiceAccount token TTL that tart-kubelet mints from 1h → 8h, so long-running jobs’ machine-metrics charts stop truncating partway through.

Why

The macOS runner metrics sampler (metrics-poll.sh) reuses the same per-Pod dispatch SA token — staged into the VM at /etc/tuist-sa-token — to POST samples for the whole job. But tart-kubelet mints that token once at VM boot and never rotates it, with a 1h TTL (satoken.go). A warm Pod ages the token while it sits idle in the pool, so by the time it claims a long job the token has little life left; once it expires the sampler’s POSTs get 401, samples stop being recorded, and the chart just ends mid-job.

Confirmed against 30 recent macOS jobs: short jobs are well-covered, but long ones truncate, and sample counts fall far below duration / 15s. Examples: a 17-min job whose metrics stopped 10 min in; a 14-min job that stopped 4 min early.

How

ExpirationSeconds: 3600 → 28800 in cmd/tart-kubelet/main.go. satoken.go already advised setting this “generously” because the token isn’t rotated — 1h was only ever correct when dispatch (a one-shot call minutes after boot) was the token’s sole consumer. Now the metrics sampler reuses it for the job’s full duration, so it must outlive warm-time + job.

The token is Pod-bound (BoundObjectRef), so the apiserver invalidates it the instant the Pod is reaped regardless of TTL — a longer TTL carries no real security cost, it just needs to survive the Pod’s active life. Updated the stale doc comments to reflect the metrics-reuse rationale.

Validation

  • go build ./..., go vet ./..., gofmt all clean.
  • No behavior change beyond the TTL value; the token is still audience-scoped (tuist-runners-dispatch), principal-checked, and Pod-bound.

Rollout / scope

  • Takes effect for Pods created after the tart-kubelet rollout; existing warm Pods keep their old 1h token until recycled.
  • This fixes the end-truncation class. A separate late-start case (some jobs’ metrics begin minutes in) is still under investigation — it isn’t token-related (it’s missing early samples, most likely VM clock/NTP skew at boot), and will be handled separately.

🤖 Generated with Claude Code

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments
F
fortmarek Jul 1, 2026

Folded into #11595 (single combined PR for the runner-metrics fixes). Same commit, cherry-picked there.