Hive
fix(infra): 8h runner SA token TTL so long-job metrics don’t truncate
GitHub issue · Closed
What
Bumps the per-Pod runner ServiceAccount token TTL that tart-kubelet mints from 1h → 8h, so long-running jobs’ machine-metrics charts stop truncating partway through.
Why
The macOS runner metrics sampler (metrics-poll.sh) reuses the same per-Pod dispatch SA token — staged into the VM at /etc/tuist-sa-token — to POST samples for the whole job. But tart-kubelet mints that token once at VM boot and never rotates it, with a 1h TTL (satoken.go). A warm Pod ages the token while it sits idle in the pool, so by the time it claims a long job the token has little life left; once it expires the sampler’s POSTs get 401, samples stop being recorded, and the chart just ends mid-job.
Confirmed against 30 recent macOS jobs: short jobs are well-covered, but long ones truncate, and sample counts fall far below duration / 15s. Examples: a 17-min job whose metrics stopped 10 min in; a 14-min job that stopped 4 min early.
How
ExpirationSeconds: 3600 → 28800 in cmd/tart-kubelet/main.go. satoken.go already advised setting this “generously” because the token isn’t rotated — 1h was only ever correct when dispatch (a one-shot call minutes after boot) was the token’s sole consumer. Now the metrics sampler reuses it for the job’s full duration, so it must outlive warm-time + job.
The token is Pod-bound (BoundObjectRef), so the apiserver invalidates it the instant the Pod is reaped regardless of TTL — a longer TTL carries no real security cost, it just needs to survive the Pod’s active life. Updated the stale doc comments to reflect the metrics-reuse rationale.
Validation
go build ./...,go vet ./...,gofmtall clean.- No behavior change beyond the TTL value; the token is still audience-scoped (
tuist-runners-dispatch), principal-checked, and Pod-bound.
Rollout / scope
- Takes effect for Pods created after the
tart-kubeletrollout; existing warm Pods keep their old 1h token until recycled. - This fixes the end-truncation class. A separate late-start case (some jobs’ metrics begin minutes in) is still under investigation — it isn’t token-related (it’s missing early samples, most likely VM clock/NTP skew at boot), and will be handled separately.
🤖 Generated with Claude Code
Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.
Start a Flight and preserve its objective, outcome, and session here.