feat(server, infra): sample runner machine metrics in-VM

What

Makes the runner-job Metrics tab actually populate by moving machine-metrics sampling into the runner VM. The original feature (#11458) shipped the server endpoint, storage, and dashboard plus a collector in the runners-controller — but in production that collector produced nothing, so every job’s Metrics tab stayed empty.

Why the controller-side collector didn’t work

The controller sampled each node’s kubelet /stats/summary through the apiserver node proxy. In our cluster that source is unavailable, and it failed on every runner node (confirmed in the controller logs — a fetch error per node every ~10s):

• macOS Tart fleet (os=darwin, tuist.dev/runtime=tart) — the custom tart-kubelet doesn’t serve the cAdvisor Summary API at all. This is the dominant runner platform, and nothing outside the VM can observe a Tart VM’s resource usage anyway.
• Linux kata nodes — the proxied request is rejected too.
• No fallback: tuist-runners cAdvisor series are deliberately dropped from Prometheus for cost, and there’s no metrics-server.

The only vantage point that can see a runner’s usage — especially a macOS Tart VM — is the VM itself. So the producer belongs in the runner.

How

Remove the broken collector — delete internal/podmetrics (sampler + kubelet source + POST client), its wiring + --metrics-sample-interval flag, and the nodes/proxy RBAC. This also stops the per-node error spam in prod.

Server — authenticate the runner directly (TuistWeb.RunnerPodAuth). The metrics endpoint previously required the runners-controller’s SA. It now accepts the runner Pod’s own per-pod token — the same one it already presents to dispatch (audience tuist-runners-dispatch, validated via TokenReview) — and requires the token’s principal to be the SA for the pod_name in the path. Since the controller mints each Pod and SA with the same name, SA name == pod name, so a Pod can report only for itself and a leaked token can’t write another Pod’s metrics. Pod→claim→job resolution and the unclaimed-Pod no-op are unchanged; pods/stopped keeps using the controller SA.

macOS sampler (infra/runner-image/metrics-poll.sh) — dispatch-poll.sh forks it into the background right before handing off to the GitHub Actions runner, so it samples for the job’s full duration and dies when the EXIT trap halts the VM. Whole-VM CPU (top), memory (vm_stat), network (netstat -ibn, differenced into per-interval throughput), disk (df of the APFS Data volume — / is the sealed system volume). Token at /etc/tuist-sa-token.

Linux sampler (infra/linux-runner-image/metrics-sampler.sh) — because the dispatch token is isolated from the customer container, the sampler runs as a dedicated metrics native sidecar (restartPolicy=Always) the controller adds alongside dind. It mounts the token (trusted code, like the poller) and the JIT emptyDir, reads VM-wide /proc (a kata pod is one microVM) for CPU/memory/network, and dfs the JIT volume’s backing filesystem for disk. It waits for the poller to stage the JIT (a claimed job) before sampling, so warm-standby Pods don’t post. Linux reports cpu_iowait_percent (macOS sends 0).

Both POST to …/pods/<pod>/metrics every 15s, fail-open (a bad sample or failed POST is skipped, never affecting the job).

Validation

• Server: mix compile --warnings-as-errors clean, mix credo clean, formatted. Tests pass — the Pod-name endpoint under the new RunnerPodAuth (records under the claimed job, unclaimed no-op, SA-mismatch → 401, missing bearer → 401, apiserver-unavailable → 503).
• runners-controller (Go): go test ./..., go vet, gofmt all clean — podtemplate tests assert the Linux metrics sidecar (token + JIT mounts, native restart policy) and that macOS gets no sidecar.
• Shell: both samplers pass shellcheck; the parsers were validated on real macOS (CPU interval sample, working-set memory, Data-volume disk) and against /proc-format fixtures for Linux (net rx/tx sum excluding lo, CPU busy/iowait delta math).

Deploy / rollout notes

• The controller change and the server change can deploy in any order (auth fail-open): until the new runner images roll out, no runner posts; the server simply accepts them once they do.
• Metrics appear once both the new runner images (Linux + macOS) are built/rolled out and the server auth change is live. Older warm Pods on prior images won’t post until recycled onto the new image.
• Known limitation: the macOS sampler reads its SA token once at job start; a job running past the token’s 1h TTL would stop posting (fail-open). Linux re-reads the kubelet-rotated token each tick.

🤖 Generated with Claude Code

No GitHub comments yet.