Hive Hive
Sign in

fix(infra): capture a final runner metric sample at job end

GitHub issue · Closed

Metadata
Source
tuist/tuist #11613
Updated
Jul 5, 2026
Domains
Compute
Details

What

Captures a final machine-metrics sample at the very end of a macOS runner job, so the charts reach the job’s end instead of stopping ~one interval short (the “Complete job” step currently has no corresponding data point).

Why

metrics-poll.sh loops sample → sleep 15s, and dispatch-poll.sh’s EXIT trap halts the VM (shutdown -h now) the instant ./run.sh returns — killing the sampler mid-sleep. So the last interval, including “Complete job” and the post steps, never gets sampled.

Confirmed on job 84735487285:

time
last metric sample 09:25:19
last step end 09:25:28
job completed 09:25:32

A ~13s tail (one interval) with no data.

How

  • metrics-poll.sh: factor the per-iteration sample+POST into an emit_sample function and add a --once mode (one sample, no loop).
  • dispatch-poll.sh: after ./run.sh exits, capture its rc, run metrics-poll.sh --once (network still up, before the VM halts), then exit with the runner’s rc.

Best-effort — it never affects the runner’s exit code, and a failed sample just leaves the tail as-is. The one-shot has no previous counters so its network deltas report 0, which is fine since end-of-job throughput is ~0.

Validation

  • shellcheck clean on metrics-poll.sh; dispatch-poll.sh shows only its two pre-existing warnings (untouched). Both pass bash -n.

Scope

  • macOS runner image only (this is where the observed gap is). The Linux kata sidecar has a similar, smaller tail on pod termination and can be handled separately.
  • Takes effect for VMs booted from the next runner-image roll.

This is the follow-up to the now-merged runner-metrics work (#11595: axis alignment + 8h token TTL + clock sync).

🤖 Generated with Claude Code

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments

No GitHub comments yet.