fix(infra): tart-kubelet GC deletes the live runner image; add golden-base reuse observability

Context

Follow-up to #11484 (golden-base VMs, now merged). After that landed, the macOS runner fleet was still re-pulling per provision: tart_kubelet_golden_base_materialized_total / pod_provision_delay_seconds_count ≈ 0.98 (≈one cold pull per provision) and provision-delay p50 stuck at ~450s, with the hot pool macos-26-5 doing ~36 materializations/2h while idle pools (26-3, 26-4-1) reused their golden fine (1 each). The golden base wasn’t delivering.

I investigated on a production host (mndbc-hz6vv, SSH over the tailnet) and ruled out the obvious culprits: 200–348 GB disk free + 86% memory free (no disk-pressure eviction, no OOM), only 14 tart-kubelet restarts since May, golden VM present on disk, tart get <golden> succeeds, and the golden name matches sha256(image)[:8]. So eviction / capacity / OOM / restart / name-mismatch are all out.

What this PR fixes

1. GC was deleting the live runner image every pass (root cause of the expensive cold path)

On the host, tart list showed zero cached OCI images, and the log repeated every ~15 min:

gc "delete stale OCI cache entry" image="ghcr.io/tuist/tuist-runner@sha256:9fac136a…"
gc "reclaimed disk" stale_oci_images:1

Root cause: expectedSet.images was keyed by the Pod’s image ref — a tag (ghcr.io/tuist/tuist-runner:macos-26-5-0.7.0). But Tart caches and tart lists images by their resolved digest (…@sha256:…). The tag↔digest comparison never matched, so the GC marked the live image stale and deleted it on every pass. With the cache gone, every golden (re)materialization paid a full multi-GB re-pull instead of the sub-second APFS clonefile the golden scheme is built around — exactly the cost #11484 set out to eliminate.

Fix: key expectedSet on the image repository (ociRepository() strips tag and/or digest, preserving a registry port) and match the cached entry’s repository against it. The aggressive disk-pressure RunOnceReclaim path still reaps OCI images regardless of the match — the cache is reconstructible by re-pull, so a genuine out-of-disk provision must win, and this bounds the superseded-digest tail a long-lived repository would otherwise accrue.

Why repository-match over digest resolution: matching tag→digest would require a registry round-trip (auth, network) on every GC pass; the repository is the stable key both refs already share, and a golden VM is the durable artifact anyway, so the OCI cache only needs to survive as a re-materialization shortcut.

2. Observability to close the remaining question

The deeper question — why a cold path is entered at all when the golden is present — was unobservable: a tart get that errors and falls through to a cold re-pull looks identical to a legitimate first-sight materialization, and pod_provision_delay_seconds (measured from Pod creation) folds scheduling/queue wait into the same number as the pull. Added:

• tart_kubelet_golden_base_reused_total{pool} — warm clonefile hits. reused/(reused+materialized) is the golden hit rate.
• tart_kubelet_vm_provision_work_seconds{pool,path=warm|cold} — the on-host pull/clone segment, isolated from queue wait and split by warm vs. cold.
• an error log in ensureGolden when tart get fails for a reason other than not-found.

Together these will show, post-deploy, whether the ~450s is a genuine re-pull (path="cold" rate high) or single-VM-per-host queue wait (vm_provision_work low while pod_provision_delay stays high) — without SSHing a host.

Impact

• Stops the ~every-15-min deletion of the live runner image, so a golden re-materialization becomes a clonefile from a warm OCI cache rather than a ~70 GB re-download.
• Makes the golden-base hit rate and the pull-vs-queue split first-class metrics.
• No behavior change for warm-path provisions (they already clone from the golden without the OCI cache).

Validation

• go build ./..., go vet ./internal/podagent/, gofmt -l, and the full module test suite pass.
• New tests: TestExpectedSetMatchesOCIImageByRepository (regression — a tag-referenced Pod image keeps the digest-listed cache entry) and TestOCIRepository (tag/digest/port parsing).

Note

This does not by itself prove the median-provision warm-path miss is gone — fix #2 is what closes that loop once deployed. Fix #1 is an unambiguous correctness fix regardless.

🤖 Generated with Claude Code

No GitHub comments yet.