fix(infra): clone macOS runners from a golden base instead of re-pulling per job

What changed

macOS runner provisioning gets two coordinated changes so a single-shot VM recycle becomes a local clone instead of a multi-GB cold pull, and so Pods actually land on the hosts where that’s possible.

1. Clone runners from a per-host golden base (not a per-recycle pull)

tart-kubelet now provisions each macOS runner VM by cloning from a per-host, per-digest golden base VM (a local APFS clonefile, zero network) instead of tart pull <oci-ref> + tart clone <oci-ref> on every recycle.

• reconciler.go — createPod calls ensureGolden(image) (pull + materialize once per image digest per host, guarded by a per-digest lock) and then tart clone <golden> <vmName>. A clone failure drops the golden so the next reconcile re-materializes; cold-path disk pressure falls back to aggressive GC reclaim.
• garbage.go — the GC keeps each pod-image’s golden in expectedSet (so it is never reaped as an orphan clone) and retains unreferenced goldens for GoldenRetention (24h) so an idle host clones the next burst from its golden rather than re-pulling. RunOnceReclaim evicts goldens aggressively only under genuine disk pressure.
• metrics.go — new tart_kubelet_golden_base_materialized_total{pool}: flat in steady state, ticks once per host on a digest roll.

2. Steer Pods toward hosts that already hold the golden

A golden only avoids a cold pull if the Pod lands on a host that has it. Today every macOS Xcode pool shares one tuist.dev/fleet nodeSelector with no image awareness, so the scheduler treats all hosts as fungible — a 26.3 job can land on a host that has only ever served 26.5 and pay a full materialize even while another idle host holds the 26.3 golden.

• tart-kubelet publishes which golden bases a host holds as tuist.dev/golden-<digest-hash> Node labels, driven off tart list (the on-disk truth, so they self-heal across restarts and GC evictions). The node maintainer gains a generic DynamicLabels provider that merges and prunes them like operator NodeLabels; the provider masks transient tart list errors with its last good result so a hiccup doesn’t flap the labels off.
• The runners-controller adds soft node-affinity (preferredDuringScheduling, weight 100) toward the label key for the pool’s image. Preferred, not required: with no warm host free a Pod still schedules onto a cold one and pays the one-time materialize rather than going Pending. macOS only; Linux kata pools are untouched.

The label key is the same 8-byte SHA-256 prefix of the image ref the golden VM name embeds, derived independently in each module (they are separate Go modules). A pinned cross-module test on both sides guards the contract; a drift fails a test rather than silently disabling affinity — which itself degrades safely to today’s image-blind placement.

Why

The macOS runner fleet (9 hosts) routinely ran only ~4 builds concurrently while the queue held 6-8. It was not an autoscaler cap — the autoscaler correctly sets desired = claimed + queued and hands out the full 9-slot host budget. The bottleneck was provisioning: each single-shot VM halts after one job, and bringing up its replacement took a median ~7.4 min.

Breaking that down with production metrics:

• tart_kubelet_pod_provision_delay_seconds (pod creation → tart run: scheduling + pull + clone) p50 ≈ 441s, and flat at ~450s across 24h including the overnight window when the fleet was nearly idle — which rules out host-slot-wait (that would collapse to ~0 when free hosts exist).
• tart_kubelet_vm_boot_duration_seconds (tart run → IP) is only 5-9s, so boot and the host are fine.
• Decisive: the macOS minis ingested 14.4 TB over 24h across 349 provisions = ~41 GB per recycle (node_network_receive_bytes_total{job=\"tuist-macos-node-exporter\"}), each of the 9 runner minis pulling 325-464 GB per 3h. A macOS+Xcode image is ~40 GB, so the image was being re-downloaded on essentially every recycle.

Root cause

No image reuse across recycles. Every clone was from the OCI ref (re-materialize), and the OCI cache did not survive the no-Pod recycle gap — the GC kept images only while a Pod referenced them, on a 5-min interval. So a single-shot recycle paid a full ~40 GB pull. And even with goldens, image-blind scheduling would re-pay the materialize whenever a minority Xcode version got shuffled onto a fresh host.

Why this solution

Cloning from a pinned, already-materialized local golden VM is a guaranteed APFS clonefile (sub-second, zero network), whereas cloning from the OCI ref re-materializes every time and tart pull re-downloads whenever the cache lapses. The 24h retention lets the golden survive idle troughs so the morning burst clones instantly. Image-aware placement then keeps each Xcode version packed onto the hosts that hold its golden, so cold materializations are rare and localized to the (Xcode, host) pairs actually used, and the retention has far less work to do.

Impact

• Provision delay ~7 min → seconds.
• ~14 TB/day of redundant image pulls eliminated.
• Steady-state concurrent macOS builds ~4 → ~9 (the independent 2-VMs-per-host lever can push this toward ~18 on top).

Rollout ordering

Order-independent. The affinity references node labels that only exist once tart-kubelet publishes them; until then the soft affinity is a no-op and placement is exactly as today. The feature fully activates once both the controller and tart-kubelet are out.

How to test locally

This is host-level tart-kubelet behavior (drives the local tart CLI on a Mac mini) plus controller pod-template construction, so the substrate behavior cannot be exercised in CI without a Tart host. Validated by:

• go build ./..., go vet ./..., gofmt -l (clean), and go test ./... in both infra/tart-kubelet and infra/runners-controller — all green, including the new golden-base, golden-retention, node-label merge/prune, golden-affinity, and the pinned cross-module label-key tests.

Post-deploy verification on the fleet:

1. rate(tart_kubelet_golden_base_materialized_total[5m]) drops to ~0 between image rolls.
2. tart_kubelet_pod_provision_delay_seconds p50 falls from ~450s to seconds.
3. macOS ingress (node_network_receive_bytes_total{job=\"tuist-macos-node-exporter\"}) drops toward near-zero between rolls.
4. max by (fleet)(tuist_runners_claims_count{lifecycle_state=\"running\", fleet=~\".*macos-26-5\"}) steady-state running climbs from ~4 toward ~9.
5. Mac mini nodes carry tuist.dev/golden-<hash> labels matching the Xcode images they have served (kubectl get nodes -l kubernetes.io/os=darwin --show-labels).

No GitHub comments yet.