fix(infra): rebind running VMs after a kubelet restart instead of looping on “already running”

What this fixes

A tart-kubelet restart could strand long-lived VM workloads — concretely the xcresult-processor, which wedged at 0/2 and blocked a production deploy (helm --wait on an --atomic release, which then risks rolling the whole tuist release back).

The VM survives a kubelet restart on purpose (tart run is Setsid-detached). But the restarted kubelet failed to rebind it, so createPod called tart run on the already-running VM, got VM is already running (exit status 2), and looped every ~30s. The Pod never reported Ready:

error: tart run …xcresult-processor…: exited immediately: exit status 2
/var/log/tart-vms/…: VM "…xcresult-processor…" is already running!

Root cause

Two issues, both fixed:

1. recoverState burned its budget probing stopped goldens. It probed every local VM with tart ip (which blocks up to 30s waiting for an IP a stopped VM never gets) under a single shared 30s context. The golden-base feature added stopped golden VMs to tart list, so one golden could exhaust the whole 30s before the actually-running workload VM was reached — leaving it unbound. So the golden-base work regressed restart recovery. Switched to IsRunning (a fast pgrep — the same canonical liveness signal podStatus already uses): instant, and false for stopped goldens/clones (which createPod still re-runs as before).

2. createPod re-ran an already-running clone. As defense-in-depth, it now adopts a running clone — registers a Store entry with no RunHandle, exactly like a recoverState-recovered entry (podStatus tracks liveness via IsRunning) — instead of tart run-ing it. A missed recovery now self-heals on the next reconcile rather than looping forever.

Why it surfaced now

Not caused by the runner OCI/Size fixes — it’s a pre-existing restart-recovery gap. But those PRs’ back-to-back operator rolls restarted tart-kubelet on the macOS fleet several times, and each restart re-orphaned the xcresult-processor VM, finally wedging a deploy.

Impact

tart-kubelet restarts (every operator roll) become non-disruptive to long-lived VM workloads: the running VM is rebound at startup and the Pod stays Ready, so deploys stop hanging on the xcresult-processor.

Validation

• go build, go vet, gofmt, full module test suite pass.
• No unit-test seam: both paths use the concrete tart.Client and pgrep/tart shell-outs (consistent with the existing untested recoverState/createPod shell paths). Plan to verify on a host post-deploy: restart tart-kubelet with a running VM present and confirm recovered VM state … matched_pods≥1 and no already running loop.

🤖 Generated with Claude Code

No GitHub comments yet.