fix(infra): macos-fleet resilience — recover Unknown nodes + unwedge xcresult processors (tailscale/NAT)

Three resilience fixes for the macos-fleet (tart-kubelet + Tart VMs), all surfaced by the 2026-06-26 incident where the xcresult processors silently stopped consuming for hours and a kubelet-restart storm stranded ~9 nodes in Ready=Unknown.

1. tart-kubelet: recover nodes stuck in Unknown after a heartbeat gap

nodeagent.refresh() snapshotted the desired status (Ready=True, capacity, nodeInfo) into the node object, then client.Patch overwrote that in-memory object with the API server’s response — which, once the node-lifecycle controller had flipped Ready to Unknown after a heartbeat gap, meant the subsequent Status().Update() just re-posted Unknown with a fresh timestamp. The node heartbeated forever but never climbed back out of Unknown.

Latent since the dynamic-label persistence landed; it only bit when a node got marked Unknown, which a single restart usually avoids. Today’s synchronized restart of all ~11 kubelets (the operator rolls the binary to every host at once) pushed many nodes past the grace period simultaneously → Unknown → and they could never recover.

Fix: snapshot desiredStatus before the Patch and restore it after, so the status update actually asserts Ready=True. New TestRefreshLiftsNodeOutOfUnknown fails without the fix (node stays Unknown) and passes with it.

2. xcresult boot chain: bound tailscale up so a tailnet hiccup doesn’t silently wedge the VM

The processor’s launchd boot chain hard-ANDs inject-env.sh && source env && tailscale-up.sh && exec tuist start. tailscale up had no timeout, so when tailscaled couldn’t reach the control plane it blocked indefinitely → tuist start never ran → no BEAM, zero consumers — yet the pod still showed Running/Ready (tart-kubelet has no container probe, so “Ready” only means the VM booted).

Fix: tailscale up --timeout=60s. On timeout it exits non-zero, the chain fails, and launchd’s KeepAlive restarts it — so once NAT/control recovers a retry succeeds and the release boots on its own, instead of stranding the VM until a manual recycle.

3. host NAT: assert the general-internet leg from the durable anchor

The actual root cause of the processor outage: after heavy VM churn, InternetSharing’s en0 NAT anchor stopped translating VM egress, so VMs sent packets to the internet with their private 192.168.64.x source. The upstream gateway dropped them → in-VM tailscaled’s SYNs to control got no SYN-ACK (SYN_SENT forever) → tailscale up hung (see #2). Confirmed at the packet level (tcpdump on en0 showed un-NAT’d private sources leaving the host).

tuist-pf-vmnat (re-run every 60s) already owns the tailnet + PN-VLAN NAT legs but left the general-internet leg to vmnet/InternetSharing — exactly what broke.

Fix: assert the default-route NAT leg from the com.apple/tuist.vmnat anchor too. That anchor is proven-enforced (the cache legs ride it) and reloaded every 60s, so a clobbered InternetSharing NAT now re-converges within a minute. Also hardened the idempotency short-circuit to reload when the live anchor is empty, so an external flush isn’t masked by a still-matching snapshot. New TestRenderVMNATScript_AssertsDefaultRouteNATLeg covers it.

Validation

• go build, go vet, gofmt, and the full macos-host-bootstrap + tart-kubelet test suites pass.
• tailscale-up.sh passes sh -n.
• During the incident, recycling each VM (which re-triggers the host NAT init) restored service — verified the fresh VMs egress NAT-translated, join the tailnet, boot the BEAM, and resume draining the process_xcresult queue on both nodes.

Deploy paths differ (note for the reviewer)

These land in one PR for review, but reach hosts via different mechanisms: the tart-kubelet fix ships in the kubelet binary roll; the tuist-pf-vmnat change requires re-running macos-host-bootstrap on the hosts; the tailscale-up.sh change requires rebuilding the xcresult-processor Tart image.

Still open (separate, larger)

A real readiness signal for the processor: tart-kubelet has no container-probe support, so a readinessProbe: would be silently ignored — the fix is either implementing probe support in the kubelet or an Oban-stall alert (available jobs but no attempted_at for N minutes). Tracked separately.

🤖 Generated with Claude Code

No GitHub comments yet.