cluster-api-provider-tuist: macOS machine deletion strands tart pods in Terminating (vm-cleanup finalizer never removed)

Summary

When a ScalewayAppleSiliconMachine is deleted (fleet churn, scale-down, MHC remediation), its tart pods can be left stuck in Terminating forever: the tart-kubelet.tuist.dev/vm-cleanup finalizer can only be removed by the tart-kubelet on that host, and the host is gone. Nothing else ever strips it.

Observed in production on 2026-07-16: the runners fleet churned and 4 pods in tuist-runners got wedged, all PodGC-deleted at the same instant (DisruptionTarget: DeletionByPodGC — "PodGC: node no longer exists"), all bound to Node objects that no longer exist. kubectl delete on them hangs indefinitely (the watch-stream INTERNAL_ERROR people see while waiting is unrelated proxy noise).

Root cause chain

1. tart-kubelet adds tart-kubelet.tuist.dev/vm-cleanup to every pod it runs, and is the only actor that removes it after VM teardown (infra/tart-kubelet/internal/podagent/reconciler.go:38-46, removal at :1102-1108).
2. reconcileDelete in infra/cluster-api-provider-tuist/controllers/macos/scalewayapplesiliconmachine_controller.go:895-1000 releases the server (Stage 1), deletes credentials (Stages 2-3), and explicitly deletes the Node object (Stage 4, :969-979) because “the host is gone, so the kubelet can’t deregister itself” — but never touches the pods bound to that node.
3. PodGC then notices the missing node, marks the pods Failed, and issues a grace-period-0 delete — which hangs on the finalizer forever. The tart-kubelet reconciler comment at reconciler.go:198-199 describes this exact terminal state.

The same logic that justifies Stage 4 (the kubelet can’t deregister its Node) applies to the finalizer: the kubelet can’t strip it either.

Proposed fix

Stage 4.5 in reconcileDelete: after deleting the Node, list pods with spec.nodeName == machine.Name that carry the vm-cleanup finalizer and strip it (patch finalizers). There is no VM left to leak — the host has been released and reinstalled back into the Scaleway pool by Stage 1, which is precisely what the finalizer exists to protect against on a live host.

Belt-and-braces (optional, covers manual kubectl delete node and any non-CAPI node removal): a small periodic sweep in the same controller manager that strips the finalizer from pods where deletionTimestamp is set and the referenced Node is NotFound.

Notes:

• The finalizer string would be referenced from capt; either share a constant with tart-kubelet or duplicate it with a comment pointing at podagent.PodFinalizer.
• Needs RBAC for pods list/patch in the capt manager if not already present.
• ScalewayElasticMetalMachine (Linux) nodes run the standard kubelet and don’t use this finalizer — no change needed there, though the generic sweep is harmless if shared.

Alternatives considered

• Sweep in runners-controller: only covers RunnerPool pods; the builders/macos fleets run other tart pods (e.g. xcresult processor), which would still wedge.
• Standalone controller: works, but an extra deployable for a small, machine-lifecycle-coupled concern that capt already owns half of (it deletes the Node for the same reason).

Remediation for currently stuck pods

Three pods are still wedged in production tuist-runners (a fourth was already manually cleared):

tuist-tuist-runner-pool-macos-26-3-runner-0034e37b    (node ...mndbc-h5dkz, gone)
tuist-tuist-runner-pool-macos-26-4-1-runner-2156b082  (node ...mndbc-k6kwt, gone)
tuist-tuist-runner-pool-macos-26-6-runner-5558f001    (node ...mndbc-hz6vv, gone)

Manual clear: kubectl -n tuist-runners patch pod <pod> --type=merge -p '{"metadata":{"finalizers":null}}'

Safe because the hosts were released back to the Scaleway pool and reinstalled — there is no VM to orphan.

No GitHub comments yet.