fix(infra): auto-recover silently-locked-up bare-metal Kura nodes so they stop wedging deploys

What & why

A silent kernel lockup on a single-box bare-metal Kura region (Dedibox / OVH / Scaleway Elastic Metal) leaves its Node NotReady indefinitely, which silently wedges every production deploy.

Mechanism: the default kernel.panic=0 never reboots a frozen box, and a self-joined node with a foreign providerID is never GC’d. A stuck node keeps its DaemonSet slot, so the observability node-exporter DaemonSet can never reach full availability (Available: 8/9), the observability chart’s helm upgrade --wait times out at N-1/N, the canary job fails, and the whole production cascade fails before it reaches prod — unrelated to whatever is being deployed.

This just recurred twice within two days on the same canary Dedibox box (run 28572967459 and the newer main-push run 28575415298), each time with no auto-recovery — a human had to IPMI-reset the box. This PR makes the box recover itself.

Root cause

Two gaps compound:

1. A frozen kernel with panic=0 and no watchdog never reboots — it sits dead forever.
2. There is no auto-remediation for the in-house CAPI provider’s Linux bare-metal fleets. The caph/Hetzner ClusterClass has MachineHealthChecks; the Dedibox/OVH/Scaleway-EM fleets have none, so nothing notices a dead box.

The fix — two layers

1. Bootstrap hardening (primary, non-destructive self-heal) — infra/cluster-api-provider-tuist/controllers/linux/linux_cloudinit.go

In the shared bootstrapBody (so every Linux fleet, and both the cloud-init and SSH render forms, get it):

• systemd hardware watchdog: /etc/systemd/system.conf.d/10-tuist-watchdog.conf (RuntimeWatchdogSec=30s) + daemon-reexec.
• panic sysctls: /etc/sysctl.d/99-tuist-hardening.conf (kernel.softlockup_panic / hardlockup_panic / panic_on_oops=1 + panic=10).

A detected lockup becomes a ~30s auto-reboot; the hardware watchdog catches a total freeze that starves the softlockup detector. Applied tolerantly (|| true) so a missing knob or an absent watchdog device can never abort the self-join.

Crucially, a reboot preserves the warm cache: /var/cache/kura is a local-path PVC on the box’s persistent /data NVMe disk, and the StatefulSet pod is pinned by the local PV’s nodeAffinity, so it re-mounts the same store on the same node after the reset. (netconsole from the original prevention plan was dropped — it needs an off-box log receiver we don’t have; a target-less netconsole is a no-op.)

2. MachineHealthCheck per fleet (backstop) — infra/helm/tuist/templates/{dedibox,ovh,kura}-fleet.yaml

A standalone cluster.x-k8s.io/v1beta1 MachineHealthCheck (the workload cluster where these fleets’ Machines live serves only core v1beta1 — v1beta2 is on the separate caph mgmt cluster) remediates a Node NotReady for 10 continuous minutes — covering a box that can’t self-reboot (dead watchdog / hardware fault). Applies to existing boxes immediately once deployed.

Why this shape (and the trade-off to review)

The provider has no reboot API for Dedibox/OVH (Scaleway EM has one but it’s unwired); reconcileDelete is a destructive wipe-and-reinstall that re-adopts the same box, with no auto-order. So MHC remediation is a ~15-30min reinstall that wipes the (regenerable, reclaimPolicy: Delete) cache. That is exactly why the caph bare-metal ClusterClass deliberately uses unhealthyInRange: "[0-0]" (operator investigates).

Chosen anyway, with guardrails: unhealthyRange: "[0-1]" remediates a single dead box but never mass-reinstalls a whole region on a cluster-wide blip, and nodeStartupTimeout: 30m avoids remediating a box that’s still provisioning. The layering means the destructive reinstall is a genuine last resort: Layer 1 self-reboots the common case in ~30s with the cache intact; Layer 2 only fires when self-reboot fails. Silently wedging every deploy forever is strictly worse than a rare reinstall. If you’d prefer the MHC start opt-in / alert-only, it’s a small change — flag it.

Layer 1 protects boxes provisioned after this change; Layer 2 protects existing boxes immediately. helm --wait does not track MachineHealthCheck (a custom kind), so the added CR cannot itself wedge the deploy — only a schema-invalid CR could, which is why the schema was validated.

Impact

• Bare-metal Kura regions self-recover from a silent lockup in ~30s (cache preserved) instead of sitting dead until a human IPMI-resets them.
• Production deploys stop being gated by a single frozen bare-metal box.
• Forward-looking hardening applies on each box’s next (re)provision; the MHC applies to existing boxes on deploy.

Note: this fix cannot deploy itself until the currently-frozen canary box is unblocked (it’s gated behind the same observability wedge). The immediate unblock (IPMI reset, or kubectl delete node) is still a manual step; this change prevents recurrence.

How to test locally

• cd infra/cluster-api-provider-tuist && go test ./controllers/linux/ — includes a new TestRenderLinux_LockupHardening asserting the watchdog + panic sysctls land in both render forms.
• Render the fleet templates and confirm each emits a MachineHealthCheck alongside its MachineDeployment: helm template tuist infra/helm/tuist -f infra/helm/tuist/values-managed-common.yaml -f infra/helm/tuist/values-managed-canary.yaml --show-only templates/dedibox-fleet.yaml

Validation run

• gofmt/go vet clean; go build ./... and the linux package tests (incl. the new hardening test) pass.
• All three MHCs render clean via helm template; the v1beta1 MHC schema was confirmed field-by-field against the served CRD with kubectl explain machinehealthcheck.spec --api-version=cluster.x-k8s.io/v1beta1.

No GitHub comments yet.