fix(infra): give bare-metal Kura fleet MHCs the full OS-reinstall window (30m→90m)

What changed

Raised nodeStartupTimeout from 30m to 90m on the three bare-metal Kura fleet MachineHealthChecks in the tuist Helm chart — dedibox-fleet.yaml, ovh-fleet.yaml, and kura-fleet.yaml (Scaleway Elastic Metal) — and replaced the misleading “generous” comment with the actual failure-mode reasoning.

Why (root cause)

Canary server deploys were wedged for 32h+. The failing step is helm upgrade --install, which uses --atomic --timeout 60m --wait. Helm’s --wait blocked on:

resource MachineDeployment/tuist-canary/tuist-tuist-dedibox-fleet not ready.
status: InProgress, message: Minimum availability requires 1 replicas, current 0 available
context deadline exceeded

The tuist-tuist-dedibox-fleet MachineDeployment was stuck ScalingUp 0/1 for 2d+ while every other fleet was healthy. The server pod itself was fine (1/1 Running) — the early /ready connection-refused messages were just normal boot.

The MHC set nodeStartupTimeout: 30m, but a Scaleway Dedibox / OVHcloud OS reinstall takes 45-60 min for sshd to come up. So every 30 min CAPI marked the still-installing box unhealthy → the provider (whose only release path is wipe-and-reinstall) reinstalled it → the ~50 min clock restarted from zero. The box never got the time it needed to join. Live event cadence confirmed the loop:

Adopted Dedibox server 133116
  → 30m → MachineMarkedUnhealthy
  → ReleasedToPool "Reinstalling Dedibox server 133116 to a clean, claimable state"
  → Adopted (next machine) → repeat

Because the fleet MachineDeployment is in the chart’s --wait set, this wedged the entire server rollout even though the server was healthy.

Why this fix (over alternatives)

nodeStartupTimeout only gates the initial startup / reinstall window — a box that dies after joining is still remediated by the unhealthyConditions (600s / 10 min) below it. So raising it to 90m has no steady-state downside; it purely gives a provisioning or reinstalling box enough time to finish before the health check gives up and triggers another wipe. 90m clears the 45-60 min worst case with margin.

All three bare-metal fleets share the identical template and the same latent bug, so all three are bumped for consistency (OVH has the same 30m and happened to converge this time; Elastic Metal has run fine but is the same wipe-and-reinstall model).

Not chosen: excluding MachineDeployments from helm --wait (bigger, riskier design change), or setting the fleet to replicas: 0 (the chart already has that escape hatch for a not-yet-prepped fleet, but it wouldn’t fix the reinstall loop — the box would still never converge).

Impact

• Breaks the reinstall loop so the Dedibox box can finish installing and join, unwedging canary deploys.
• No effect on remediation of a box that fails after joining (still 10 min).

Validation

• helm template renders all three fleet MHCs with nodeStartupTimeout: 90m and valid YAML for the canary values.
• Root cause verified against live canary cluster state (read-only): MachineDeployment ScalingUp 0/1 for 2d+, MHC nodeStartupTimeout=30m, and the Adopted→MarkedUnhealthy→ReleasedToPool 30-min reinstall cadence.

kubectl patch machinehealthcheck tuist-tuist-dedibox-fleet -n tuist-canary \
  --type merge -p '{"spec":{"nodeStartupTimeout":"90m"}}'
# wait ~50m for Dedibox 133116 to finish installing and the Node to go Ready (MD 1/1)
gh run rerun --failed 28647029868

🤖 Generated with Claude Code

No GitHub comments yet.