Kura reconciler can’t self-heal a manifest-only regression (silent manifest change deadlock)

Summary

The Kura server reconciler (Tuist.Kura.Reconciler) keys convergence on only two things — the observed image tag and the @manifest_revision constant. A change to the rendered manifest that bumps neither cannot self-heal an already-provisioned instance. On 2026-07-01 this wedged kura-tuist-eu-central-1 for ~6 hours (16:00→22:00Z) with no public endpoint, and it could only be recovered by forcing a Kura runtime image-tag change.

What happened (incident)

• #11605 (5e77e0ec60, “use the shared regional gateway for bare-metal Kura regions”) changed gateway_assignment/2 so bare-metal (host-network) regions use the shared kura-eu-central ingress class instead of a per-account kgw-<acct>-eu-central gateway that can never bind :80/:443 on the single dedibox node.
• It changed the rendered manifest (the instance’s ingressClassName, and it stopped emitting a dedicated KuraGateway) but did not bump @manifest_revision — a silent manifest change.
• The already-provisioned kura-tuist-eu-central-1 stayed on the dedicated (unschedulable) gateway class. Its ingresses had no running controller → no address → external-dns published nothing → tuist-eu-central-1.kura.tuist.dev was NXDOMAIN → activation looped ([Kura.Reconciler] waiting on DNS ... :nxdomain) for ~6h.
• Deploying the server binary with the #11605 fix did nothing — both convergence keys (image 0.10.15, revision 2026-06-19-...) were unchanged, so the reconciler considered the instance fully converged.
• What finally healed it: a Kura runtime image-tag change (0.10.15 → sha-5e77e0ec60c9, same content, new tag). That gave the image-keyed rollout path a mismatch → apply_deployment → Provisioner.rollout re-rendered under #11605 → shared class, ingresses bound, DNS published, /up 200, :active.

Root cause: the reconciler can’t heal a manifest-only regression

• Rollout fast path (reconcile_deployment) is image-keyed only: if observedImage == deployment.image_tag it goes to activate_and_mark_succeeded (probe only) and never re-renders.
• Manifest-revision drift check lives in the projection pass (project_server → reconcile_manifest_revision → apply_current_manifest), but that pass is skipped whenever the server has an open :pending/:running deployment.
• A server wedged in the activation loop (stuck :running deployment, endpoint never /up) is therefore blocked from both re-apply paths. Neither a plain server redeploy nor a @manifest_revision bump reaches it — only an image-tag change (or manual surgery) breaks out.

Proposed fixes

1. [preferred] Derive @manifest_revision from a content hash of the rendered manifest inputs (instead of a hand-maintained string), so any rendered-output change automatically shifts the revision and the projection drift path re-applies. This is the primary fix — it makes silent manifest regressions impossible. At minimum, add a CI guard that fails when the renderer changes but the constant doesn’t.
2. Teach reconcile_deployment to re-apply on manifest-revision drift, not just image drift, so a wedged (:running) instance can heal without an image change.
3. GC unassigned gateways inside Provisioner.rollout — a dedicated→shared transition currently leaves the old KuraGateway CR behind (Pending, with perpetually-unschedulable controller pods); only destroy deletes gateways today.
4. /ops “force re-apply” button — insert a kura_deployments row with a new force flag honored by reconcile_deployment, giving a one-click manual override for a wedged instance (with audit trail via the existing deployment history). Note the button must bypass the image-equality gate; a plain same-image insert re-enters the deadlock.

Immediate recovery runbook (for the next time an instance is wedged)

• Force an image-tag change (Kura runtime bump), or
• Surgical patch: set spec.ingressClassName on the KuraInstance + its ingresses to the correct shared class and kubectl delete kuragateway <stuck>, or
• kubectl delete kurainstance <name> to force the :not_found → apply_deployment re-render path.

Cleanup still pending from this incident

Orphaned kgw-799b0dea3467-eu-central KuraGateway (Pending, + 2 unschedulable controller pods) — kubectl -n kura delete kuragateway kgw-799b0dea3467-eu-central.