Never let a stuck deployment block the next Kura version

Summary

The Kura version scheduler refuses to schedule a new runtime image for a server while any deployment row for that server is open (:pending or :running). A deployment stays open for as long as its instance takes to converge, and several convergence-wait states are unbounded by design, so an instance stuck converging holds its deployment open indefinitely and every future release queues behind it. The circular dependency is the sharp edge: the release carrying the fix for a wedge cannot reach the instance because of the wedge.

This spec removes that coupling with two mechanisms. Newest intent wins: when a newer runtime tag is released, any open deployment for the server is closed as :superseded and the new tag is scheduled in the same reconciler tick, making a release always schedulable. Failing replicas roll together: on an image change, the Go controller proactively deletes the instance’s not-Ready pods so they recreate on the new revision in parallel, instead of receiving the fix one readiness-gated ordinal at a time. Control-plane intent becomes independent of data-plane convergence, and the pods most in need of a fix are the first to get it.

Failed (closed) deployments do not block newer releases and are out of scope here, as is re-attempting a failed tag; those are a separate concern.

Motivation

The 0.19.0 deployment on kura-tuist-eu-central-1 stayed open for over 14 hours (2026-07-21 21:59 to 2026-07-22 12:06) while the instance ground through a bootstrap wedge behind the /ready gate. Any release shipped in that window would have queued behind it, including a release carrying a fix for the wedge itself. The diagnostic gauges built to confirm the wedge hypothesis (PR #12018) were in exactly this position: instrumentation for a stuck instance, blocked by the stuck instance.

The window is not bounded at 14 hours. The reconciler’s activation path deliberately loops several wait states forever: public endpoint not ready (the workload is up on the new image but still replicating behind the bootstrap gate, surfaced as :replicating), DNS not yet propagated, node-port chain not yet observed. Those waits are correct for the deployment itself, but a pathological convergence, such as the non-terminating bootstrap re-walk observed in the same incident, holds the deployment open without limit, and with it the entire release pipeline for that server.

Kura is on the critical path for cache reads and writes, and the instances most likely to be stuck converging are precisely the ones most in need of the next release.

Current state

• servers_needing_version_query (server/lib/tuist/kura.ex) excludes a server from version scheduling while any deployment row for it is in :pending or :running (the open_deployment_exists exclusion).
• A deployment closes on success only end to end: the backing KuraInstance reports the target image and Kura.activate_server/2 confirms the regional public endpoint answers /up. The wait states on that path (public_endpoint_not_ready, public_host_not_resolvable, node_port_endpoint_not_ready) leave the deployment :running and retry next tick, with no lifetime bound.
• There is no supersede notion: the deployment status machine (pending, running, succeeded, failed, cancelled) has no terminal state for “displaced by a newer release”, and the scheduler has no path that closes an open row.
• The backing workload is a StatefulSet with ParallelPodManagement (infra/kura-controller/controllers/kurainstance_controller.go) and the default RollingUpdate strategy with no partition. Parallel pod management affects scaling only; updates proceed one pod at a time, highest ordinal first, and each updated pod must become Running and Ready before the next ordinal is touched. A pod that is not on the update revision is deleted regardless of its own readiness, so the new image does reach a failing top-ordinal replica immediately, but every replica below the first updated-but-still-not-Ready ordinal waits behind the readiness gate. With several not-Ready replicas, the fix propagates one gated ordinal at a time, and stalls entirely if the first updated pod stays not-Ready.
• The warm-handoff exclusion (move_phase == :none) also defers new releases, deliberately and boundedly: a :moving_in target must warm on the source’s image, and the promoted row re-enters the scheduler on the first tick after the move completes. That guardrail stays as is.
• Deliberate properties worth preserving: version rollouts must reach degraded servers, and a still-serving endpoint is never trampled by status projection.

Proposal

Newest intent wins: supersede open deployments

When the scheduler holds a newer released tag than an open deployment’s image_tag, it closes the open row with a new terminal status, :superseded, and schedules the new tag for that server in the same tick. The open-deployment exclusion keeps existing solely to serialize work on one server within a single release, not across releases.

Safety follows from the architecture as it already is:

• The Go controller is level-based: the KuraInstance spec is desired state. Re-targeting a mid-roll instance to a newer image is a native Kubernetes rolling-update re-target; pods that rolled onto the superseded image simply continue rolling to the new one.
• Kura’s rollout-safety rule already requires every adjacent version pair to interoperate on the peer and client protocols, so a mesh briefly mixing the superseded and new versions is inside the existing skew budget.
• Activation semantics are unchanged: the new deployment succeeds only when the instance observes the new image and the public endpoint serves, exactly as today. Supersede changes which image an open rollout converges toward; it never skips the convergence gate.
• A still-serving endpoint is never touched: superseding rewrites intent (the spec of the backing resource), and the endpoint only flips when the new revision is actually ready, same as any rollout.

Superseded rows stay in the deployment history with their status, so the audit trail records what was attempted and displaced rather than losing it.

Failing replicas roll together: delete not-Ready pods on image change

Supersede puts the new tag on the KuraInstance spec, but the StatefulSet’s ordered, readiness-gated update walk still delivers it to a degraded instance one ordinal at a time: after replacing the top not-Ready replica, the walk waits for that pod to become Ready before touching the next. An instance with several wedged replicas receives the fix serially, gated on each replica recovering, and not at all below the first replica that stays not-Ready on the new version too.

The change: when the Go controller observes an image change on the instance, it proactively deletes every pod of that instance that is not Ready. The StatefulSet controller recreates them on the new revision, and with ParallelPodManagement the recreations start in parallel. Ready pods are never touched by this path; they keep serving and roll through the ordinary one-at-a-time readiness-gated walk.

Safety:

• A not-Ready pod serves no load-balanced traffic (readiness is what gates routing), so deleting it costs nothing that is currently being provided. Its PVC and dataset persist across the recreation; only in-memory bootstrap progress is lost, and that progress was being made by the version the fix is displacing.
• The mesh briefly mixes versions across the recreated pods, which the one-version-skew rule already covers.
• The serving path is strictly protected: Ready pods only ever roll behind the readiness gate, exactly as today.

In the motivating incident this is the difference between the fix reaching one wedged replica at a time, gated on a bootstrap that may itself be the bug, and both wedged eu-central replicas jumping to the fixed version simultaneously while nothing serving was disturbed.

What this deliberately does not change

• Failed deployments. A closed :failed row already does not block newer releases. Whether and how a failed tag is re-attempted (the once-per-tag rule, operator retries) is a separate concern with its own trade-offs, out of scope here.
• The warm-handoff deferral. Supersede does not apply to servers mid-move; their bounded deferral stands.
• Wave pacing. Under the health-gated progressive rollout spec, supersede operates on whatever the pacing layer has scheduled: a newer tag displacing a paused or in-flight rollout’s open work is the same mechanism, applied at rollout scope.

Observability

• Count superseded closes per server and tag, and log each one with the displaced and displacing tags.
• Count proactive not-Ready pod deletions per instance and image change, so a rollout that leaned on the fast path is distinguishable from one that walked ordinals.
• The /ops deployment history keeps every row, including superseded ones, so an operator reading a server’s history sees the actual sequence of intent: scheduled, displaced, rescheduled, converged.
• A server accumulating superseded rows without ever converging is a new, visible signal: releases are flowing, the instance is not. That is precisely the situation that today hides inside one silently open deployment.

Interaction with the progressive rollout spec

Complementary, not competing. The health-gated progressive rollout spec decides when a healthy rollout schedules the next wave; this spec guarantees that when a new release needs to go out, the scheduler is never wedged shut by an open deployment from the previous one. Supersede is also the natural primitive for that spec’s expedited mode: an urgent fix displaces whatever the paced rollout had open.

Alternatives considered

• A TTL on open deployments instead of supersede. A lifetime bound still serializes the new release behind the wedged deployment until expiry, and pressures the TTL downward, toward misfiring on legitimate long convergences: the 14-hour bootstrap that eventually succeeded was healthy, and a TTL short enough to be useful would have killed it. Supersede needs no timer: the trigger is the thing that actually matters, a newer release existing.
• Raising maxUnavailable on the StatefulSet instead of deleting not-Ready pods. maxUnavailable for StatefulSets widens the update window for every rollout, healthy or not, trading steady-state availability for the degraded case. Deleting only not-Ready pods on image change is scoped to exactly the pods that are already providing nothing.
• Do nothing, ship fixes by destroying stuck servers. Destroying loses the PVC dataset and costs hours of cold re-bootstrap on a component whose entire value is having the data local, and it converts a routine release into an operator intervention.