Health-gated progressive rollout of Kura runtime versions

Summary

Today a new Kura runtime image reaches the entire managed fleet in a single reconciler tick: the control plane schedules a deployment for every eligible server at once, every backing KuraInstance rolls in parallel, and the only per-server health authority is the public /up probe. A version that boots but degrades under real traffic (replication backlog growth, peer-plane failures, memory pressure) lands on every account mesh at the same time.

This RFC makes the fleet rollout progressive and health-gated. A new runtime tag rolls out in ordered waves, starting with Tuist-owned accounts, and each wave must converge and then hold a health gate for a soak period before the next wave schedules. The gate consumes the per-node signals Kura already exposes on /status/rollout, surfaced to the control plane through the KuraInstance status that the Go controller already derives from polling those same endpoints. A wave that cannot pass the gate pauses the rollout and alerts instead of proceeding. An expedited mode preserves today’s all-at-once behavior for urgent delivery: rollbacks to a known-good tag and forward hotfixes both stay a minutes-scale operation.

Waves apply to the production fleet. The canary environment keeps today’s immediate fan-out, and its convergence becomes an explicit gate the pipeline checks before promoting to production. The production rollout itself runs off the deploy path: pinning the tag completes the deploy, and the reconciler converges the fleet asynchronously under Grafana’s watch.

Motivation

Kura is on the critical path for cache reads and writes. A bad runtime version is not a degraded feature, it is a fleet-wide incident, and the current rollout hands it the whole fleet in one tick.

Three observations shape the proposal:

1. /up attests liveness, not health. A node can answer /up with 2xx while its outbox backlog grows without bound, its file-descriptor waits time out under load, or it sits at critical memory pressure. The signals that distinguish “up” from “healthy” already exist in the /status/rollout report: readiness, traffic state, ring membership, bootstrap progress, outbox depth, file-descriptor wait timeouts, and memory pressure. The one signal it lacks, a peer-connection failure counter, this RFC adds.
2. A health-gated rollout primitive already exists but is not wired into the production path. The standalone chart ships a partitioned rollout driver whose gate waits for the whole cluster to hold steady on /status/rollout conditions, with a longer soak for the first (canary) pod. The managed KuraInstance path never consumes it.
3. The Go controller already polls /status/rollout on pod IPs every reconcile to pick the routable primary. The health surface is plumbed to the controller today; it is just not published upward or used for pacing.

The remaining work is orchestration in the Elixir reconciler, which already owns rollout intent in Postgres, plus a small additive status surface on the KuraInstance resource.

Current state

• The deploy pipeline resolves the highest kura@<semver> tag and pins it through Helm as TUIST_KURA_RUNTIME_IMAGE_TAG. An out-of-band image can already be delivered without waiting for a release train: ad-hoc runtime images can be built from any commit, and the deploy workflow accepts an explicit tag input that takes precedence over the resolved release tag.
• The reconciler cron fires every minute. Tuist.Kura.schedule_runtime_image_deployments/0 selects every server behind the configured tag (excluding only warm-handoff moves and terminal servers) and records a deployment for each in one pass. The per-tick ceiling of 200 is a runaway guard, not a canary batch.
• Deployment scheduling is once-per-server-per-tag for the lifetime of the fleet: a failed deployment for the configured tag is documented as intentionally not retried, and the manual retry path only accepts servers that never activated. A previously-active server whose deployment failed has no recovery path until the next Kura release.
• The provisioner patches each backing KuraInstance with the new image; the Go controller rolls the per-instance StatefulSet with the usual one-pod-at-a-time rolling update and disruption budget.
• A server is marked active only after its public endpoint answers /up.
• The server deploy pipeline cascades canary, then production, so the tag reaches the canary environment’s fleet before production’s. That sequencing is between environments; within the production fleet there is no ordering at all, and nothing in the pipeline observes whether either environment’s Kura fleet actually converged.

Deliberate properties worth preserving: version rollouts reach degraded servers (a fix must land on a :failed server, not only healthy ones), and a still-serving endpoint is never trampled by status projection.

Proposal

A rollout is control-plane state, advanced by the reconciler

The control plane gains a durable notion of “the rollout of tag X”: which wave it is in, when the wave started, since when the wave has been continuously healthy, whether it is running, paused, completed, aborted, or superseded, and why it paused. The reconciler advances that state one tick at a time, the same way it already converges servers and deployments; if the BEAM dies mid-rollout, the next tick observes the same state and continues. Scheduling changes from “record a deployment for every eligible server” to “record deployments for the current wave only, and only open the next wave when the gate has held for the soak period”.

Only production paces; canary stays immediate; neither blocks the deploy job on waves

Every environment mints a rollout record for a new tag, but only production paces it. The canary and staging environments run their rollouts in expedited mode, which is today’s immediate fan-out: their role in the cascade is to expose the new version to real usage as early as possible, and pacing them would only delay the signal they exist to produce.

The rollout record still buys something in canary: the pipeline’s production stage gains a promotion gate that waits for the canary environment’s rollout to complete, meaning every server converged on the new image and ready, which takes minutes. A canary rollout that is paused, aborted, or superseded blocks promotion. Production’s wave 0 therefore never starts on an image the canary environment has not actually run.

Production’s progressive rollout is deliberately off the deploy path. The deploy job finishes when the tag is pinned and the rollout record is created; it does not wait for waves. A green deploy means intent was recorded, not that the fleet converged; convergence and its failure modes are watched by the Grafana alerting described below, not by a CI job.

Waves are assigned per account, our own accounts first

Wave assignment is deterministic at rollout creation and grouped by account, so all servers of one account update in the same wave. An account mesh spans that account’s servers across regions plus its self-hosted peers; splitting one account’s managed servers across waves would hold cross-version skew open inside a single mesh for the whole rollout instead of one wave.

• Wave 0 (canary): servers belonging to Tuist-owned accounts, and only those.
• Wave 1: roughly 5 percent of accounts.
• Wave 2: roughly 25 percent of accounts.
• Wave 3: the remainder, with the highest-usage accounts last.

Runner-cache servers get no special wave placement: they are ordinary servers that follow their account’s wave, which also keeps an account’s mesh and its runner cache on the same version. Within waves 1 to 3, accounts order by recent Kura usage ascending, tie-broken deterministically, so lower-traffic accounts absorb earlier exposure. An empty wave (for example no Tuist-owned servers in an environment) advances immediately.

Servers created mid-rollout inherit their account’s wave state rather than jumping to the configured tag: until the account’s wave has completed, a new server provisions on the rollout’s baseline tag; after it, on the target. This keeps a paused-as-suspect version off fresh servers and keeps an account’s mesh on a single version ahead of its wave, which matters most for runner-cache servers since they take traffic the moment they exist.

Health reaches the control plane through the instance status

The Go controller already fetches /status/rollout from every ready pod each reconcile. It additionally publishes an aggregate on the KuraInstance status with explicit per-field semantics, because no single worst-of rule reproduces the standalone gate: readiness and serving state are conjunctions across pods, generation is an explicit all-consistent flag, bootstrap in-flight and the regression counters (outbox depth, file-descriptor wait timeouts, peer-connection failures) are sums with per-pod reset clamping, memory pressure is the maximum, and the sample timestamp is the oldest. The effect is that a sick standby drags the aggregate down, which is intentional: a standby that cannot bootstrap on the new image is exactly the regression signal the gate exists to catch.

One naming correction relative to the standalone gate’s framing: fd_timeout_count counts file-descriptor wait timeouts, which is fd-pool pressure, not peer failures. The gate keeps consuming it under its real name, since an fd-wait-timeout spike after an upgrade is a genuine regression signal. Peer-plane health, the thing the old framing suggested was being measured, gets a real signal instead: the rollout report gains a peer-connection failure counter, and the gate consumes both.

The Elixir reconciler already reads the backing resource per server each tick for the observed image and endpoint readiness; it reads the health aggregate through the same seam. The status addition is additive and chart CRDs are applied automatically on every deploy, so there is no migration step. The health surface never crosses the public gateway, and the sample timestamp lets the control plane treat stale samples as unhealthy rather than silently passing a frozen snapshot.

The gate measures regression, not absolute health

The gate is evaluated over every server updated so far in the rollout, not only the current wave, so a slow-burn regression surfacing in the canary still stops wave 2 from scheduling.

A server passes the gate when it has converged on the target image, reports ready and serving, has no bootstrap in flight, is not at critical memory pressure, its sample is fresh, and its counters have not regressed relative to a baseline captured just before its wave scheduled: outbox depth within 10 percent of the pre-upgrade level (with a small absolute floor), and no growth in file-descriptor wait timeouts or peer-connection failures. Comparing against a pre-upgrade baseline makes the question “did the new version regress this server” rather than an absolute threshold that would misfire on naturally busy meshes. Counter resets from pod restarts are clamped, matching the standalone gate’s behavior.

Convergence and soak eligibility are separate questions. Every scheduled deployment, including one for a server that was unhealthy when its wave scheduled, must observe the target image before the wave can complete: the fleet-reaches-degraded-servers property means the fix must land there, not that the wave may finish without it. What a baseline-unhealthy server is excluded from is the comparative health soak, where its pre-existing sickness would read as a regression it did not cause. A deployment that reaches terminal failure pauses the rollout immediately, whatever the server’s baseline was. A server being destroyed mid-rollout drops out of both convergence and gate scope.

Soak, then pause rather than proceed

Mirroring the standalone gate, transient blips reset the clock rather than pausing the world. A wave completes when every deployment has converged and all soak-eligible servers have passed the gate continuously for the soak period: proposed 15 minutes for the canary and 5 minutes for later waves. Any failing tick resets the continuous-health clock. The rollout pauses when a wave exceeds its deadline (proposed 60 minutes) without completing, or immediately on a hard signal: a deployment reaching terminal failure, critical memory pressure, or a server regressing to failed after converging on the new image.

A paused rollout schedules nothing further and records the wave, server, and failing signal that stopped it; how operators find out is covered under Observability and alerting. A healthy four-wave rollout completes in roughly 45 to 75 minutes, comfortably inside the one-version skew budget that Kura’s compatibility rules already require every release to satisfy.

Supersede, expedite, resume, abort

A new tag arriving while a rollout is non-terminal supersedes it: the old rollout stops, its unapplied deployments are cancelled, and the new rollout starts from the canary. To collapse skew quickly, a superseding rollout orders accounts still on the oldest image first after the canary.

Deployment scheduling becomes rollout-scoped, which deliberately changes an invariant the current scheduler documents as intentional. Today a server-tag pair is attempted at most once ever and failed deployments are never retried; under this RFC a server is scheduled at most once per rollout attempt instead. Resume therefore means re-attempt, not re-evaluate: it cancels the current wave’s failed deployments, mints fresh ones for every non-converged server including those whose previous attempt failed, re-captures baselines, and re-enters the gate. Without this, a wave that failed for an infrastructure reason, a bad node, a storage class, a manifest problem, would stay a dead end until an unrelated Kura release arrived to re-trigger scheduling, because nothing else ever would.

Expediting flips a rollout to all-at-once fan-out, which is today’s behavior, and applies at creation or mid-flight, so progressive pacing never stands between an incident and its fix. The two urgent cases differ in how much safety expediting gives up:

• Rolling back: a tag that previously completed a rollout in this environment already proved itself fleet-wide, so expediting it at creation carries no new risk. Re-pin, expedite, done in minutes.
• Hotfixing forward: the fix is a new, production-untested image. It ships through the existing out-of-band path (ad-hoc image, explicit tag input on the deploy workflow) and supersedes the in-flight rollout. The default is the middle path: the canary wave runs and soaks, then the operator expedites the remainder when the outage math beats the residual risk. Skipping the canary as well is possible but is an explicit operator override, never the default for an image production has not run. When the fleet is already degraded by the bad version, the waves cost mostly soak time anyway: servers unhealthy at baseline are excluded from the soak, so the gate never stands between a broken fleet and its fix.

Pause, resume, expedite, and abort are operator verbs, and the ops surface for them ships in the first version: the internal Kura ops view gains the rollout with its wave progress and the controls next to it, so the person paged during an incident acts from the same screen that explains the situation instead of reaching for a console. Manual pause exists alongside the automatic gate pause, for the case where a human sees something the gate does not.

Two guardrails keep these verbs deliberate. A server-only deploy that carries an unchanged Kura tag leaves an active or paused rollout untouched: supersede triggers on a tag change, never on a redeploy of the same intent, so a paused rollout cannot be accidentally reset by the next unrelated merge. And expediting is always an explicit act, a deployment input or an ops-view action, never inferred from commit messages or the pipeline’s hotfix fast-path. Every operator verb records its actor, reason, source and target tags, and whether the target tag had previously completed in the environment, so an expedited rollout is auditable after the incident.

Observability and alerting

Grafana is the paging authority, not the control plane. An in-band notification from the reconciler carries the richest context, but it cannot be trusted to report the failure modes that matter most: if the rollout stalled because the server is crashlooping or the reconciler stopped ticking, the component that would send the alert is the component that is broken. The control plane therefore exports rollout metrics following the existing per-domain PromEx pattern, and Grafana alert rules page through the existing IRM and on-call escalation:

• Rollout paused, sustained for a few minutes: page, carrying the tag and wave.
• Rollout active for longer than a few hours: warn. This also covers the forgotten pause, since a paused rollout stays active.
• Rollout metrics absent while a rollout was recently active: page. This is the case in-band alerting structurally misses: a control plane that dies mid-rollout does not raise a paused signal, it stops reporting entirely.

The same metrics make the rollout observable, not just its failures. The fleet’s version distribution (servers per observed image tag) turns convergence from the old tag to the new one into a visible curve; alongside it sit the active rollout’s tag, wave, paused state, wave age, and converged-versus-gated server counts, aggregated so multi-replica servers do not double-count. A rollout dashboard, provisioned as code next to the existing runners dashboard, joins these control-plane series with the per-node Kura metrics behind the gate (outbox depth, file-descriptor pressure, peer-connection failures, memory pressure), so a pause is explainable from the dashboard alone: which wave stopped, which server held it, and which signal regressed, without shelling into a cluster.

Slack stays for context, not paging: the reconciler posts on rollout start, pause (with server, region, and failing signals), completion, supersede, and expedite. There is no internal ops-Slack path in the server today, so this is new, best-effort plumbing; losing it costs color, never detection.

Scope

In scope: the managed fleet’s runtime-image rollout, both the per-account meshes and the runner-cache servers. Progressive pacing applies to the production environment; canary and staging mint rollout records in expedited mode and keep today’s immediate fan-out.

Out of scope:

• Automatic rollback. Rolling back stays an operator decision: re-pin the previous tag and expedite.
• Version management for self-hosted peers. Customers upgrade their own binaries; the one-version skew tolerance covers them.
• The within-instance update strategy. Most instances run one or two replicas; the controller’s rolling update and disruption budget are sufficient at that scale. The missing pacing dimension is across accounts, not within an instance.
• Client-side and cas-plugin version gating.
• The standalone chart’s rollout driver, which self-hosters keep.

Trade-offs

Advantages

• A bad version stops at our own accounts or a few low-traffic meshes instead of the whole fleet.
• The gate catches degraded-but-up regressions that /up structurally cannot: replication backlog, file-descriptor pressure, peer-plane failures, memory pressure.
• Reuses signals and thresholds the standalone gate has already run with in production, rather than inventing new health heuristics.
• Production promotion becomes conditional on the canary environment having actually converged on the new Kura, a check that exists nowhere today.
• Rollout intent, history, and pause state become durable and observable instead of implicit in a one-tick fan-out, and the fleet’s version convergence becomes a first-class Grafana surface.
• Incident response is a single surface: the page links to a dashboard that explains the pause, and the ops view that shows the rollout also carries the controls to act on it.

Disadvantages

• Fix delivery slows from minutes to about an hour on the happy path; the expedited mode exists precisely for urgent rollbacks and hotfixes.
• A paused rollout left unattended extends version skew; bounded by the long-running-rollout alert and supersede semantics.
• Gate false positives would erode trust in the mechanism; the soak-reset-then-deadline design tolerates transient blips by construction and thresholds start from the standalone gate’s proven values.
• The canary wave sees only Tuist-owned traffic, which is lighter than the busiest customer meshes; regressions that only appear under heavy load surface in later waves, which is what the fleet-wide gate scope and pause exist to catch.
• More moving parts in the reconciler, one more status surface on the instance resource, and an ops UI in the first version rather than after it.

Alternatives considered

Argo Rollouts or another progressive-delivery controller

The unit of rollout here is a fleet of per-account custom resources driven by our own controller, not a single Deployment. Off-the-shelf progressive delivery does not sequence across resources, and Postgres already owns rollout intent; a second orchestration brain would split that ownership.

GitOps (Flux) as the rollout mechanism

GitOps improves the layer above this RFC: the runtime tag becomes a committed value per environment, bumps become PRs, rollback becomes a revert, and cross-environment promotion becomes explicit. None of that paces the layer below: the fleet’s desired state is minted per account from Postgres, not from git, and a Flux-applied tag bump fans out exactly as abruptly as today’s deploy-time value. Flagger canaries a single Deployment behind traffic shifting and has no notion of a fleet of per-account StatefulSets. If Flux is adopted, it composes with this RFC rather than replacing it: git owns which tag an environment should run, the reconciler owns how the fleet gets there, and cross-environment promotion can gate on the rollout record completing.

Blocking the production deploy job until the production rollout completes

Rejected. It would hold a CI job open for the better part of an hour on the happy path and indefinitely on a pause, tie the rollout’s lifetime to job timeouts, and duplicate the paging authority that Grafana owns. Combined with supersede semantics it would also turn every subsequent merge red while a rollout sits paused, making unrelated server deploys casualties of a Kura investigation. The pipeline gates on the canary environment’s convergence, which takes minutes and catches images that cannot run at all; the production rollout record and its alerts are the authority thereafter.

Driving the standalone rollout scripts from CI against the managed fleet

Shell-driven, no durable state across server deploys, and CI has no authenticated path to every account mesh’s pods. The reconciler already has the state store, the cadence, and the provisioner access.

Polling /status/rollout from the server through the public endpoints

Exposes an internal health surface publicly and adds hundreds of public HTTP polls per tick. The controller already reaches pod IPs privately and already parses these reports.

Partitioned updates within each StatefulSet

Instances run one or two replicas, so within-instance pacing adds nearly nothing. The missing dimension is across instances.

Time-based waves without a health gate

Strictly worse: it slows the blast radius without ever noticing the blast.

Rollout

1. Health publication. The rollout report gains the peer-connection failure counter, and the controller publishes the per-field aggregate on the instance status. Additive, no consumer. Validated on staging by inspecting resources during a normal version bump.
2. Interim pacing. Cap per-tick scheduling and order Tuist-owned accounts first, opening the next batch only when the previous one has converged and stayed ready. No new state, immediate win over all-at-once while the full machinery lands.
3. Wave machinery and ops surface. The rollout state machine, rollout-scoped scheduling, wave assignment, gate, pause and supersede semantics, and the operator verbs with their ops-view controls, behind a feature flag that falls back to interim pacing, together with the rollout metrics, the Grafana dashboard, and the alert rules. Progressive mode is exercised on staging with the flag enabled through at least two real Kura releases; staging returns to expedited mode afterwards, and canary stays expedited throughout. Drills: one induced degraded-but-up failure (for example firewalling a peer port on one updated node) to prove the pause fires, the page reaches on-call, and the dashboard explains which server and signal stopped the wave; one infrastructure-caused wave failure that is fixed and then resumed, to prove resume re-attempts the failed deployments rather than skipping them; and supersede, expedite, and hotfix drills (ad-hoc image through the explicit tag input, expedited after canary soak) driven through the ops view rather than a console.
4. Pipeline gate and production enablement. Add the promotion gate on the canary environment’s rollout completion to the deploy pipeline, enable progressive mode in production, route the alert rules to the production on-call escalation, and remove the interim path and the flag.

Rolling the feature itself back is the flag flip back to fan-out scheduling; the health publication is read-only status and needs no rollback.

Open questions

• Wave sizing and soak durations are proposed as 5/25/70 percent and 15/5 minutes; revisit after the first monitored production rollout.

References

• Kura context module
• Kura reconciler
• Kubernetes controller provisioner
• KuraInstance controller
• Standalone chart rollout gate
• Standalone chart rollout driver
• Rollout status report
• Kura rollout safety rules
• Server deployment workflow
• Ad-hoc Kura runtime image workflow
• Managed Helm values
• Runners PromEx plugin (metrics pattern)
• Runners Grafana dashboard (provisioning precedent)
• GitOps and health monitoring for the workload clusters

E

eduardo.ext@tuist.dev Jul 20, 2026

Great spec — the wave design grounded in signals the standalone gate has already proven in production, the regression-vs-baseline framing instead of absolute thresholds, and the honest treatment of the expedite trade-offs make this feel very buildable. The “Grafana is the paging authority because the reconciler can’t report its own death” reasoning is exactly right.

One suggestion on the operator verbs:

Make retrying the same version a first-class operation — a failed rollout can’t always wait for a new tag.

A rollout can fail for reasons that will never produce a new Kura version: an infrastructure change (node pool, storage class, DNS, secrets, manifest revision), a transient cluster condition, or a bad box. Once the cause is fixed, the image that failed is still the correct image — but nothing new arrives to re-trigger it.

Today’s scheduler makes that a dead end by design: each (server, image_tag) pair is scheduled at most once, and a failed deployment for the configured tag is “intentionally not retried” (Tuist.Kura.schedule_version_deployments/1). The manual escape hatch, retry_server/2, only accepts servers that never activated, so a previously-active server whose drift deployment failed has no recovery path until the next Kura release, however unrelated the original failure was.

The RFC’s verbs get close but don’t quite cover this: “Resume re-captures baselines and re-enters the current wave’s gate” implies re-attempting the wave — but if resume is built on the existing scheduling predicates, servers whose deployment for the target tag already failed are silently skipped. The wave re-enters its gate with the wedged servers still wedged, fails again, and pauses again.

Suggestion: make deployment scheduling rollout-scoped rather than server-tag-lifetime-scoped — a server is scheduled at most once per rollout (or per resume), so resuming after an infra fix forces a fresh deploy attempt of the same tag onto every non-converged server in the wave, including those with a prior failed deployment row. In short: resume should mean “re-attempt”, not “re-evaluate”. Probably worth a sentence in the operator-verbs section, since it changes a scheduling invariant the current code documents as intentional.

P

pedro@tuist.dev Jul 20, 2026

I’m aligned with this direction. Durable per-environment rollout state, account-grouped waves, regression-based health gates, and explicit operator controls are the right shape. I would support moving forward once these details are reflected in the design:

1. Gate production on canary rollout completion. The current deployment flow can finish the canary server deployment and acceptance tests before the asynchronous Kura rollout has completed all waves and soaks. Production should require the canary rollout record for the target tag to be completed; paused, aborted, or superseded must block promotion.

2. Do not let servers created mid-rollout bypass the rollout. Provisioning directly on the configured target can introduce a version after its rollout has paused and can split an existing account mesh before that account’s wave. A new server should inherit the account’s safe version and join its assigned wave, or remain unavailable until that account is eligible. This is especially important for runner-cache servers, which can receive traffic immediately.

3. Correct the health signal. The existing fd_timeout_count field counts file descriptor wait timeouts, not peer connection failures. The design should either gate on file descriptor pressure and name it accordingly, or add a real peer connection failure signal.

4. Separate convergence from health-soak eligibility. Every scheduled deployment, including one for a server that was unhealthy at baseline, should have to observe the target image before its wave can complete, and a terminal deployment failure should always pause the rollout. Baseline-unhealthy servers can be excluded from the comparative health soak without being excluded from convergence.

5. Define aggregation per field. A generic worst-of aggregate cannot reproduce the existing gate. Readiness needs conjunction, pressure needs a maximum, counters need totals with reset handling, and generation needs an explicit consistency result. These semantics should be part of the status contract.

6. Make hotfix behavior explicit.

   • A server-only hotfix with an unchanged Kura tag must leave an active or paused Kura rollout untouched.
   • A rollback to a tag that previously completed successfully in the target environment may be explicitly expedited at creation.
   • A new forward Kura hotfix must pass the canary soak before reaching production. An operator may then explicitly expedite production when incident impact outweighs the remaining risk.
   • Expediting should be an explicit deployment input or operator action, never inferred from (hotfix) in a commit message.
   • Expedited actions should record the actor, reason, source tag, target tag, and whether the target previously completed successfully.
   • The deployment workflow should wait for the applicable rollout to complete and fail if it pauses, aborts, or is superseded.

I also agree with Eduardo’s retry point: resume should re-attempt non-converged deployments of the same tag using rollout-scoped attempts, not merely re-evaluate the existing failed rows.

M

marek@tuist.dev Jul 21, 2026

Thank you both — revision 7 folds the feedback in. How each point landed:

Eduardo, rollout-scoped retries: adopted in full. I verified the invariants you cited: the scheduler documents failed deployments for the configured tag as intentionally not retried, and retry_server/2 only accepts servers that never activated, so the wedged-until-next-release dead end is real. The spec now makes scheduling at-most-once per rollout attempt instead of per server-tag lifetime, defines resume as re-attempt (cancel failed rows, mint fresh deployments for every non-converged server, re-capture baselines, re-enter the gate), calls out that this deliberately changes a documented invariant, and adds a validation drill for exactly your scenario: fail a wave for an infra reason, fix it, resume, prove the failed deployments are re-attempted rather than skipped.

Pedro 2, mid-rollout servers: adopted. New servers inherit the account’s wave state — baseline tag until the account’s wave completes, target tag after — so a paused-as-suspect version never reaches fresh servers and an account mesh isn’t split ahead of its wave. The runner-cache emphasis is called out since those take traffic immediately.

Pedro 3, health signal: adopted, you’re right. Verified in metrics.rs: fd_timeout_count increments on file-descriptor wait timeouts, not peer failures. The spec now gates on it under its real name (fd pressure is still a genuine regression signal) and adds a peer-connection failure counter to the rollout report so peer-plane health gets a real signal; the gate consumes both.

Pedro 4, convergence vs. soak eligibility: adopted. Every scheduled deployment, including baseline-unhealthy servers, must observe the target image before its wave completes; the exclusion applies only to the comparative health soak. Terminal deployment failure now pauses immediately rather than waiting out the wave deadline.

Pedro 5, aggregation semantics: adopted. Per-field semantics are now part of the status contract: readiness/serving as conjunctions, generation as an explicit all-consistent flag, counters as sums with per-pod reset clamping, pressure as maximum, sample timestamp as oldest.

Pedro 6, hotfix behavior: adopted except the last bullet. Same-tag server deploys leave an active or paused rollout untouched (supersede triggers on tag change only); rollback to a previously-completed tag may expedite at creation; a forward hotfix defaults to canary-soak-then-expedite with the full skip as an explicit operator override; expedite is never inferred from commit messages or the hotfix fast-path; and every operator verb records actor, reason, source/target tags, and whether the target previously completed.

Pedro 1 and the last bullet of 6, pipeline coupling: partially adopted, with Marek’s direction shaping the split. The canary environment keeps today’s immediate fan-out — its purpose in the cascade is to produce the exposure signal as fast as possible, so pacing it would be self-defeating. But the substance of your point is kept: canary now mints an expedited rollout record, and the pipeline’s production stage gains a promotion gate that waits for that record to complete, with paused/aborted/superseded blocking promotion. That check exists nowhere today, so production’s wave 0 can currently start on an image canary never actually converged on.

What we deliberately did not adopt is the deploy workflow waiting on the production rollout. That would hold a CI job open ~an hour on the happy path and indefinitely on a pause, tie the rollout’s lifetime to job timeouts, duplicate the paging authority Grafana owns, and — combined with supersede — turn every subsequent merge red while a rollout sits paused, making unrelated server deploys casualties of a Kura investigation. The production rollout is deliberately off the deploy path: a green deploy means intent recorded, and the rollout record plus its alert rules are the authority for convergence. This is now an explicit entry under Alternatives considered.