feat(infra): HA stable-egress gateway with failover controller (+ revert Keygen boot hotfix)

Added Option A — reserved stable egress set on top of the HA work (commit c426853):

• Customers now allowlist a fixed reserved set of egress IPs, not a single address — so future capacity growth or an IP migration within the set never forces a customer allowlist change (slow, high-friction enterprise ops). On Hetzner Cloud there’s no owned contiguous CIDR/BYOIP, so the “set” is a reserved pool of individual Floating IPs.
• Controller gains --egress-ip-allowlist (CIDRs) and fails closed: it refuses to activate a Floating IP whose address is outside the documented set, so we can never egress from an un-allowlisted IP (a gap, never a leak). Unit-tested.
• network.md reframed to a reserved set with a maintainer note keeping it in lockstep with egressIpAllowlist.

Operator step (gates the full benefit): reserve the extra Floating IPs in tuist-workloads, then add their /32s to both values-tuist.yaml‘s egressIpAllowlist and network.md before they’re used. Today only 116.202.0.10 is provisioned, so the set is currently a set of one — the machinery is in place to grow it without touching customer allowlists.

Deploy safety + review findings

Is merging dangerous to server availability? Inbound (web/API) is low-risk — nothing here touches ingress or rolls the server/DB by design. The real exposure is a brief outbound egress blip during the one-time gateway cutover, plus a high-blast-radius item to verify. Mitigated by staging + a safe default (see below).

Review findings

• [P1] Stale active labels outside md-egress — fixed in commit dc7a99cdff: reconcileActiveLabel now lists active-labelled nodes cluster-wide and strips the label from any non-elected node, so the hand-labelled general worker is cleaned up on cutover. I kept the dataplane selectors active-label-only (not active + candidate) on purpose: with the cluster-wide strip, a two-label selector would only add a cutover gap where neither the old node (no candidate label) nor the not-yet-elected md-egress node matches. (Rationale in the README.)
• [P1] go.sum missing → image can’t build — confirmed merge blocker; I can’t run go mod tidy in this sandbox (no module network). Now neutralized for merge safety: the production overlay defaults failoverController.enabled: false, so the chart never references the missing image. Committing go.sum is a prerequisite to enabling.
• [P2] mutable latest + no release path — production overlay is now enabled: false with image.tag documented as “pin to released semver, not latest.” Full release.yml wiring (semver tag + chart-pin bump, mirroring hetzner-robot-controller) is flagged as a required follow-up in the controller’s AGENTS.md — I deliberately did not hand-edit the production cascade workflow blind.

Staged rollout (now in the platform README)

1. Commit go.sum → build + publish image → pin image.tag.
2. Apply ClusterClass + md-egress; verify kubectl diff shows no roll of existing pools (md-0 runs server + CNPG); wait for md-egress nodes Ready + candidate label.
3. Flip failoverController.enabled: true; controller does the one-time IP/label cutover (~seconds of egress blip, inbound unaffected); verify curl api.ipify.org = egress IP.
4. Deploy the server image (Keygen revert) last.

Net: as committed, merging is safe (HA machinery lands dormant); enabling HA is a deliberate, staged follow-up.

Release wiring added (P2 resolved)

Wired stable-egress-controller into the standard component release flow, mirroring hetzner-robot-controller so the image is semver-tagged and the prod pin is no longer latest:

• components.json — registered the component; release:check detects it (verified locally: next: stable-egress-controller@0.1.0, release? true).
• release.yml — check-releases outputs, a release-stable-egress-controller job (changelog + multi-arch build/push), the publish aggregation needs, artifact download, and the GitHub Release step.
• cliff.toml — scoped to (stable-egress-controller) commits.
• renovate.json — tracks the ghcr semver and bumps the platform-chart pin (failoverController.image.tag in values-tuist.yaml); the platform chart deploy rolls it out on merge (same model as the hetzner pin, just via the platform deploy instead of mgmt-cluster-apply).
• values-tuist.yaml — pinned tag: 0.1.0 with a Renovate marker; stays enabled: false until the first release exists.

Validated: components.json + renovate.json parse; release.yml parses; check.sh stable-egress-controller resolves the initial version and should-release.

The single remaining prerequisite is still go.sum (go mod tidy, needs network) — the release job’s Docker build does go mod download, so it can’t build until go.sum is committed. That’s the one thing I can’t do from here.

e2e validated on staging (without merging)

Got a :sha image from this branch (temporarily added the branch to the image workflow’s push: trigger — workflow_dispatch can’t reach a workflow that isn’t on the default branch yet — then reverted), deployed the controller to staging by hand (it mounts staging’s in-cluster hcloud secret, so no token handling), candidate-labeled the two staging general nodes, and ran the full drill. Staging was restored to its original state afterward.

What passed:

• Sticky election / no churn — with the current FIP holder (4jv57) already a candidate, the controller kept it; zero cutover, allowlist guard logged active egress IP 78.47.186.71.
• Failover — dropping 4jv57 from the candidate pool, the controller reassigned the Hetzner Floating IP via the API (fromServer→toServer logged), stripped the stale active label cluster-wide, set it on h6prq; the host-configurer DaemonSet followed; failback to 4jv57 likewise.
• Host datapath on the new node verified — eth0 had 78.47.186.71/32, the from 78.47.186.71 lookup 2010 rule, and the table-2010 default route.
• Egress IP preserved — 78.47.186.71 from a staging server pod across the drill (active/standby holds the single allowlisted IP).

Key finding worth acting on: Cilium OSS egress-gateway datapath convergence onto a node it has never used as a gateway took >40s the first time (egress curl timed out during that window), but ~12s on subsequent moves once Cilium knew the node. The controller/host config were correct throughout — this is the Cilium egress-gateway layer (consistent with cilium/cilium#30157).

Implication: a real prod node-replacement failover targets a brand-new node → the slow first-time path, so failover egress downtime can exceed the ~30–60s estimate, and the worst case (does it always converge, or occasionally need a cilium-agent nudge / policy re-apply?) needs validating before we lean on prod auto-failover. I’d treat that as a follow-up to investigate/mitigate (e.g., pre-warm the egress node, or have the controller re-poke the policy on cutover). The controller itself — the part this PR adds — works as designed.

Timed convergence on staging (answering “is it a problem with redundancy?”)

Ran timed failovers on staging:

• Warm failover (target node was a gateway before): seamless — active label flips by ~6s and egress never dropped at 4–5s poll resolution.
• Cold failover (a node Cilium has never used as a gateway): the >40s seen once, on the node’s first-ever activation.
• Always converges, never stuck across ~5 failovers — the safety property that matters.

Conclusion: redundancy removes the catastrophic mode. With the 2-node pool there’s always a healthy target and the FIP move is fast, so the worst case is a bounded, auto-healed egress blip — never the indefinite, human-gated outage from the incident. The one nuance: in steady state only the active node is a programmed Cilium gateway, so the standby is “cold” — the first real failover (a node replacement) hits the >40s cold path, then it’s seamless thereafter.

So: not a blocker. Optional follow-up to make even the first failover seamless — keep the standby warm (periodically rotate the active between the two nodes, or list both in Cilium’s egressGateways). I’d ship the HA as-is and treat warming as a later optimization.

(Staging was used as the test bed via a temporary branch push-trigger on the image workflow + a hand-deployed controller; both reverted/torn down, staging restored to its original single-gateway state.)

Reserved egress set provisioned — no longer single-IP-dependent

Created 3 more Hetzner Floating IPs in the tuist-workloads project, so prod egress is now a reserved set of 4, not one address:

(Hetzner hands out scattered /32s, not a contiguous block — so it’s a documented set, as discussed.) Wired all four into the controller’s egressIpAllowlist (values-tuist.yaml) and the customer guide (network.md). Customers allowlist the set once; the active egress migrates within it with no customer allowlist change ever again. The controller still operates 116.202.0.10 as the active member; the other three are pre-allowlisted spares (active/standby, one live at a time).

The HA stable-egress gateway with failover controller changes from this PR are now available in version xcresult-processor-image@0.23.0. Update to this version to use the new controller and infrastructure improvements.