feat(infra): move Kura cache regions to the in-house bare-metal CAPI provider (no-LB direct routing + gapless deploys)

This branch builds the in-house multi-vendor bare-metal CAPI provider and uses it to move customer-facing Kura cache regions off Hetzner ccx13 cloud VMs onto bare metal: eu-central onto Scaleway Dedibox (all environments) and a new ca-east onto OVH (staging and canary while validated). It also reworks the cross-region peer plane to route directly pod-to-pod with no load balancer, serves the off-cluster self-hosted peer plane host-network (SNI passthrough + a per-region failover IP, still no LB), and makes bare-metal regions survive a rolling deploy without dropping traffic. Targets main (stacked on #11256, since merged).

It is large because it spans the provider foundation and the first real cutovers that exercise it end to end. The halves are separable on request; they are together because the cutovers are what validated the provider against live boxes.

The in-house bare-metal CAPI provider (foundation)

The provider (cluster-api-provider-tuist) now hosts macOS (Apple Silicon), Scaleway Elastic Metal, OVH, and Dedibox machine kinds.

• OVHDedicatedMachine and DediboxMachine, both adopt-by-self-join. The operator pre-orders AND prepares each box out of band — install Ubuntu, authorize the fleet SSH key, grant sudo, verify reachability, then stamp the fleet marker (mise run baremetal:prep-{ovh,dedibox} + the mark-{ovh,dedibox} task). There is deliberately no inline order: OVH ordering is a multi-step cart/checkout/payment flow, so it stays off the deploy path entirely and out-of-stock risk never reaches a deploy. The controller then adopts a free prepped box — claim + self-join only, no OS install on the adopt path — which is what makes adoption a ~2-5 min self-join, so the fleet MachineDeployment goes Ready quickly and never wedges helm --wait. The only install the controller drives is on the release path: a released box is reinstalled back to a clean, claimable state (re-establishing the fleet key) so a re-claimed box rejoins from a clean OS. Each scopes adoption to its fleet with a required per-box marker, the environment boundary since one vendor account holds every env’s boxes: a Scaleway tag for Dedibox, the OVH-side displayName for OVH. Foreign ovh:// / dedibox:// providerIDs keep the Hetzner CCM from reaping the node; release drops the Node + identity but does not terminate the box (monthly contracts).
• Renamed the provider module/dir to cluster-api-provider-tuist and the shared infra-cluster stub ScalewayAppleSiliconCluster to TuistCluster; repackaged the controllers into {macos,linux,shared}. Operational identifiers (image name, release tag prefix, helm componentName) were kept stable so deploy/image resolution stays byte-stable.
• ScalewayElasticMetalMachine moved onto the same adopt/claim model (release-to-pool, not terminate).
• All vendor kinds reuse the vendor-agnostic Linux self-join, system:node credentials, and kubeconfig builder. The self-join establishes its own NOPASSWD sudo (from the prep-set sudo password, used once), relocates containerd + the kubelet root onto the box’s large /data disk (Dedibox/OVH ship a small root + a big separate data partition), and is idempotent across re-adoption (stale CNI sysctl drop-ins, apt keyring writes).

Moving Kura regions to bare metal

regions.ex now backs two regions with bare metal, both with local-NVMe storage and a hostNetwork regional gateway bound to the box’s public IP (neither vendor has a Hetzner LB, so serving traffic hits the box NIC directly). Region id, cluster_id, ingress class, and public hostnames are unchanged, so the change is invisible to customers and the CLI.

• eu-central → Scaleway Dedibox (kura-dedibox pool, each env’s dediboxFleet). Exposed in every environment, so the cutover is live on merge for canary and production.
• ca-east → OVH (Beauharnois / BHS, kura-ca-east pool, the ovhFleet). A new geographic region that replaces the throwaway ovh-staging region; gated by TUIST_KURA_AVAILABLE_REGIONS to staging and canary while the OVH integration is validated, so merging does not cut production over to OVH (production serves us-east and us-west on OVH instead).

Why bare metal. ccx13‘s “dedicated” is vCPU, not NIC (best-effort shared NIC + network-attached storage), and Hetzner’s June 2026 increase took it from EUR 15.99 to EUR 42.99/mo. A cache is bandwidth- and IO-bound, so bare metal with a real NIC + local NVMe is both cheaper per unit of capacity and far faster.

Peer plane: direct in-cluster routing, no load balancer

The cross-region mesh bootstrap previously dialed each peer’s advertised public gateway URL, a type: LoadBalancer Service. That is dead on bare metal (the nodes are excluded from the Hetzner LB and aren’t hcloud), so the bare-metal regions were stuck 0/1, unable to bootstrap.

The fix routes managed↔managed peers (all in-cluster, any region) directly pod-to-pod instead of through a gateway URL. Kura’s discovery scope (Local vs Global) is addressing-only, not a replication-policy switch: a Local-scope peer returns its direct node URL and is dialed pod-to-pod; a Global-scope peer returns its gateway URL. So this is a control-plane env change in the kura-controller — set KURA_DISCOVERY_DNS_NAME (Local scope) to the account peer Service and stop wiring the global discovery name — with no Rust change. The leaf cert SAN already covers the account peer Service name, so direct-dial SNI validates.

This is why there is no LB anywhere on the bare-metal path: customer cache serving is hostNetwork on the box NIC, and cross-region replication is direct pod-to-pod over the node NIC. It removes the bandwidth cap and per-LB cost an LB-fronted peer plane would have imposed, and it sidesteps cross-provider LB integration entirely. The off-cluster self-hosted-node ingress applies the same no-LB principle — see Self-hosted peer ingress below.

Self-hosted peer ingress (host-network, no LB)

Every managed public region is now bare metal, so a customer’s off-cluster self-hosted Kura node can’t dial the managed mesh the old way: the public peer endpoint (:7443) was a type: LoadBalancer that never gets an address off a cloud LB. This serves it without one, end to end (the peer plane is already mTLS, so nothing terminates TLS):

• On a host-network region the per-instance peer Service is ClusterIP, and a region-keyed controller (PeerDemuxReconciler) runs a host-network L4 SNI-passthrough demux (stock nginx ssl_preread) on each box’s :7443, routing by SNI to the addressed account’s peer Service (cross-box pod routing covers a pod on another box).
• A per-account DNSEndpoint publishes the region’s peer.<host> to a per-region failover IP (external-dns gains the crd source). The CAPI provider keeps that IP routed to a healthy box (OVH /ip/{ip}/move, Dedibox failover-ips/attach) and drains it off a box whose demux is rolling — so the peer plane survives a box loss and a deploy.
• Failover IPs are operator-provisioned per region (a FailoverIP CR + TUIST_KURA_PEER_FAILOVER_IPS, parked in values until ordered), like the fleets.

Gapless rolling deploy

The cache is single-writer and primary-pinned with no read-through, so reads can’t be load-balanced across replicas. Bare-metal regions now run replicas: 2 (a warm standby co-located on the box until a second box lands) and the kura-controller fails the cache Service over to a standby that is Ready and runtime-confirmed serving, instead of stalling on the old minPrimaryPodAge gate (which would leave a freshly-rolled standby ineligible exactly when the primary rolls). The standby is a near-real-time copy via the async outbox, so a promoted replica already holds everything but the in-flight window: the deploy cuts over without a write stall, and the only cost is a brief window of misses on the very-newest artifacts, which regenerate.

Supporting changes

• feat(kura) parallelize the bootstrap. A joining node replicated a peer serially (one fetch + apply per artifact), so it was round-trip-bound rather than bandwidth-bound (~7 MB/s against a 512 MB/s cap). Now bounded-concurrent (buffer_unordered, default 16). ~8-10x faster in validation.
• feat(server) a replicating server status, so a cache whose pod is up but still mesh-bootstrapping surfaces a “Replicating” badge instead of “Deploying” for the whole bootstrap.
• fix(infra) the hostNetwork gateway sets use-proxy-protocol: false. The shared regional config turns it on for the Hetzner LB’s PROXY header; with no LB on the hostNetwork path, nginx mangled every connection (HTTP 000 on :80/:443). Found during staging validation.

Validation (staging, end to end)

Provisioned both bare-metal regions and exercised the full cache path against live boxes:

• Provider/nodes: Dedibox and OVH boxes Ready with foreign dedibox:// / ovh:// providerIDs; cache pods scheduled on the bare metal; 50Gi local-NVMe PVCs bound.
• Serving: hostNetwork gateways bound to each box’s public IP, external-dns A records to the box, https://<host>/up + the gRPC host return HTTP 200 with valid Let’s Encrypt certs — the exact request that was :nxdomain / 000 before.
• Mesh (no LB): all cross-region peers bootstrap pod-to-pod with no gateway-URL hairpin; the account’s 5-pod mesh (staging writer + eu-central + ca-east) converges with the writer holding the lock and 4 bootstrapped peers.
• Cache push/pull/replication: a blob written to the writer is read back (read-after-write) and replicates to peers across all three boxes; verified by reading each replica directly by pod IP (no read-through, so a hit proves a local replicated copy). Confirmed on 4 of 5 pods — see open items for the 5th.
• Auth: account-token introspection against the cache works end to end (opaque tuist account tokens resolve to active: true with the right project grants; the cache authorizes the namespace). No auth bug.
• ca-east comes up 2/2 Ready and serving (gapless-capable).

Known issues / open items

• eu-central second replica was wedged on a kura bootstrap disk-amplification bug — now fixed on main (#11551). The second eu-central replica filled its 50Gi volume during bootstrap (ENOSPC) because a node bootstrapping from several peers concurrently (buffer_unordered(16) over each peer’s overlapping manifest) wrote duplicate/raced copies of the same artifacts. fix(kura): serialize replicated artifact applies per key (#11551) serializes per-key applies; this branch inherits it on rebase, so eu-central runs its second replica (gapless) like ca-east.
• kubectl exec/logs is broken for bare-metal pods. The apiserver dials the node Hostname (unresolvable for cross-cloud nodes that report only an InternalIP). The fix — set the kubeletPreferredAddressTypes ClusterClass variable to put InternalIP ahead of Hostname — is already on main (cluster-*.yaml) but the staging control-plane never rolled it to the apiserver (a stuck mgmt-cluster KCP rollout). Logs remain reachable via Loki regardless. Resolving the roll is a mgmt-cluster operation.

Merge prerequisites (canary + production)

The eu-central Dedibox cutover is live on merge for canary and production. Before merging:

• A Dedibox box per env pre-ordered AND prepared out of band: install Ubuntu, authorize the fleet key, grant sudo, verify reachability, then stamp the fleet tag tuist-kura-canary / tuist-kura-production (mise run baremetal:prep-dedibox then mark-dedibox). The controller adopts it by claim + self-join (no OS install on adopt; the /data disk relocation happens in the self-join). First entry is this manual prep; subsequent release→re-claim cycles are controller-reinstalled.
• DEDIBOX_SCW_API present in the canary and production vaults.
• helm --wait gates on the fleet MachineDeployment being Available, so a box that can’t join wedges the deploy into an --atomic rollback. Boxes must be ordered, prepped, and reliably joinable before merging; a fleet stays parked (enabled: false) until its box is.

ca-east/OVH stays staging/canary-gated (TUIST_KURA_AVAILABLE_REGIONS). Production’s OVH regions are us-east + us-west, now included in this PR (their ovhFleets, the host-network platform ingress, and the Hetzner us-east/us-west pools dropped from cluster-production.yaml); eu-central (Dedibox) is in as well. Bringing a region’s box online: order + prep it with mise run baremetal:prep-ovh / prep-dedibox (installs Ubuntu + the fleet key, then names/tags it into the pool), with the OVH_API token scoped for the whole flow (server reads + serviceInfos + GET /services/* + PUT /service/* + reinstall). The full runbook lives in infra/cluster-api-provider-tuist/AGENTS.md.

Follow-ups

• Roll the kubeletPreferredAddressTypes change onto the live staging control plane (stuck mgmt-cluster KCP rollout) to restore exec/logs on bare-metal pods.
• Collapse the dual LB / host-network paths to host-network-only once no managed Kura region runs on a Hetzner cloud VM: drop the :hetzner gateway mode, hcloud-volumes default, and hetzner_location (+ its regions_test cross-check) in regions.ex; remove the base LoadBalancer service types + Hetzner LB annotations in the platform values; drop mesh_public_peer_lb_annotations / gateway_load_balancer_annotations and the LB branches in kubernetes_controller.ex; make the host-network branch unconditional in kuragateway_controller.go; remove leftover Hetzner kura node pools and ccx13 references.
• Gate dedicated KuraGateways off on host-network regions. They’re LB-dependent (a dedicated IP) and on bare metal would host-network-conflict with the regional gateway on the box’s :443; post-cutover every public region is bare-metal, so they have no working home. Enterprise isolation on bare metal is the egress floor + a per-account host/cert on the shared regional gateway. (The host-network gateway config fix in this PR stays as defensive correctness.)
• Multi-box customer routing. This PR routes the peer plane per-region (a shared failover IP + SNI demux). The customer cache host still publishes every box’s IP via reportNodeInternalIp, which round-robins once a region has >1 box. Since an account is pinned to one box per region (co-location, this PR), route each account’s customer host to its box with the same per-account DNSEndpoint mechanism this PR introduces for the peer plane; the regional gateway’s cross-route via the cache Service is the during-reassignment fallback.
• Egress floor refinement: reserve the tuist.dev/egress-mbps floor on the elected primary only (the co-located warm standby doesn’t serve customers), or bake the 2× into the floor value, so an account doesn’t reserve twice its floor on a box.

🤖 Generated with Claude Code

No GitHub comments yet.