Hive Hive
Sign in

feat: multi-box bare-metal Kura — per-account DNS routing + warm-handoff moves

GitHub issue · Closed

Metadata
Source
tuist/tuist #11611
Updated
Jul 5, 2026
Domains
Kura
Details

Makes bare-metal Kura regions scale to N boxes per region, with tenants distributed across boxes, each tenant reaching its own box, and accounts movable between boxes with no cold-cache dip.

Phase 1 — per-account routing (controller + server)

Today a bare-metal region is a single box: the host-network gateway DaemonSet reports the node’s InternalIP into each Ingress’s status, and external-dns publishes <account host> → that box. With N boxes the DaemonSet reports all node IPs, so an account’s traffic round-robins across boxes — including boxes that don’t host its pod. Per-account routing fixes that:

  • kura-controller (reconcilePublicDNSEndpoint): a per-account DNSEndpoint A-record → the box the account’s pods run on (Status.NodeAddress = the box’s public IP on bare metal), TTL 60. Mirrors the shipped peer-plane pattern. New publicHostNetwork spec field gates it; instancePeerNodeIP generalized to instanceNodeIP.
  • server: threads publicHostNetwork onto the manifest for host-network regions; bumps the manifest revision.

Why DNS, not a failover IP

The box already answers on its main public IP for free. A failover IP would need a new privileged per-box component to configure the IP on the interface as accounts move (plus Dedibox virtual-MAC handling), and the shipped per-region FailoverIP scaffolding was never exercised. DNS reuses code already in production for the peer plane, needs zero box-side config, and for our persistent-gRPC clients the cutover is gentler (natural drain vs. a hard connection break). Fail-open (miss → origin) covers stragglers either way.

Phase 2 — warm-handoff moves (rebalancing)

Rebalancing an account off a hot box, with no cold-cache dip:

  • Schema: move_phase (none/moving_in/moving_out) + target_node; a partial unique index keeps exactly one steady-state (:none) server per (account, region) while a transient move pair coexists.
  • move_server/2: provisions a second :moving_in instance pinned to the destination box; it warms from the source over the account peer plane (same-account mesh discovery) while the source keeps serving.
  • Reconciler: promotes the target once it has caught up — readiness is the peer-plane bootstrap gate (status.phase == "Ready" via caught_up?), not the public /up probe (a moving_in has no public endpoint). Promotion atomically swaps host ownership (source :none → :moving_out, target :moving_in → :none) and re-renders both manifests, so the customer host has a single owner throughout and there are never two same-host Ingresses. The source drains for a window, then is destroyed.
  • Only the :none row carries the customer host (owns_customer_host?), so the cutover needs no controller change — an empty publicHost already leaves the Ingress/DNS/Certificate unreconciled. Cache-endpoint teardown is move-safe (the draining source shares the customer URL with the promoted target).

Placement & scaling

Cross-account spread uses the shipped egress + cpu/memory bin-packing (fill a box, spill to the next). Scaling a region = replicas: on the fleet MachineDeployment.

Follow-ups (not in this PR)

  • Operator/dashboard move_server action.
  • Optional *.kura.tuist.dev wildcard cert on bare-metal gateways to make the move cutover fully gap-free (today a per-instance ACME cert issues at promotion; fail-open covers the brief gap).
  • Staging 2-box e2e (routing + warm move + box-loss).

Validation

  • Go: go vet + go test ./... in infra/kura-controller (per-account DNSEndpoint publish/skip/teardown).
  • Elixir: mix compile clean; unit tests added for move manifest rendering, move_server (happy/non-movable/concurrent/non-active), and reconciler promote/warming/drain. Full mix test runs in CI (local test-env recompile was pathologically slow this session).

🤖 Generated with Claude Code

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments

No GitHub comments yet.