Hive
feat: multi-box bare-metal Kura — per-account DNS routing + warm-handoff moves
GitHub issue · Closed
Makes bare-metal Kura regions scale to N boxes per region, with tenants distributed across boxes, each tenant reaching its own box, and accounts movable between boxes with no cold-cache dip.
Phase 1 — per-account routing (controller + server)
Today a bare-metal region is a single box: the host-network gateway DaemonSet reports the node’s InternalIP into each Ingress’s status, and external-dns publishes <account host> → that box. With N boxes the DaemonSet reports all node IPs, so an account’s traffic round-robins across boxes — including boxes that don’t host its pod. Per-account routing fixes that:
- kura-controller (
reconcilePublicDNSEndpoint): a per-accountDNSEndpointA-record → the box the account’s pods run on (Status.NodeAddress= the box’s public IP on bare metal), TTL 60. Mirrors the shipped peer-plane pattern. NewpublicHostNetworkspec field gates it;instancePeerNodeIPgeneralized toinstanceNodeIP. - server: threads
publicHostNetworkonto the manifest for host-network regions; bumps the manifest revision.
Why DNS, not a failover IP
The box already answers on its main public IP for free. A failover IP would need a new privileged per-box component to configure the IP on the interface as accounts move (plus Dedibox virtual-MAC handling), and the shipped per-region FailoverIP scaffolding was never exercised. DNS reuses code already in production for the peer plane, needs zero box-side config, and for our persistent-gRPC clients the cutover is gentler (natural drain vs. a hard connection break). Fail-open (miss → origin) covers stragglers either way.
Phase 2 — warm-handoff moves (rebalancing)
Rebalancing an account off a hot box, with no cold-cache dip:
- Schema:
move_phase(none/moving_in/moving_out) +target_node; a partial unique index keeps exactly one steady-state (:none) server per(account, region)while a transient move pair coexists. move_server/2: provisions a second:moving_ininstance pinned to the destination box; it warms from the source over the account peer plane (same-account mesh discovery) while the source keeps serving.- Reconciler: promotes the target once it has caught up — readiness is the peer-plane bootstrap gate (
status.phase == "Ready"viacaught_up?), not the public/upprobe (amoving_inhas no public endpoint). Promotion atomically swaps host ownership (source:none → :moving_out, target:moving_in → :none) and re-renders both manifests, so the customer host has a single owner throughout and there are never two same-host Ingresses. The source drains for a window, then is destroyed. - Only the
:nonerow carries the customer host (owns_customer_host?), so the cutover needs no controller change — an emptypublicHostalready leaves the Ingress/DNS/Certificate unreconciled. Cache-endpoint teardown is move-safe (the draining source shares the customer URL with the promoted target).
Placement & scaling
Cross-account spread uses the shipped egress + cpu/memory bin-packing (fill a box, spill to the next). Scaling a region = replicas: on the fleet MachineDeployment.
Follow-ups (not in this PR)
- Operator/dashboard
move_serveraction. - Optional
*.kura.tuist.devwildcard cert on bare-metal gateways to make the move cutover fully gap-free (today a per-instance ACME cert issues at promotion; fail-open covers the brief gap). - Staging 2-box e2e (routing + warm move + box-loss).
Validation
- Go:
go vet+go test ./...ininfra/kura-controller(per-account DNSEndpoint publish/skip/teardown). - Elixir:
mix compileclean; unit tests added for move manifest rendering,move_server(happy/non-movable/concurrent/non-active), and reconciler promote/warming/drain. Fullmix testruns in CI (local test-env recompile was pathologically slow this session).
🤖 Generated with Claude Code
Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.
Start a Flight and preserve its objective, outcome, and session here.
No GitHub comments yet.