Hive Hive
Sign in

fix(infra): tolerate the kura-cache taint on bare-metal Kura gateways

GitHub issue · Closed

Metadata
Source
tuist/tuist #11599
Updated
Jul 5, 2026
Domains
Kura
Details

What

Gives the host-network regional Kura gateway the tuist.dev/kura-cache toleration so it can schedule onto the bare-metal cache node, mirroring how the cache instance pods already get it.

Why

Bare-metal cache nodes (Dedibox/OVH) carry tuist.dev/kura-cache=true:NoSchedule to keep general workloads off the box. The cache instance pods tolerate it — regions.ex puts the toleration in provisioner_config, and kubernetes_controller.ex threads it into the KuraInstance manifest ("tolerations" => tolerations(region)), so they schedule and run.

The gateway never got the same treatment:

  • gateway_manifest set nodeSelector and hostNetwork but not tolerations.
  • KuraGatewaySpec had no Tolerations field, and gatewayPodTemplate never set one.

So on a bare-metal region the ingress-nginx gateway is pinned (correctly) to the one kura-dedibox node — it must run there to bind the box’s public IP — but can’t tolerate that node’s taint. Result observed in production after the eu-central cutover:

kgw-...-eu-central-controller 0/2 FailedScheduling
0/28 nodes are available: 20 node(s) had untolerated taint(s),
7 node(s) didn't match Pod's node affinity/selector.

Both replicas sit unschedulable, the region has no ingress, and the public cache endpoint never serves — even though the cache StatefulSet pods are healthy on the same node. This affects every public bare-metal region (eu-central, us-east, us-west), and it also keeps the server’s activation check from confirming the endpoint, so the dashboard stays stuck at “Deploying”.

There is no durable workaround: patching the ingress-nginx Deployment directly is reverted by the kura-controller on its next reconcile, and the CR had no field to carry the intent. It needs the code fix.

How

Mirrors the existing cache-pod path exactly:

  • kura-controller
    • KuraGatewaySpec gains Tolerations []corev1.Toleration (json:"tolerations,omitempty").
    • gatewayPodTemplate sets Tolerations: gateway.Spec.Tolerations.
    • deepcopy + the hand-maintained CRD schema (kura.tuist.dev_kuragateways.yaml) updated to match.
  • server
    • gateway_manifest threads "tolerations" => tolerations(region) — the same helper (tolerations/1) the KuraInstance manifest already uses. nil is dropped by the existing Enum.reject, so cloud/LB regions are unaffected.

Since regions.ex already declares the tuist.dev/kura-cache toleration in provisioner_config for the managed regions, no region config change is needed — the gateway now simply picks up what the cache pods already had.

Validation

  • go build ./..., go vet, and the kura-controller gateway tests pass, including a new TestGatewayPodTemplatePropagatesTolerations.
  • New Elixir tests assert gateway_manifest carries the toleration on a bare-metal region and omits it when the region declares none. (Local mix test run is pending; the Go side and static checks are green.)
  • CRD tolerations schema mirrors the kurainstances CRD (type: array / items: object with x-kubernetes-preserve-unknown-fields).

Once deployed (kura-controller image + server), the eu-central gateway schedules onto the Dedibox box and the region serves.

🤖 Generated with Claude Code

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments

No GitHub comments yet.