Hive
fix(infra): tolerate the kura-cache taint on bare-metal Kura gateways
GitHub issue · Closed
What
Gives the host-network regional Kura gateway the tuist.dev/kura-cache toleration so it can schedule onto the bare-metal cache node, mirroring how the cache instance pods already get it.
Why
Bare-metal cache nodes (Dedibox/OVH) carry tuist.dev/kura-cache=true:NoSchedule to keep general workloads off the box. The cache instance pods tolerate it — regions.ex puts the toleration in provisioner_config, and kubernetes_controller.ex threads it into the KuraInstance manifest ("tolerations" => tolerations(region)), so they schedule and run.
The gateway never got the same treatment:
gateway_manifestsetnodeSelectorandhostNetworkbut not tolerations.KuraGatewaySpechad noTolerationsfield, andgatewayPodTemplatenever set one.
So on a bare-metal region the ingress-nginx gateway is pinned (correctly) to the one kura-dedibox node — it must run there to bind the box’s public IP — but can’t tolerate that node’s taint. Result observed in production after the eu-central cutover:
kgw-...-eu-central-controller 0/2 FailedScheduling
0/28 nodes are available: 20 node(s) had untolerated taint(s),
7 node(s) didn't match Pod's node affinity/selector.
Both replicas sit unschedulable, the region has no ingress, and the public cache endpoint never serves — even though the cache StatefulSet pods are healthy on the same node. This affects every public bare-metal region (eu-central, us-east, us-west), and it also keeps the server’s activation check from confirming the endpoint, so the dashboard stays stuck at “Deploying”.
There is no durable workaround: patching the ingress-nginx Deployment directly is reverted by the kura-controller on its next reconcile, and the CR had no field to carry the intent. It needs the code fix.
How
Mirrors the existing cache-pod path exactly:
- kura-controller
KuraGatewaySpecgainsTolerations []corev1.Toleration(json:"tolerations,omitempty").gatewayPodTemplatesetsTolerations: gateway.Spec.Tolerations.- deepcopy + the hand-maintained CRD schema (
kura.tuist.dev_kuragateways.yaml) updated to match.
- server
gateway_manifestthreads"tolerations" => tolerations(region)— the same helper (tolerations/1) the KuraInstance manifest already uses.nilis dropped by the existingEnum.reject, so cloud/LB regions are unaffected.
Since regions.ex already declares the tuist.dev/kura-cache toleration in provisioner_config for the managed regions, no region config change is needed — the gateway now simply picks up what the cache pods already had.
Validation
go build ./...,go vet, and the kura-controller gateway tests pass, including a newTestGatewayPodTemplatePropagatesTolerations.- New Elixir tests assert
gateway_manifestcarries the toleration on a bare-metal region and omits it when the region declares none. (Localmix testrun is pending; the Go side and static checks are green.) - CRD
tolerationsschema mirrors the kurainstances CRD (type: array/items: objectwithx-kubernetes-preserve-unknown-fields).
Once deployed (kura-controller image + server), the eu-central gateway schedules onto the Dedibox box and the region serves.
🤖 Generated with Claude Code
Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.
Start a Flight and preserve its objective, outcome, and session here.
No GitHub comments yet.