Kura: per-pod egress ceiling is bypassed on host-network regions

GitHub issue · Open

Open on GitHub

Metadata

Source

tuist/tuist #12363

Updated

Aug 14, 2026

Domains

Kura

Details

Summary

On the bare-metal Kura regions, the per-pod egress ceiling (kubernetes.io/egress-bandwidth, enforced by the Cilium bandwidth manager) does not apply to the customer read path. The shaper itself works correctly — it is simply not in that path. A cache pod can therefore serve reads at many times its configured ceiling, which is the opposite of what the setting is there for.

The ceiling exists so that “one account’s restore burst can’t monopolize the shared box NIC” (server/lib/tuist/kura/regions.ex). On every region whose gateway is host-network, that protection is currently absent.

Mechanism

The Cilium bandwidth manager works in two halves:

A BPF program on the pod’s egress stamps each packet with an Earliest Departure Time derived from the pod’s configured rate.
An fq qdisc on the node’s native device paces packets to those timestamps.

The stamp alone slows nothing; the pacing happens at the physical NIC.

The customer read path on a host-network region is:

kura pod  ->  co-located hostNetwork ingress-nginx  ->  client

The first hop is node-local. Packets cross the pod veth into the host namespace and are consumed by an nginx socket. They never reach the native device, so fq never paces them.
The second hop originates in the host network namespace. Host-namespace traffic carries no pod identity, so a per-pod annotation cannot apply to it.

Reproduction

Measured against a Tuist-owned instance (our own account, no customer data) on a bare-metal region node whose pod is annotated kubernetes.io/egress-bandwidth: 500M. A 200 MiB object was pushed to the cache and read back over three paths, from two throwaway client pods.

Path	Description	Throughput
A	cross-node, direct to the pod IP (pod traffic over the NIC)	462–479 Mbit/s
B	cross-node, through the regional host-network ingress	903–958 Mbit/s
C	same-node, direct to the pod IP (node-local)	13.5–15.5 Gbit/s

Concurrency is the decisive part, because the pacer is per-endpoint and a real cap must hold as an aggregate:

Streams	A aggregate	B aggregate
1	438–459 Mbit/s	534–609 Mbit/s
4	479 Mbit/s	958 Mbit/s
8	462 Mbit/s	903 Mbit/s

Path A is invariant under concurrency at the annotated rate minus VXLAN/IP/TCP overhead — the signature of an aggregate shaper. Path B roughly doubles when streams are added and settles at about twice the ceiling. Path C, which never touches the NIC, ran at ~27× the ceiling.

Path B’s true ceiling was not found; it is bounded by the client’s own network path and TLS cost, not by the cache. The finding rests on B exceeding the cap and scaling with concurrency while A does neither.

The configuration is correct — this is not a misconfiguration

Verified on the same node that served the measurements:

BandwidthManager:  EDT with BPF [BBR] [<native devices>]
Routing:           Network: Tunnel [vxlan]   Host: BPF

qdisc mq <handle>: root
qdisc fq  <handle>: parent ... limit 10000p flow_limit 100p ... horizon 2s horizon_drop

mq root with an fq child on every hardware queue, on every native device. The measured pod’s endpoint is present in the BPF bandwidth map at its annotated rate. Same endpoint, same map entry, same node — three different outcomes, decided only by whether the bytes cross the native device as pod traffic.

Impact

The per-tenant burst ceiling governs none of the customer read path on host-network regions. A single tenant’s restore burst can take the whole box NIC.
The scheduler-side reservation is unaffected but is also only a scheduling promise: tuist.dev/egress-mbps bin-packs guaranteed floors against the node budget the CAPI provider advertises (infra/cluster-api-provider-tuist/controllers/shared/node_egress.go). Nothing enforces it at runtime, and the ceiling that was supposed to do the enforcing is bypassed.
A box can therefore exceed the egress budget it advertises, with no shaping and no alerting to show it.

Where the settings live

Region ceiling (egress_burst_mbps) and floor (egress_guaranteed_mbps): server/lib/tuist/kura/regions.ex
Rendered onto the pod: server/lib/tuist/kura/provisioner/kubernetes_controller.ex
Applied to the pod template and the extended-resource request: infra/kura-controller/controllers/kurainstance_controller.go
Node budget advertisement: infra/cluster-api-provider-tuist/controllers/shared/node_egress.go

Possible directions

Shaping has to move to where the bytes actually leave the box, or the data plane has to stop traversing a host-network proxy:

Shape at the ingress. A per-ingress limit_rate / limit_rate_after on the generated Kura Ingress, driven by the same per-region value that produces the annotation today. Closest to the current intent, and it applies exactly where the bytes leave.
Shape in the host namespace. A per-backend cap applied to the gateway’s own egress. More invasive and harder to attribute per tenant.
Bypass the host-network proxy for the data plane so pod traffic reaches the NIC as pod traffic and the existing pacer applies unchanged. Largest change, but it makes the current mechanism correct rather than adding a second one.

Whichever direction is chosen, the observability gap should be closed alongside it: the fleet ingress controllers currently emit no nginx_ingress_controller_* series and no access logs, and the fleet nodes have no TCP retransmit or disk I/O metrics, so neither saturation nor per-request outcomes can be seen today.

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.

No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments

pepicrft Aug 17, 2026

Spent some time on this against the source. Here’s what I propose, plus the three alternatives I discarded so we don’t relitigate them later.

What I propose

Replace the regional gateway with HAProxy and enforce the per-tenant ceiling there, using a shared bwlim-out filter keyed by the tenant hostname.

Why this one:

It’s a true aggregate. One limiter per tenant across every connection, so it holds under concurrency. That’s the property path A has and path B doesn’t, it’s the decisive test in the description, and nothing else on the table passes it at comparable cost.
It covers both protocols. HTTP and REAPI/ByteStream both traverse the gateway, so there’s no protocol-shaped gap.
It survives replica handover. The gateway is the aggregation point, so the two-replica overlap during primary handover doesn’t turn into two independent buckets.
Tenant attribution already exists. Each hostname resolves to exactly one backend and each cert carries only that hostname, so keying on it is safe and needs no new identity plumbing.
It’s the only option with a credible path to real wire-level enforcement. HAProxy’s set-fc-mark sets the mark on packets sent to the client from an L7 expression, which is the prerequisite for a classful qdisc keyed by tenant. Nothing else gets us there.

The cost should be priced up front: this is a gateway migration on four regions, including the regex path split for /google\.bytestream\. and the REAPI prefixes, the backend-protocol split, and the HTTP/2 window and timeout tuning. On why nginx can’t stay: there’s no standard directive that sets SO_MARK on the downstream socket from request context (proxy_bind transparent marks the upstream connection, which is the wrong socket), so keeping it would mean a custom dynamic module and a patched image, and permanent coupling to nginx internals.

What this deliberately does not cover: TLS overhead, TCP retransmissions, bytes already queued in socket buffers, replication, and anything else on the box. It’s a forwarded-data ceiling, not a physical one, and we shouldn’t describe it as the latter.

If that residual turns out to matter, the follow-on is set-fc-mark plus a classful HTB tree at the physical interface, with Cilium’s bandwidth manager disabled on these pools rather than coexisting with it. Since all customer traffic already goes through the host-network proxy, pod-level pacing contributes nothing to the customer read path today, so we lose almost nothing and we remove an ownership conflict that would otherwise silently revert on a Cilium upgrade. It’s enabled globally in infra/k8s/mgmt/bootstrap/cilium-values.yaml, so it needs a CiliumNodeConfig override plus an agent restart, the tree needs a single privileged DaemonSet as its sole owner rather than being hand-managed, and it’s worth benchmarking whether one classful root goes CPU bound at 3 Gbit/s before committing. I’d make that its own decision once we have the observability to tell whether it’s needed.

What I discarded

limit_rate on the generated ingress (option 1 in the description). It’s a no-op as currently configured, not merely weak. The ingress-nginx limit-rate annotation requires response buffering, and streamingIngressAnnotations sets proxy-buffering: off on both generated ingresses (infra/kura-controller/controllers/kurainstance_controller.go:1949). nginx’s non-buffered upstream path disables limit_rate, and grpc_pass is non-buffered by nature, so the directive does nothing on either location. Enabling buffering would make the HTTP side active but it stays per request, so eight streams get eight limits, which is the path B behaviour we’re trying to remove. It would also put buffering back into a streaming path we unbuffered on purpose.

An aggregate token bucket inside Kura. This was my first instinct and I wanted it to win: instance identity is tenant identity, it’s topology invariant, and it would apply equally on the NodePort and ClusterIP regions. It doesn’t hold up, for reasons that aren’t fixable by implementing it well:

Kura counts a payload byte once. Under loss the wire carries it repeatedly, with no fixed payload-to-wire multiplier, so a payload counter can’t make a claim about a physical NIC.
Kura can write into gateway and kernel buffers while the client-facing path is stalled. On recovery the gateway releases the accumulated backlog, which is bounded by memory and connection count rather than by the configured bucket depth, and grows with concurrency.
There isn’t one write path. Ordinary HTTP, gRPC ByteStream, and the accelerated HTTP/1 sendfile/splice transfers are separate implementations (kura/src/accelerated_file_serving.rs, kura/src/app.rs:361), so a response-body wrapper misses the fast path by construction.
An instance is two replicas. During primary handover, established connections drain on the old primary while new ones reach the new one, so two process-local buckets permit roughly 2C.

If anyone reaches for this anyway, don’t reuse BandwidthLimiter as is: acquire advances next_available before it sleeps (kura/src/bandwidth.rs:46), so a cancelled request leaves reservation debt that stalls later traffic.

Bypassing the host-network proxy for the data plane (option 3 in the description). Moving TLS into Kura disables the sendfile accelerator (kura/src/accelerated_file_serving.rs:296), though a TLS sidecar sharing the pod’s network namespace would avoid that specific problem. The real blocker is everything else: per-instance public addressing on OVH and Dedibox, DNS lifecycle, and the per-replica aggregation problem coming straight back. Largest change of the three for the smallest marginal gain over the proposal above.

Related things I found while digging

The peer plane has the same bypass. peer_demux_controller.go runs a host-network L4 SNI-passthrough demux on :7443, so bootstrap responses take an equivalent local-proxy path. Any fix that only covers :443 leaves this open.
Kura’s replication limiter can’t protect the NIC either. The default is 512 MiB/s (kura/src/config.rs:97), roughly 4.3 Gbit/s, above both the ~1 Gbit and ~3 Gbit interfaces.
The 25 Mbit enterprise floor is requested per replica, so a two-replica instance consumes two reservations in the scheduler’s bin-packing. Separate bug, worth fixing before we add real floors.
The observability gap has a specific address: infra/helm/k8s-monitoring/values.yaml sets useDefaultAllowList: false with an explicit metric list carrying no TCP retransmission and no disk-operation series.

Sequencing

Observability first. Gateway metrics and access logs, TCP retransmit and disk metrics, an external synthetic probe. Right now we can’t tell whether a fix works or regresses.
HAProxy with shared bwlim-out on one regional gateway, deployed alongside nginx on a test hostname before any cutover.
Roll to the remaining regions.
Kernel-level enforcement (marking plus HTB, Cilium BWM off) as a separate decision, informed by what step 1 shows.
Floors last, after the per-replica double-count is fixed.

Acceptance has to measure at the physical interface over short intervals, not as a whole-run client-side average, and it has to be rerun after restarting the gateway, Cilium and the node. The failure mode here wasn’t a wrong value, it was a correct configuration that traffic stopped traversing, so the test needs to be shaped to catch exactly that. Alerting should cover both halves: the configuration disappeared, and the configuration is still present but bytes are landing outside it.

The thing that shouldn’t survive in any case is the current state, where egress_burst_mbps reads as an enforced ceiling and enforces nothing on four regions.