fix(kura): bound bootstrap tmp-dir staging so large accounts converge

What

Bound the temp-dir staging a kura node does while bootstrapping an account from its peers, so a node can always converge to Ready under a fixed KURA_TMP_DIR_MAX_BYTES budget regardless of how large the account’s cache is.

Why (production incident)

After the production kura StatefulSets rolled (and a new node joined), every freshly-created cache pod for the tuist account stayed 0/1 — GET /ready returned 503 forever, so the pod was held out of its headless Service and the mesh ran degraded on the surviving older replicas. The decisive log line from a stuck pod:

bootstrap from <peer> failed: tmp dir budget exhausted:
  7887454208 bytes staged, 784550880 bytes remaining

Readiness is gated on bootstrap completing (note_bootstrap_* / peers_needing_bootstrap in state.rs). A bootstrap that always fails on the budget → readiness never settles → permanent 503.

Note this was not the 0.10.4 segment-ring cache change (a prior investigation cleared that and it tests correct in isolation). The 0.10.4 correlation was incidental: the new pods happened to roll onto 0.10.4, and bootstrapping the now-large account cache is what tripped the limit. The older -0/-1 pods are fine only because they bootstrapped long ago when the cache was small — a fresh 0.10.3 pod would fail identically. Rolling back would not have fixed it.

Root cause

Unbounded concurrent staging, not a reservation leak (the per-artifact temp file is removed promptly after it’s copied into a segment). Up to DEFAULT_BOOTSTRAP_MAX_CONCURRENT_PEERS (8) bootstrap tasks each stream artifacts into the same tmp dir, and the only guard was a point-in-time, non-reserving ensure_tmp_dir_capacity() that just measured the dir and returned, holding nothing. With slow network fetches, many peers held large partially-written temp files open at once, so peak tmp usage scaled with peers × artifact size rather than with the budget — and overshot for a large account. (Secondary latent bug: a chunked response with no Content-Length requested the full 2 GiB ceiling and could fail even a single artifact.)

The fix

• New TmpBudget (atomic reserved-bytes counter + Notify) and an RAII TmpReservation guard in utils.rs. reserve(bytes) reserves against the capacity and waits when full instead of erroring; Drop releases and wakes waiters. A request larger than the whole budget is clamped to the budget so a single oversized artifact still makes progress once the dir drains (the per-body byte ceiling still applies while streaming).
• Wired an Arc<TmpBudget> (sized to tmp_dir_max_bytes) into AppState. In the bootstrap path each artifact reserves min(size, budget) before streaming and holds the reservation until after it’s persisted; the racy ensure_tmp_dir_capacity() call is removed.
• The waiter registers (Notified::enable()) before the capacity check, so a release()/notify_waiters() racing the check is not a lost wakeup — without this the final waiter could miss the final release and hang, re-creating the same 503 by a different path.

Because the sum of concurrently-held reservations can never exceed the budget, peak tmp staging is now O(in-flight) not O(account size), and stagers wait rather than fail, so bootstrap always converges. (tokio::sync::Semaphore isn’t usable here — acquire_many takes a u32, which can’t represent multi-GB byte budgets, hence the custom AtomicU64 budget.)

Validation

• Full kura suite: 256 passed, 0 failed; cargo fmt --check and cargo clippy --all-targets clean.
• New regression tests: tmp_budget_* (cap/serialize-far-exceeding-capacity/clamp-oversized) and replication::tests::bootstrap_succeeds_when_total_artifacts_exceed_tmp_budget.
• replication::tests::concurrent_peer_bootstraps_converge_and_bound_peak_tmp drives 8 peers streaming slowly/concurrently, samples on-disk tmp size, and asserts peak ≤ budget. Verified it fails against simulated pre-fix code with the exact production error class (tmp dir budget exhausted: ...) and passes with the fix.

Default budget

KURA_TMP_DIR_MAX_BYTES’s default (4 × MAX_REPLICATION_BODY_BYTES = 8 GiB) does not need raising once this lands — peak tmp is now bounded by in-flight artifacts, not account size.

🤖 Generated with Claude Code