kura: two-peer bootstrap evicts its own just-fetched artifacts (CAS ring overshoot) → local-miss re-fetch thrash, bootstrap never converges

Summary

A node bootstrapping from two peers at once can drive its bounded CAS segment ring to evict artifacts it just fetched and applied. Because segment eviction also deletes the evicted segment’s backing manifests, those artifacts then read as a local miss on the next digest pass and are re-fetched — a fetch → evict → re-fetch thrash in which the node never simultaneously holds the complete dataset, so bootstrap never converges (digest_buckets{result="matched"} stays 0, ready=false forever).

Critically, this triggers even when the dataset fits in the CAS cap from a single source — the two-peer bootstrap inflates the actively-written footprint enough to overshoot.

Where (main @ 9b44fa8280)

• CAS is a bounded segment ring capped by KURA_CAS_CAPACITY_BYTES (kura/src/constants.rs:17); when full it evicts the oldest segment.
• Evicting a segment removes its backing manifests — Store::evict_segment/evict_segments (kura/src/store.rs:1711), covered by evict_segment_removes_segment_backed_manifests (store.rs:6548). Eviction is driven purely by ring age/capacity and is not bootstrap-aware.
• The bootstrap pre-check is manifest-presence based — artifact_apply_outcome → manifest_from_db (kura/src/store.rs:3225). An artifact whose manifest was evicted reads as absent → Applied, so the range walk re-fetches it (kura/src/replication/mod.rs:560-645, and bootstrap_artifact_from_peer at :700). The apply itself is correct — it stores the blob and the source’s version_ms (mod.rs:757/804) — it just gets reclaimed before the sweep finishes.

Note: the codebase already recognizes a different multi-peer disk-accumulation path — the per-artifact write lock at store.rs:702-709 stops concurrent same-key applies from each appending a full copy (“the bootstrap-from-many-peers ENOSPC”). This issue is a separate path: sequential eviction of already-committed artifacts while a bootstrap is still building toward the complete set.

Evidence — kura-tuist-eu-central-1 (#11955)

kura-tuist-eu-central-1-1 bootstraps from two peers (scw-fr-par-0 + eu-central-1-0). Since its last restart:

The node evicts almost exactly as many artifacts as it applies, never holds a complete bucket (so no digest ever agrees), and cannot reach a serving state. scw-fr-par-0 holding the same dataset without a single eviction shows the raw dataset fits — the two-peer bootstrap is what overshoots.

Why it’s slow and non-converging

Two separate effects compound:

1. Non-convergence (this issue): eviction reclaims just-fetched artifacts → local miss → re-fetch → never complete.
2. Per-pass cost: artifacts average ~5 KB and replication bandwidth is 512 MiB/s (idle), so a pass is request-bound — ~3M individual artifact round-trips at concurrency 16 (~150–170/s) plus per-artifact apply (segment-append lock) and memory-pressure-degraded RocksDB lookups. Each full sweep takes hours, and eviction keeps every sweep at full item count instead of a shrinking delta.

Impact

• A region whose working set is anywhere near the per-node CAS cap can never complete a two-peer bootstrap → permanent 0/1 → region 503 (the eu-central outage).
• Self-reinforcing across restarts: each OOM/restart re-enters the same thrash from scratch.

Proposed directions

• Bootstrap from a single complete source at a time (sequential bring-up), keeping the active footprint at ~1× — this is the concrete reason behind the existing “converge from a complete source” recovery, not just a heuristic.
• Make eviction bootstrap-aware: protect/pin segments written by the in-progress bootstrap from reclamation (or defer eviction until the pass completes), so a bootstrap can’t evict its own working set.
• Size CAS ≥ the account working set — necessary but not sufficient alone; two-peer inflation can still overshoot a cap the single-source dataset fits under.
• Detect it: alert when rate(kura_segment_evicted_artifacts_total) ≈ bootstrap applied rate while ready=false.

Related

• #11955 — the eu-central bootstrap wedge; this is the root-cause mechanism.
• #12039 (ignored_stale equal-vs-older split) and #12018 / kura@0.20.0 (per-peer pass-progress metrics) — diagnostics that surface adjacent symptoms of the same thrash.

🤖 Generated with Claude Code

No GitHub comments yet.