fix(kura): serialize replicated artifact applies per key to stop bootstrap disk amplification

What changed

Two complementary dedup layers on the replication apply/bootstrap path, so a fresh node materializes the cache at ~1x the dataset in both on-disk bytes and WAN transfer instead of ~Nx (N = peer count):

1. Per-key apply lock (kura/src/store.rs) — persist_artifact_from_path_with_version now holds a striped per-artifact_id write lock across its read-check → segment append → metadata commit. The first writer commits; concurrent writers for the same key re-read it and short-circuit to IgnoredStale without appending. Fixes the disk amplification (the production ENOSPC).
2. Per-artifact bootstrap fetch gate (kura/src/replication/mod.rs, kura/src/state.rs) — bootstrap_artifact_from_peer acquires a bootstrap-scoped striped gate for the artifact id, re-checks presence, and skips the body download if it is already applied. Single-flights the fetch across peers, fixing the network amplification.

Plus a note in kura/docs/architecture.md documenting both layers.

Why

A fresh node bootstraps from every known peer concurrently (it must, to capture artifacts that exist on only one peer in a leaderless, eventually-consistent mesh). When several peers serve the same artifact (same artifact_id, same version_ms), two distinct things went wrong.

Disk amplification — the incident

On staging (account tuist), the 2nd eu-central replica kura-tuist-eu-central-1-1 (50Gi PVC, ~9.2G real dataset) never reached Ready. All N applies of a shared key raced through the apply path: each read “absent”, each appended a full copy of the bytes to a segment, and only the last manifest write won in RocksDB — leaving the other N−1 copies as orphaned bytes. Bootstrapping from 4 peers wrote ~4x the dataset and exhausted the volume:

failed to persist bootstrap body: No space left on device (os error 28)
disk_full: insufficient free space for segment rotation: 0 bytes available, 1073741824 required

Bootstrap froze at the partial point (applied ~7,800, error climbing, bootstrap_completed_peers=0, bootstrap_inflight_peers=4); recreating the pod+PVC re-filled at the same point — reproducible amplification, not leaked garbage. segment_write_lock serialized the appends but never deduplicated them, and bootstrap_staging_budget only bounds tmp staging, not the data store.

Network amplification — the follow-up, now fixed in this PR

The apply lock stops the duplicate writes, but the bodies were still downloaded once per peer: at cold start all peer-tasks pre-check the same artifact as absent and each fetch it. For eu-central that’s ~9.2G × 4 ≈ 37G of WAN transfer to materialize 9.2G, and a slower bootstrap. The fetch gate single-flights this: the first peer-task to claim a key downloads+applies it; the rest block on the gate, re-check, and skip the network entirely.

Design notes

• Readiness is preserved. Every fetch path still returns Applied (we fetched) or IgnoredStale (confirmed present), so a clean per-peer pass still means that peer’s whole manifest is present locally. The naive “skip-set” (skip if another task is fetching) would have broken this: a peer’s pass could be marked clean while trusting an owner that later fails or leaves the mesh, letting the node serve with a missing artifact. The gate+recheck avoids that — it changes who fetches, never whether presence is verified.
• Self-healing. A failed owner releases the gate with the key still absent, so the next waiter re-checks, sees the gap, and fetches from its own peer.
• Load and concurrency. Gate ownership is decided per key by whichever task wins first, so it spreads roughly evenly across peers; ingress stays ~16-way pipelined (one peer’s worth), which saturates the link while moving 1x not Nx the bytes — net faster bootstrap.
• Two separate locks on purpose. The fetch gate is bootstrap-scoped so it never blocks live-replication applies (the node accepts writes while joining). The apply lock stays short (local append+commit only) and remains the last-line write-dedup for live replication and as defense-in-depth if a body is ever fetched twice.
• Why locks over manifest dedup. Minimal, always-on, correct for every concurrent path, and free of the readiness hazards of coordinating the independently-spawned per-peer bootstrap tasks. Striped (fixed stripe arrays) rather than per-key maps to keep memory constant for large accounts. Purely node-local — no wire/format/config change — so it is safe under the rolling-upgrade overlap.

How to test locally

From kura/:

mise run clippy        # Bazel clippy aspect, warnings as errors — passes
mise run test-unit     # bazel test //... — 278 passed, incl. the new tests

Both invariants are pinned by tight regression tests, made deterministic with a sleep failpoint between the durable append and the metadata commit:

• concurrent_cross_peer_bootstrap_fetches_and_writes_each_artifact_once (replication) — 4 peers serve identical manifests/bodies; asserts the segment store stays ~1x the dataset (apply lock) and the peers serve each body once total via a shared request counter (gate). Verified it fails without the gate: 24 body requests instead of 6 (4 peers × 6 artifacts).
• concurrent_replicated_applies_of_same_key_write_once (store) — fires concurrent same-key applies directly and asserts exactly one writes (~1x on disk), guarding the apply lock independent of the gate. Verified it fails without the lock: 4 writes instead of 1.

Validation on staging (for the reviewer / after merge)

Delete kura-tuist-eu-central-1-1 and its PVC and watch it reach Ready (2/2 for the eu-central KuraInstance) without ENOSPC.

No GitHub comments yet.