What
Group-commit the segment durability fsync in the kura store so concurrent artifact writes share a single fsync instead of one fsync per write. This removes the last serialization point on the cold-mesh-bootstrap write path that the parallel bootstrap fetch (#11464) left in place.
Why
Every artifact write (put_blob_artifact → append_reader_to_segment) held the global segment_write_lock across an fsync (sync_data). So all writes — foreground CAS saves and inbound peer replication alike — serialized at one fsync per write, capping write throughput at roughly 1 / fsync_latency, single-threaded, no matter how many writers were in flight.
This is the missing half of the cold-mesh-bootstrap throughput work. #11464 made a fresh node fetch bootstrap artifacts concurrently (buffer_unordered(BOOTSTRAP_ARTIFACT_FETCH_CONCURRENCY), 16 at a time) so it can finish a full bootstrap within KURA_BOOTSTRAP_TIMEOUT_MS instead of streaming serially. But all 16 of those concurrent fetches converge on the same per-write fsync and re-serialize there — the fetch is parallel, the durability is not, so the intended speedup is throttled back down to one-fsync-at-a-time. The same ceiling bites the cold-start replication storm, where many segments replicate into a recovering node at once.
Group-commit lets those N concurrent bootstrap/replication writes share ~1 fsync, so a fresh node finishes bootstrap — and a degraded mesh re-converges — materially faster, and the win grows with how much write concurrency is in flight (i.e. exactly the cold-start regime #11464 targets).
Reads were never the problem; this is purely the write/durability path.
Out of scope: this is unrelated to the Xcode-CAS-vs-public-endpoint benchmark. That slowness was a client-side routing issue (the launchd CAS daemon never reached the PN node), fixed separately in #11501; the warm-cache benchmark is read-mostly and does not exercise this path.
How
- Appends stay serialized and ordered under
segment_write_lock (and still fsync the outgoing segment on rotation) but no longer fsync per write. Each append reserves a monotonic pending_seq.
- Durability happens after the lock in
ensure_segment_durable: the first writer to win a separate fsync_lock performs a single fsync of the active segment and advances a durable_seq watermark to the latest reserved sequence. Writers already <= durable_seq return without syncing.
- Correctness across rotation and the lock-free fsync race rests on one invariant: a segment is fsynced when it rotates out, so only the active segment can ever hold un-synced bytes. If the active segment rotated between a writer’s append and the group fsync, the rotation already made that writer’s bytes durable; otherwise the group fsync covers them. Either way every append
<= durable_seq is durable.
Net effect: N concurrent writes produce ~1 fsync instead of N, and it batches harder under more load.
Validation
- TDD test
concurrent_artifact_writes_batch_segment_fsyncs: slows the fsync via a BeforeSegmentFsync failpoint, fires 16 concurrent writes (matching BOOTSTRAP_ARTIFACT_FETCH_CONCURRENCY) and asserts they batch. It fails on the first commit (16 fsyncs, RED) and passes here (GREEN); its wall-clock drops ~0.92s → ~0.10s.
- Companion test
concurrent_replicated_artifact_applies_batch_segment_fsyncs proves the same batching on the replication-apply path a bootstrapping node actually drives (apply_replicated_artifact_from_bytes → the shared persist_artifact_from_path_with_version), so the parallel bootstrap fetch isn’t silently re-serialized at the fsync.
- New metric
kura_segment_fsyncs_total (group-commit + rotation fsyncs), sampled from the store snapshot. Its rate against kura_artifact_writes_total is the on-node signal for the cold-node gate below — fsyncs should track batches, not writes.
- All 276 unit tests pass, including the durability/persistence/rotation tests (
segment_state_snapshot_survives_reopen, sweep_orphaned_segments_reclaims_crash_window_segment_and_metadata, bytestream_writes_persist_completely_under_concurrency).
cargo fmt --check and cargo clippy --all-targets both clean.
Reviewer notes
- This is the durability-critical write path; the watermark/rotation reasoning above is what to scrutinize. The load-bearing invariant is “rotation fsyncs the outgoing segment.”
- Not yet load-tested on a real node. Kept as a draft pending a cold-node validation: bring up a fresh node against a populated account (or replay a replication storm into a recovering mesh) and confirm the bootstrap/replication wall-clock drops (existing
kura_bootstrap_duration_seconds) while rate(kura_segment_fsyncs_total) stays well below rate(kura_artifact_writes_total) under concurrent inbound writes. The warm-cache CAS benchmark is the wrong instrument here — it’s read-mostly and exercises the separate routing path fixed in #11501.
🤖 Generated with Claude Code
F