Presence-based sync for content-addressed Kura artifacts

Summary

Kura’s replication and bootstrap decide whether to move an artifact by comparing version_ms, a wall-clock stamp assigned at write time. For the content-addressed artifact class — where the client’s key is itself a content digest, so the same artifact_id always means the same bytes — that comparison can never choose between different contents. And for the action-cache class, whose keys are deterministic action digests, byte wobble between recomputations of the same action is not information either: any stored result is a valid answer. In both cases the stamp comparison only generates redundant work — re-fetches, re-appends, segment churn, and digest buckets that read as divergent while holding equivalent data.

This spec removes version_ms from every decision it currently drives:

• Presence is the sync criterion. An id already held locally is satisfied, whatever its stamp, at every gate — one rule, with one holdback: generic keyvalue keeps today’s behavior until its key contract is audited and stated (see the exception below).
• The anti-entropy digest folds the artifact_id alone, uniformly for all entries. Adds and removes flip buckets; nothing else does.
• Re-publishes are metadata-only. A publish whose key the node already holds records the new stamp in the manifest and never rewrites the stored bytes.

What remains of version_ms is bookkeeping: it is still recorded on every publish because two node-local mechanisms legitimately consume a write time — the action-cache TTL sweep and the snapshot watermark — but it no longer decides what is stored, what is transferred, or what is walked.

Seven days of production data say this targets the dominant traffic: reapi is 85.1% of egress bytes, and 79.4% of bootstrap apply-checks are already re-verification of resident data, with the resident-but-redundantly-refetched share on top of that, decided purely by whose wall clock stamped later.

Current state

The two artifact classes

Everything Kura stores is an artifact whose artifact_id is a hash of the logical key (producer, tenant, namespace, key). Content-addressing is transitive: it holds only when the client’s key is itself a content digest.

• Blob class (content-addressed): REAPI CAS blobs plus the Xcode, Gradle, and Module blob paths. Same id, same bytes — by contract.
• ActionCache class: REAPI action results and generic keyvalue entries. For REAPI action results the key is a deterministic action digest and the wobble-tolerance is spec-backed: recomputations may produce byte-different but semantically equivalent values, and any of them is a valid answer. For generic keyvalue the same tolerance is a client assumption, not a stated or enforced contract: mechanically, the surface is an arbitrary key-to-value API.

One asymmetry matters for scoping: REAPI CAS enforces the content contract, rejecting uploads whose bytes do not match the digest in the key. The HTTP blob paths accept bytes unvalidated, so their “same id, same bytes” is a client promise, not a server invariant.

The keyvalue side carries the same kind of asymmetry. Today’s keyvalue consumers derive their keys deterministically from content or configuration, so any stored value for a key is acceptable in practice — but nothing states or enforces that. A consumer using a stable key with genuinely latest-wins semantics (a “current” pointer, a rolling index) would be silently broken by first-value-wins: its updates would never take effect on nodes already holding the key, and unlike the action cache there is no blob-liveness gate to age the stale value out into a miss.

The version gate lives in three places

1. The persist/apply gate in the store (persist_artifact_from_path_with_version and the inline variant): an incoming artifact short-circuits to IgnoredStale only when it is already present and its stamp is not newer than the stored one.
2. The bootstrap pre-check (Store::artifact_apply_outcome, consulted per manifest listed in a walked page): a resident artifact whose peer copy carries a newer stamp is classified Applied, which triggers a WAN body fetch of bytes already held.
3. The live receive gate (internal_replicate_artifact): the same pre-check, run before reading the request body, so stale live pushes already cost no body bytes. That property is worth preserving.

Why the gate is effectively inert

Client write paths stamp version_ms with the current time, which always beats the stored copy. So the IgnoredStale short-circuit essentially never fires for a re-upload: byte-identical content re-appends to a fresh segment (orphaning the old copy until reclaim) and enqueues replication every time. The gate that looks like dedup is, on the client-write path, a no-op.

The protocol makes this doubly wasteful. REAPI clients are expected to check before uploading — FindMissingBlobs before CAS writes, GetActionResult before a republish — and send only what the server reports absent. A redundant upload reaching the server is therefore already the protocol’s exception path (a check-to-use race, an eviction between check and use, a non-conforming client) — and today each such exception is amplified into a fresh segment append, an orphaned old copy, and a replication fan-out.

Three related facts close off common misconceptions:

• version_ms is not a retention or recency signal. Segment eviction is a FIFO ring by segment creation order, and keep-alive is the access-driven read-path promotion worker; neither consults the stamp. Nothing is lost by ignoring it for sync.
• The outbox enqueue is already conditional on the Applied path, so stopping redundant enqueues is a consequence of fixing the pre-check, not a separate mechanism.
• Kura already proves metadata-only refresh is viable: the action-cache refresh damping compares exact bytes and tag and suppresses the rewrite of an identical republish inside its window. This spec makes damping’s conclusion the rule — and drops the byte comparison, since under a deterministic key wobble carries no information worth acting on.

The digest folds the stamp

The anti-entropy digest (Store::manifests_digest) partitions the sorted id space into buckets and folds (artifact_id, version_ms) per entry. Stamp-only divergence between two nodes holding equivalent data therefore renders buckets mismatched, and bootstrap walks — and fetches — data both sides already hold. Over seven days, 39.6% of digest buckets were walked.

ActionCache needs no stamp or value convergence

Per the REAPI spec, GetActionResult promises only “a cached execution result”: any stored result for an action digest is a valid answer, entry lifetime is implementation-specific and recency-of-use based, and servers may legitimately store differing bytes per node. Cross-node value drift is bounded by the blob-liveness gate: an entry referencing evicted blobs degrades to a miss, never a wrong answer — and a pathological stored value (one whose referenced blobs never landed) self-heals through the same path: liveness gate → miss → cascade-delete → the republish arrives at an absent key and stores normally. So there is no correctness requirement forcing stamps or values to converge across nodes, and every mechanism that pays to converge them buys nothing.

The Tuist branch tag needs no anti-entropy either: it is not part of the cache key (lookups are tag-blind), its only consumer is the trunk-scoped snapshot filter, and tag changes propagate as ordinary republishes through live replication. A node that misses a retag keeps the old tag until the key is next republished to it; the consequence is a temporarily diverged snapshot view, which degrades to tag-blind per-key lookups — a miss at worst.

Proposal

Presence at every gate

All three gates stop consulting the stamp. An id already held locally means satisfied — no byte comparison, no rewrite, no fetch, no re-append, no fresh segment copy. For REAPI action results this holds even when a recomputation wobbled the bytes: the first stored value stands, and under a deterministic action digest any recomputation is equally valid. The live receive gate keeps rejecting before it reads the body.

The one exception: generic keyvalue keeps today’s last-writer-wins behavior for now. Its wobble-tolerance is an unstated client assumption (see current state), and first-value-wins is only safe once the contract is audited and documented — keyvalue keys must be deterministic derivations of their value’s inputs; latest-wins overwrite semantics are unsupported. Until then, applying the presence rule there would trade measurable waste for a silent correctness risk.

The branch tag plays no role in any of this. It is manifest metadata, exactly like the stamp: a republish updates it through the existing sticky-branch rules as part of the same metadata-only write (below), and it appears in no gate, no fold, and no rewrite rule.

Outside that holdback, last-writer-wins by stamp stops deciding anything. The redundant outbox enqueue disappears for free, since enqueue already fires only on the applied path.

Re-publishes refresh metadata; content is never rewritten

On the client write path, a publish whose key the node already holds becomes a metadata-only operation: record the new version_ms — and, for ActionCache, the tag per the existing sticky-branch rules — in the manifest, a RocksDB metadata write, and keep the stored bytes untouched. No segment append, no orphaned copy, no byte comparison.

The stamp refresh is not decorative: it is what feeds the two legitimate consumers of a write time — the action-cache TTL sweep (a refreshed entry stays alive) and the snapshot watermark (a refreshed entry re-enters deltas). A metadata-only refresh also has nothing to replicate: the stamp no longer participates in anti-entropy, and receivers would presence-ignore the body anyway. The stamp update is node-local bookkeeping.

Fold only the artifact id — one uniform rule for all entries

The digest fold becomes artifact_id alone, for every entry regardless of class. Adds and removes still flip buckets; nothing else does.

• For the Blob class this is load-bearing, not optional: presence-based applies with a stamp-folding digest would leave stamp-only divergence permanently mismatched and recreate the non-terminating re-walk by another road.
• For the ActionCache class it drops reconciliation that nothing needs: value wobble, stamp wobble, and retags all stop flipping buckets, with bounded, self-healing divergence as argued above.
• The uniform rule is also the simplest implementation: no per-class branch inside the fold loop, no class detection at digest time.
• During the keyvalue holdback, generic keyvalue entries keep folding the stamp — the temporary per-class branch exists precisely because a latest-wins consumer’s updates must keep their anti-entropy backstop for as long as latest-wins might be load-bearing. It is removed when rollout step 4 lands, at which point the fold is one uniform rule.

One constraint remains: the fold change must be negotiated, like the digest endpoint itself. Kura rolls with mixed versions live; a peer folding differently would see permanent bucket mismatch. The negotiation covers the fold algorithm version, with the existing full-walk fallback for a peer that does not speak it. The digest only decides which ranges to enumerate — apply semantics are unchanged — so a fold defect can at worst cause an unnecessary walk, never a wrong apply. That property is preserved.

Instrumentation first

The measurement story is largely in place: PR #12043 splits the ignored_stale outcome into ignored_equal (converged duplicate), ignored_stale (genuine one-directional skew), and ignored_missing (peer advertised then 404’d the body), across the bootstrap pre-check and the persist paths. One residual gap remains: applied still conflates “absent locally” (a legitimate fetch) with “present but the peer’s stamp is newer” (exactly the redundant fetch this spec eliminates). A follow-up split of applied in the same style lands the direct before/after measure and should ship ahead of the behavior change.

Rollout order

1. The applied outcome split (measurement baseline).
2. Presence gates and metadata-only refresh for REAPI CAS (content contract server-enforced) and the REAPI action cache, together with the uniform id-only fold (one negotiated algorithm version).
3. Widen the presence gates to the Xcode/Gradle/Module blob paths only after adding upload-time hash validation there — until the server enforces “same id, same bytes”, presence-based sync would silently trust a contract it cannot check.
4. Extend the presence rule to generic keyvalue only after auditing its consumers and stating the key contract (deterministic key derivations only; latest-wins overwrites unsupported) — the keyvalue analogue of step 3’s validation gate.

What it buys

• No WAN transfer, segment re-append, or orphaned duplicate copy for equivalent content — on the class that carries 85.1% of egress — and no local churn for the re-uploads that slip past the clients’ own check-before-upload protocol.
• Fewer artificially divergent digest buckets everywhere: stamp-only Blob divergence, ActionCache republish wobble, and retag churn all stop driving walks, so warm bootstrap passes walk less and converge faster.
• A cleaner diagnostic surface: after the change, ignored_equal volume during a re-walk collapses into digest matched buckets that are never walked at all, and a divergent bucket means exactly one thing — the key sets differ.

One deliberate non-claim: this does not address eviction-driven bootstrap churn (the eu-central signature where applied tracked segment evictions and no buckets matched) — that content was genuinely absent, and a presence check cannot skip absent content. This spec removes redundant movement of present content; capacity-driven re-fetching is a different problem.

Alternatives considered

• Keep last-writer-wins rewrites for differing ActionCache values (earlier drafts of this spec): stamp-ordered rewrites churn bytes to arbitrate between equally-valid recomputations of a deterministic action — wobble is not information, and the arbitration decides nothing a client can observe. The tag needs no special rule either: it is manifest metadata, refreshed in the same metadata-only write.
• Fold (artifact_id, branch) for ActionCache (an earlier draft): reconciles the branch tag through anti-entropy — but the tag is not part of the cache key, its only consumer is the snapshot filter, retags already propagate through live replication, and a missed retag degrades to a tag-blind per-key lookup. Paying walk traffic and a per-class fold branch to backstop a filter-only field is not worth it.
• Normalize stamps instead of ignoring them (rewrite version_ms to a canonical value): converges the digest without touching the gates, but requires a data migration across every node’s manifest column family, breaks the TTL sweep and snapshot watermark that legitimately consume write times, and still leaves the inert client-write gate re-appending segments locally.
• Fold-only change, gates untouched: shrinks bootstrap walks but leaves the live path re-fetching and re-appending equivalent bytes whenever stamps differ; half the benefit for most of the rollout risk.
• Do nothing: the current behavior is not wrong, only wasteful — but the waste lands on the dominant traffic class, and the gate’s dedup appearance is misleading enough that it has already cost analysis time in two incident investigations.