Chunked transfer for runner cache-volume masters

Summary

The per-account cache master introduced by spec #76 is a single sparse APFS disk image, content-addressed by its inventory digest, and every convergence re-pulls the whole thing. This RFC proposes transferring it as fixed-size 1 MiB content-addressed chunks behind a manifest: the HEAD’s digest names a manifest listing chunk digests in order, a converging host fetches only the chunks it does not already hold, and the assembled image is still verified against the HEAD’s inventory digest before adoption.

Chunking is a storage and transfer representation only. The host keeps one master.sparseimage on disk and its hdiutil attach path is unchanged, so the “exactly one file crosses the virtio-fs share” invariant from spec #76 is untouched.

The headline decision is the chunk size, and it is not the one originally proposed. A 16 MiB chunk was the obvious starting point; measurement rejects it. At 16 MiB a converging host still fetches slightly over half the image, which is a rounding error rather than a win. At 1 MiB it fetches 3 to 8 percent. The rest of the design follows from that number.

Motivation

Convergence egress is roughly 2.5 TB/day fleet-wide (164 successful convergences in 24h at 12 to 25 GB each) against a measured fleet baseline of about 7 TB/day. That is the dominant single line item in the fleet’s egress, and it is pure re-transfer of bytes the receiving host mostly already has.

It also fails outright at the top of the size range. For account 2053, whose master is 18.4 GB, 18 of 20 convergence downloads failed in a 24h window, 12 of them by exhausting the 30-minute convergence deadline. That object needs sustained bandwidth in the low hundreds of Mbps just to fit the window. PR tuist/tuist#12526 made those failures survivable by resuming within and across convergences, but it treats the symptom: the account still has to move the whole image to move one job’s worth of change.

Generation N+1 is produced by APFS-clonefiling generation N and letting a job write into the clone, so copy-on-write should leave untouched blocks at their original byte offsets. If that holds, consecutive generations differ by roughly what the job wrote, and delta transfer is a large win. That property had never been measured. It has now.

Current state

• The master is one immutable object per generation, keyed under the account’s prefix by the inventory digest the guest computes on the detached image. VolumeHeads.bump_head is a compare-and-swap on the generation, and the digest doubles as the object key, which is what makes each distinct promoted set a distinct object.
• Dispatch mints exactly one presigned GET for the current HEAD’s object, in volume_head_payload. The guest stages it into the status share; the host reads it from there. The host has no authenticated channel to the server of its own. The guest is the authenticated party, which is why the promote-time upload URL is minted by the guest at the moment it knows its new digest, in volume_master_upload_url.
• The host verifies before adopting. convergeMaster measures the downloaded image’s inventory digest through a read-only attach and declines anything that does not reproduce the advertised digest. This check has already caught a real fleet-wide incident and is never softened.
• A content-addressed partial store already exists under _partials/, added by PR tuist/tuist#12526, banking an incomplete whole-image download across convergences so a large master stops treadmilling the deadline.
• Superseded objects are reclaimed on a delay by PruneVolumeMasterWorker, and rejected-promote uploads by PruneVolumeMasterOrphanWorker, both after a grace equal to the presigned-URL lifetime and both re-checking the live HEAD before deleting.
• Compaction is deliberately on hold for this volume. That is load-bearing here, not incidental: see “Compaction is now a correctness dependency”.
• Masters bloat monotonically. Freed blocks inside the image are not reused within a session, so the physical file tracks cumulative unique writes, bounded by the per-master cap and resolved by watermark eviction.

Measured validation (local replication, 2026-08-20)

Measured by replicating the fleet mechanism exactly on macOS 26.3, the same major version the fleet runs: create a sparse APFS image, clonefile it, attach it with -owners off, run a job’s worth of work, detach, and compare chunk digests between generations. The workload was modelled on the real cache layout (the CAS store plus the binary subtrees the inventory digest walks) and included the two things the guest actually does that a naive simulation would omit: the recursive chmod over the tuist subtree on every attach, and a read pass, since the guest attaches without noatime.

Content-addressed miss rate, meaning the fraction a digest-keyed store must actually fetch, for a realistic job against a 13.4 GiB image:

chunk size	must fetch	chunks per manifest
64 KiB	2.7%	222,481
256 KiB	3.8%	55,621
1 MiB	7.9%	13,906
4 MiB	21.0%	3,477
16 MiB	52.1%	870

Three findings drive the design.

Blocks do not move. At every granularity the positional diff and the content-addressed diff are identical, meaning no block appears at a different offset in the next generation. Copy-on-write preserves byte offsets exactly as hoped, and new data appends at the tail as sparse-band allocation predicts. This is what makes fixed-size chunking sufficient and content-defined chunking unnecessary.

Scatter, not job size, sets the floor at coarse chunk sizes. Across a 25x range of job write volume (20 MiB to 500 MiB added), the 16 MiB miss rate moves only from 39% to 63%. Even a 20 MiB write dirties 39% of 16 MiB chunks. Decomposing the churn at 16 MiB on a smaller image: attaching and detaching with zero I/O costs 2.2%, the guest’s own recursive chmod costs 7.4%, a read-only pass with no writes at all costs 27%, and a single 1 MiB file written costs 3.5%. Most of what dirties a coarse chunk is not the job’s payload.

Fine chunks get better as masters grow; coarse chunks do not. Running the identical job against a 4.9 GiB and a 13.4 GiB image, the absolute number of dirty 1 MiB chunks barely moves (915 to 1091), so the fraction falls with image size (17.9% to 7.9%). At 16 MiB the dirty count scales with the image (178 to 453) and the fraction stays pinned near 52%. Fine chunking improves on exactly the large masters that hurt today.

Cross-checked against production through read-only sources only (Atlas Postgres for the HEADs, Grafana Loki for object sizes, which appear in the convergence failure logs as the byte count remaining on a truncated transfer). Account 3: 12.6 GB master at generation 405, roughly one generation per 30 to 45 minutes. Account 2053: 18.4 GB master at generation 88. Since these images never reclaim, master size divided by generation approximates bytes added per generation, giving about 31 MB/generation for account 3 and about 209 MB/generation for account 2053. Both land inside the swept range, projecting to roughly 3% fetched for account 3 and 5 to 8% for account 2053 at 1 MiB, against roughly 40% and 55 to 63% at 16 MiB.

Proposal

Chunk size: 1 MiB fixed (decided)

Fixed-size 1 MiB chunks of the image file, each keyed by its own digest under the account’s existing object prefix. 1 MiB is the point where the miss rate is low enough to be a real win (3 to 8 percent, an order of magnitude better than today) while the object count stays in a range the request budget tolerates. 256 KiB is better on bytes but quadruples every count for a further gain of a few percent; 4 MiB and above give back most of the win.

Fixed-size rather than content-defined: the measurement shows blocks never move, so the insertion problem that rolling-hash chunking exists to solve does not occur here. Fixed offsets also make a chunk’s position in the manifest its position in the file, which is what lets a host patch a clone in place rather than reassemble a whole image.

The manifest is addressed by the digest the HEAD already carries (decided)

The manifest lists the chunk digests in file order. It is stored as its own object, content-addressed by the same inventory digest that keys the whole-image object today, so its key is derivable from the HEAD exactly as the image key is.

This is the whole of the server HEAD model change: there is none. runner_volume_heads keeps carrying a generation and a tree digest, VolumeHeads.bump_head keeps its compare-and-swap semantics unchanged, and the fast-forward, the cold-promote rule, and the unverifiable-HEAD retirement escape all continue to operate on the digest without knowing whether it resolves to an image or a manifest. Only the object layout under the account prefix gains members.

Host converge path: clone locally, patch the difference (decided)

A host that holds a master for the account keeps a sidecar manifest beside it, written whenever it installs or promotes a master, listing that master’s chunk digests. Convergence then becomes: read the HEAD manifest, diff it against the local sidecar, fetch the missing chunks, and assemble.

Assembly clonefiles the local master into the staging path, which is instant and shares every block, and then overwrites only the chunk ranges that differ. The host therefore writes a few hundred MB rather than re-writing the whole image, and the result is installed through the existing InstallMaster path, so the atomic whole-image replace gated on a newer generation is unchanged. A host with no local master has nothing to clone and fetches every chunk, which is exactly today’s cost and no worse.

A missing or unreadable sidecar is not a failure: the host rebuilds it by re-hashing its local master, or treats itself as cold and fetches everything.

This supersedes the _partials/ whole-image banking from PR tuist/tuist#12526 for the chunked path, and generalizes it. Each chunk is its own atomic unit, so a convergence killed halfway has already durably banked every completed chunk with no resume bookkeeping at all. _partials/ becomes a local chunk store keyed by chunk digest, kept per account, matching the existing per-account master directories: cross-account deduplication is declined deliberately, since the marginal saving on cache content is small and it would let one account’s storage residency depend on another’s content.

Minting reads for chunks, given that the host cannot ask (decided)

The host has no authenticated channel to the server, so it cannot request presigned URLs mid-convergence, and dispatch cannot pre-mint them because at dispatch time the server does not yet know which generation the host holds.

The guest closes this, exactly as it already does for the promote-time upload URL. The host stages its base generation for the guest at materialize, which is an existing mechanism. The guest asks the server for the read set, passing that base generation; the server diffs the two generations’ manifests, which it can read itself, and returns presigned reads for the difference alongside the manifest URL; the guest stages them for the host. The SSRF guard and the account authorization that already gate the single download URL apply per URL unchanged.

The server’s diff is a prediction about what the host holds, not ground truth, since the host may have been evicted or may hold a master the server has no manifest for. Over-delivery is harmless. Under-delivery falls back, per the next section. When the host’s base generation names a manifest the server no longer retains, there is nothing to diff and the host is served the whole-image fallback.

Fallback, at every layer (decided)

Convergence is best-effort and must never fail a job, so every step degrades rather than aborting:

• Manifest missing, unreadable, or naming a chunk count that does not match the expected image size: fall back to the whole-image object, and if that is gone, decline convergence and leave the host on its local master.
• A chunk that will not fetch, or that arrives not matching its own digest: retry within the deadline, then fall back to the whole image, then decline.
• Assembly completing but the assembled image failing the inventory digest check: unchanged behavior, including staging the disproved digest for the lineage-retirement escape.

That last one only stays sound because of per-chunk verification. Today a digest mismatch is proof about the object, reproducible on every host, which is what licenses retiring a poisoned lineage. If a corrupt chunk could reach assembly undetected, a mismatch would instead be a local fault, and treating it as proof would let one host with bad bytes retire a healthy HEAD fleet-wide. Verifying each chunk against its own digest on arrival is what keeps the assembled bytes a deterministic function of the manifest, and so keeps the existing retirement rule correct. It is not merely an early corruption check.

Chunk garbage collection by reference counting from live manifests (decided)

Chunks are reclaimed only when no live manifest references them. This layers onto the existing prune workers rather than replacing them: when a manifest is superseded, a sweep is scheduled after the same grace those workers already use, and it deletes only the chunks that manifest referenced which no other retained manifest also references, re-checking against the live HEAD exactly as prune_superseded_volume_master does today.

The hard requirement is that a host must never lose a chunk a manifest it is currently fetching still needs. Two independent things provide that. The grace before any sweep exceeds the maximum convergence window, which is bounded by the 30-minute convergence deadline, so a convergence that started against a then-live manifest cannot outlive the grace. And if that is ever wrong, a vanished chunk is a fetch failure, which falls back rather than corrupting anything. Safety by construction, with the residual race downgraded to a warmth loss.

Object-count budget

The counts that matter are per operation, not per bucket. A 12.6 GB master is about 12,600 chunks in its manifest, but a convergence fetches only the difference, measured at roughly 300 to 450 chunks for account 3 and 900 to 1,500 for account 2053. A promote uploads the same order. That is an order of magnitude below the roughly 17,000 requests that proved painful elsewhere against a much smaller chunk ceiling, and the comparison is favourable in the direction that matters, since that figure was a per-operation request count too.

Stored object population is the number that grows: on the order of 15,000 objects per account rather than a handful. Object count at rest is a storage-accounting concern rather than a request-rate one, but it is the quantity to watch as masters grow, because a manifest scales linearly with image size. A 50 GB master would be 50,000 chunks, at which point the chunk size deserves revisiting.

The manifest itself is one object and one request. At 12,600 entries it is small enough that no chunking of the manifest is proposed.

Compaction is now a correctness dependency

The entire win rests on untouched blocks keeping their byte offsets across generations. Compaction rewrites block addresses by construction, so enabling it would scatter the diff and collapse the hit rate back toward the whole-image case, silently: convergence would keep working and simply stop saving anything.

Compaction is currently on hold, which is what preserves the property. This RFC promotes that from a deferred decision to a stated dependency. If image-space reclamation becomes necessary, it has to be co-designed with this scheme, and the natural shape is a rebuild that republishes a fresh manifest and accepts one expensive generation, rather than an in-place compaction that quietly degrades every subsequent one.

Scope

In scope: the guest’s promote-time chunking and upload, the server’s manifest storage, read-set minting and chunk reclamation, the host’s converge and assembly path, and the metrics that show whether the chunked path is actually saving what it claims.

Out of scope: the inventory digest itself and the fast-forward semantics, both unchanged; the guest’s attach path and the single-file share invariant, both unchanged; compaction, which stays on hold; Linux runners, which have no cache volume.

Trade-offs

Advantages

• Convergence egress falls by roughly an order of magnitude on the measured accounts, against a line item currently running at about 2.5 TB/day.
• The 30-minute deadline stops binding. Account 2053 currently needs sustained bandwidth in the low hundreds of Mbps to fit its image in the window; fetching 5 to 8 percent of it needs single-digit Mbps, which turns a chronic failure into a non-event.
• The benefit grows with master size, which is precisely where the current design hurts most.
• Interrupted convergences resume for free, since every completed chunk is durably banked by construction, retiring the resume bookkeeping added by PR tuist/tuist#12526.
• Host disk writes during convergence drop from a full image rewrite to the patched difference, by reusing the same clonefile primitive the feature is already built on.
• No change to the HEAD model, the compare-and-swap, or the verification that stops a corrupt master propagating.

Disadvantages

• A new failure surface on a feature with a fragile production history: manifests, per-chunk verification, assembly, and reference-counted reclamation are all new, and all of them can only ever cost warmth, but they are still new.
• Stored object count per account grows by roughly four orders of magnitude, and a manifest grows linearly with image size, so the chunk size is a decision with a shelf life.
• The read set is minted from a server-side prediction of what the host holds, so it can be wrong in the under-delivering direction and fall back, spending a whole-image transfer on a convergence that a correct prediction would have made cheap.
• It adds a guest round trip at materialize.
• It makes the hold on compaction load-bearing, coupling two decisions that were previously independent.
• The upload-side saving is not realized until the fleet stops dual-publishing whole images, so the first phase carries the storage cost of both representations.

Alternatives considered

16 MiB chunks

The originally proposed size, rejected on measurement. It fetches 52% of the image and, unlike fine chunking, does not improve as masters grow, because the dirty chunk count scales with image size at that granularity. The saving is under 2x for a scheme that costs manifests, reclamation, and a new fallback surface. No plausible job write volume rescues it: across a 25x sweep the miss rate only moves from 39% to 63%.

256 KiB or 64 KiB chunks

Better on bytes (3.8% and 2.7%) but 4x and 16x the object and request counts. The marginal byte saving over 1 MiB is a few percent of the image, against a per-convergence request count that would rise into the thousands and a stored population into the hundreds of thousands per account. 1 MiB is the knee.

Content-defined chunking

Rolling-hash boundaries would tolerate insertion and block movement. Measurement shows neither occurs: positional and content-addressed diffs are identical at every granularity, because copy-on-write preserves offsets and new data appends at the tail. It would buy nothing and cost a rolling hash over every byte on both sides, plus variable-size chunks that no longer map to file offsets, which is what the in-place patch of a clone depends on.

Overlay layers, Docker-style

Not available. macOS has no union filesystem and APFS has no union mount, so anything layered has to be applied to produce one image rather than stacked at mount time. Chunking is that application step, done at transfer rather than at attach.

Binary-diff the previous generation

Transfer a delta computed against generation N rather than a chunk set. It would be marginally smaller than 1 MiB chunking, but it makes every transfer pairwise, so a host two generations behind needs either a chain of deltas or a bespoke diff, and the server has to compute and store deltas per pair. Content-addressed chunks are pairwise-free: any host, at any generation, fetches exactly what it lacks.

Scoped short-lived credentials instead of a minted read set

A credential scoped to the account’s chunk prefix would let the host fetch chunks directly and delete both the prediction and the guest round trip. It is the cleaner design if the storage layer supports prefix-scoped temporary credentials. Deferred rather than rejected, since it depends on a capability not currently relied on anywhere in this path. See open questions.

Keeping the whole-image object permanently as the fallback

Simple and safe, but it defeats the upload-side saving entirely and doubles storage for every generation forever. Dual-publishing is proposed for the rollout window only.

Rollout

1. Publish both representations. The guest keeps uploading the whole image and additionally uploads the new chunks and the manifest. Nothing reads the chunks yet. This is the phase that proves manifests are correct against a known-good object: the manifest for a generation must assemble into bytes that reproduce that generation’s inventory digest.
2. Read the chunked path, preferring it when it is usable. Hosts that find a manifest and hold a usable base converge through chunks; everything else takes the whole image. Both paths end at the same unchanged inventory-digest check, so a defect in the chunked path shows up as declined convergences and falls back, rather than as adopted bad masters.
3. Stop publishing whole images, once the fallback rate is low and stable and the fleet is fully upgraded. This is where the upload-side saving and the storage saving land.
4. Enable chunk reclamation only after step 3 has settled, so the window in which a sweep could race a fetch is never open while the fallback is also being exercised heavily.

Metrics gate each step, following the precedent set by PR tuist/tuist#12526 that a fix must not be able to hide what it absorbs: chunks fetched against chunks reused from the local master, bytes fetched against full image size, fallback count by reason, and assembly failures separated from fetch failures. A rising fallback rate with a flat convergence failure rate means the chunked path is quietly doing nothing while the whole-image path carries the fleet, which is the specific way this change could look healthy while having no effect.

Open questions

• Does the storage layer support prefix-scoped short-lived credentials? If so, the minted read set and the server-side prediction both disappear, and the host fetches chunks directly.
• At what master size does 1 MiB stop being right? The manifest scales linearly, so a 50 GB master is 50,000 entries. Whether that is the point to move to 4 MiB, or to a two-level manifest, is worth deciding before masters reach it rather than after.
• Should the manifest be a compact binary encoding rather than a textual digest list? At 12,600 entries it does not matter; at 50,000 across every convergence it might.
• Is the per-account chunk store the right isolation boundary, or is there enough cross-account commonality in cache content to justify revisiting it? The measurement here says nothing about cross-account overlap, and the tenancy argument stands regardless of what the numbers would show.

References

• Spec #76, Account cache volumes for the macOS runner fleet: establishes the master image, the fast-forward HEAD, and the single-file share invariant this RFC preserves. Its “Cross-host shared masters (object storage) in v1” alternative is what shipped and what this RFC now makes incremental.
• Spec #69, Remote container builders: the generic-volume direction the same substrate feeds.
• PR tuist/tuist#12526, convergence download resume and partial banking: the symptom-level fix this RFC supersedes for the chunked path, and the source of the egress and failure-population numbers in the motivation.

No comments yet

Comments from contributors and members will appear here.