Account cache volumes for the macOS runner fleet

Summary

This RFC pivots the macOS runner module-cache architecture from on-host Kura serving (spec #75, PR tuist/tuist#11701) to per-account cache volumes: an APFS disk image per account on each Mac host, CoW-cloned per job, attached to the runner VM as a block device, and mounted at the Tuist cache root. The volume carries the entire cache directory, not just Binaries/: cached Project.swift/Package.swift manifests, compiled ProjectDescription helpers, plugins, and the mapper’s persisted metadata. That makes VM freshness irrelevant instead of making the cache wire shorter, which is why it beats on-host serving: benchmarking showed on-host Kura removes only the smallest of the three per-job cold costs. Kura is untouched: the private-network Kura node remains the source of truth, receives all uploads, and serves cold volumes’ pulls, so the hub-and-spoke topology and “a job never depends on partially replicated state” decisions from spec #75 carry over verbatim.

The scope is deliberately the managed, behind-the-scenes Tuist cache volume, and macOS runners only for now. The substrate underneath (named account-scoped volumes with a CoW branch-per-job lifecycle and dispatch affinity) is designed so that user-declared generic volumes, the customer-facing product spec #69 (remote container builders) proposes in the direction of what Namespace and Blacksmith sell, are an additive step on the same primitive rather than a redesign; see “Paving the way for generic volumes”.

Motivation

Every runner job boots a fresh VM, so every job starts with an empty Tuist cache. The cost of that is not just the binary pull. Measured on the staging M2 fleet (prod-shape 6 vCPU VM, the tuist fixture, ~233 cached targets), a job pays three cold costs before compiling anything:

1. Manifest and helper compilation: minutes on manifest-heavy projects (the cached-manifest and ProjectDescriptionHelpers directories are empty in a fresh VM).
2. The mapper’s cold-metadata phase: ~12s of single-threaded Mach-O metadata loading and hashing over freshly materialized artifacts. This state is persisted in the cache directory, which is why a prior tuist cache in the same environment pre-warms it; a fresh VM never has it.
3. The binary pull: ~4 to 5s over the private network. Validation showed this cost is largely hidden under generate concurrency even at PN distance (~2s of visible wall-clock), so eliminating it buys modest wall-clock; what it buys unconditionally is the per-job PN egress (~420 MiB on this fixture) and EM-node load.

Costs 1 and 2 dominate the wall-clock and are untouched by any serving-side change, because they are not transfers: they are state that only exists after work has been done inside an environment. On-host Kura addressed only the wire inside cost 3, which is why it was superseded.

A mounted volume deletes all three at once: the artifacts are already at their final paths (a local cache hit, no pull, no materialization), the manifests and helpers are already compiled (their cache keys are content plus tuist-version plus environment hashes, with no machine binding, so they are portable across VMs), and the metadata phase is warm. The residual per-job cost approaches the warm-generate floor, which is generate-pipeline work (hashing, mapping, generation) and is a CLI concern, not a cache-architecture one (see the follow-on CLI track below).

Current state

• The production data plane is off-host: a shared elastic-metal (EM) Kura node on the private network, reached via a per-account NodePort. It stays, as source of truth and cold-path server.
• PR tuist/tuist#11701 (on-host Kura serving) is mergeable and ships dark. Under this RFC it stays dark and unmerged: the superseded alternative.
• Spec #75 specified the production gates for on-host serving (peered replication, resource governance for host processes, fan-out safety). It is archived in favor of this RFC; its topology decisions (hub-and-spoke, PN Kura as single upstream, no job depends on partially replicated state) are inherited here.
• Dispatch binds accounts to runners at job-assignment time, not at pod placement: runner VMs boot as an unbound shared warm pool, poll the server, and the server’s dispatch match picks a queued job for the polling runner. Volume affinity therefore lives in the dispatch scoring, not in Kubernetes scheduling.
• The cache directory is one root: CacheDirectoriesProvider places binaries, manifests, projectDescriptionHelpers, plugins, selectiveTests, editProjects, and runs under Environment.current.cacheDirectory. One mount warms everything; the volume carries the whole root, with no category splitting (the non-cache categories are small, and splitting the root would add mount complexity for no measured benefit).
• The local binaries cache is already LRU-bounded by the CLI: every tuist generate runs CacheLocalStorage.clean(maxAge: 7 days) and every local cache hit refreshes the entry’s modification time, so the cache converges to the last-7-days working set wherever it lives. An unused maxEntries parameter exists as an additional bound.
• Manifest cache keys are portable (CachedManifestLoader: manifest content hashes, tuist version, environment hash). Binary artifacts are not: ArtifactSigner signs each cached artifact with a payload that is the machine’s MAC address (encrypt(payload, signableAttribute: \.macAddress)), validated on load. A volume written by one VM would fail validation in the next VM.
• tart-kubelet already manages VM disk images (golden-image clones per pod) and already stages a per-VM virtiofs share for the cache endpoint marker. Tart supports additional block-device attachments (tart run --disk); the kubelet wrapper does not pass them yet.
• Measured mechanics on the fleet hardware: APFS clonefile of large trees and image files is instant CoW with no VM penalty (0.48s for a 2 GB tree, versus 4.8s byte-copy at a 3.2x virtualization penalty); virtiofs-style metadata operations are the profiled hot path of the mapper (getattrlistbulk storms), which is why the volume must be a block device rather than a directory share.

Proposal

The volume model

Per account, per host, a sparse APFS disk image (the master) under the host’s runner-cache root. Per job:

1. The reconciler APFS-clones the master image file (instant, CoW) into a per-VM branch.
2. The branch is attached to the runner VM as an additional block device (tart run --disk).
3. The guest mounts it and the job’s environment points the Tuist cache root at it (runner-image change; mount plus XDG_CACHE_HOME or a symlink of the cache root). The attached volume auto-mounts a few seconds after boot, so the runner-image step must gate on the mount before the job’s first tuist invocation (observed in validation; a bounded wait suffices).
4. On job end, the guest reports through the existing per-VM share whether the job actually changed the cache (a dirty marker: artifacts added or evicted, manifests or helpers compiled). The reconciler promotes the branch to be the new master (atomic rename; last-writer-wins) only on job success and dirty; otherwise the branch is discarded. A missing marker (crashed job) means discard. The gate is not an optimization nicety: under last-writer-wins, a read-only job’s unchanged branch promoted after a concurrent writer’s branch would silently clobber the writer’s freshly captured warmth. Pure-hit jobs do technically write (the LRU bookkeeping refreshes modification times on every hit), but mtime-only deltas do not count as dirty: their value is bounded, and any job that pulls or compiles something new promotes and carries its refreshed mtimes along.

Where the read/write logic lives: almost nowhere new. The reconciler in tart-kubelet owns the whole image lifecycle (create and format on first use, clone, attach, promote or discard, admission, eviction, rebuild), as a second image kind next to the golden images it already manages. The runner image contributes only the mount gate and the dirty marker. The CLI is unchanged as a cache reader and writer: reads are its ordinary local cache hits, misses pull per-item from the PN Kura exactly as today and materialize into the directory. There is deliberately no separate volume-warming or sync code path: warming a cold or partial volume is the CLI’s normal miss-fill behavior happening to write into a directory that persists. The only CLI/EE touches are policy (the byte-budget prune and the signing grant), not I/O. The server only scores affinity and mints grants; it never touches volume data.

A cold (empty or absent) volume is the status quo, not a degraded mode: the job pulls from the PN Kura exactly as today, and those pulls warm the volume as a side effect for the next job. There is no wait window, no readiness marker, and no coupling between a warming host and the fallback path.

Volume size management

Two scopes need bounding: what is inside one volume, and what all volumes together take from the host. The inside bound is content policy; the host bound must be enforcement, not policy, because running the host out of disk breaks the VM clone path and fails jobs, which is the one non-degradable failure in this design.

Inside a volume: content is pruned to fit, by size, not only by age (v1 requirement). The CLI already runs CacheLocalStorage.clean(maxAge: 7 days) on every tuist generate, and every local cache hit refreshes the entry’s modification time, so pruning is true LRU by last use. Age alone is not sufficient: if an account’s 7-day working set exceeds the volume, the age prune removes nothing, the volume hits ENOSPC mid-pull, an arbitrary subset of artifacts never fits, and the full volume keeps being promoted (persistent churn). The v1 invariant is therefore a byte budget of ~80% of the volume’s capacity (headroom for one job’s writes), enforced in the same per-generate clean hook that runs today: clean evicts least-recently-used entries past the budget. The existing clean already sorts by last use and supports maxEntries; the maxBytes variant is a small EE change, injectable per runner environment like the signing payload, and the 7-day age bound stays as the secondary cap. Manifest and helper subtrees churn with the tuist version; the same hook prunes non-current-version subtrees. With the byte budget in place, an oversized working set degrades gracefully instead of churning: the volume becomes a hot tier. LRU keeps the most-used artifacts local, and the tail misses per-item to the PN Kura exactly as today (partial warmth plus remote fill, still strictly better than cold).

Across the host: four layers keep volumes from exhausting the disk. Sparse masters are over-committed by design (N masters times the provisioned cap can exceed the disk on paper), and a reactive evictor alone can be outrun by a burst of jobs writing into their branches between reconcile ticks, so the bound is layered:

1. A filesystem-enforced ceiling. The runner-cache root lives on its own quota-bounded APFS volume (APFS supports per-volume quotas at creation), sized at host provisioning as what the disk leaves after golden images, the maximum concurrent pod clones, and OS headroom. Even with a buggy reconciler, cache volumes can then only ENOSPC themselves, which degrades jobs to the cold path; they can never consume the disk the VM images and golden clones need. The quota statically encodes the priority that cache volumes always lose to the VM path.
2. Admission control at clone and create time. Before cloning a master for a job, or creating a new master for a first-contact account, the reconciler checks that the quota volume’s free space covers the worst-case job growth. The v1 estimate is the conservative one, the per-volume provisioned cap, refined to an observed p95 per-job delta once telemetry exists. If it does not fit, it synchronously evicts least-recently-used masters to make room; if room still cannot be made, the job runs without a volume, on the status-quo cold path. Warmth is the only thing ever sacrificed.
3. Background watermark eviction. The reconcile loop keeps free space above a low watermark (starting value: 20% of quota) by evicting whole masters LRU, stops admitting new masters below a high-pressure mark (starting value: 10% free), and (if TRIM does not propagate; see below) rebuilds bloated images. Eviction is pure account-LRU in v1; size-aware weighting (evicting large cold masters before small ones) is adopted only if telemetry shows large masters crowding out many small accounts. Masters are disposable by construction: deleting one costs exactly one status-quo cold job for that account on that host.
4. Truthful accounting. Free space (statfs) on the quota volume is the ground truth for admission and watermarks. Per-file allocated sizes cannot be summed for this (CoW clones share blocks, so a master plus its branch double-counts); per-file stats only rank eviction victims.

All starting values (budget fraction, watermarks, growth estimate) are configuration, tuned from the admission and eviction telemetry rather than re-litigated in this spec.

Physical image mechanics. The APFS volume inside each image is fixed-size at creation, so a single image can never exceed its provisioned cap, and because the image is sparse the cap can be generous (20 GiB comfortably held this fixture) with the quota above as the real aggregate bound. The auto-grow signal is host-observable with no new channel: a master whose allocated size sits near its provisioned cap at promote time is under content pressure, and the reconciler grows it at the next rebuild. Whether an image file tracks live content or drifts toward its cap depends on guest TRIM/discard propagating through the block attachment to hole-punch the host sparse file; Virtualization.framework supports discard for disk-image attachments, and tart’s attach path gets a one-off verification during implementation. If TRIM propagates, nothing more is needed; if not, the host periodically rebuilds a master (fresh sparse image, copy live content in off the job path, atomic swap through the same promote mechanism), triggered when allocated size diverges from live content.

One behavior to verify during implementation: a job that fills its volume mid-run must degrade to cache-write failures (the cache is best-effort; a full cache must not fail the job) and its branch must be discarded, not promoted. The byte-budget headroom makes this rare rather than routine.

Host lifecycle: volume inventory, affinity, and fleet-wide distribution

The reconciler manages master images the same way it manages golden images today: create on first use, clone per job, promote or discard on job end, evict whole masters by LRU under the disk watermark, and (if TRIM does not propagate) periodically rebuild masters to reclaim dead blocks. Three questions determine whether the model works at fleet scale.

How many volumes a machine holds. A budget division, not an emergent property: the resident-master count is the volume quota divided by typical allocated master size, self-adjusting per account working set. For example, a ~400 GiB quota over 10 to 15 GiB typical allocations holds roughly 25 to 40 account masters; the admission and watermark layers above keep the set within quota as working sets differ. The set is adaptive by construction: the accounts that actually run on that host keep their masters hot, cold accounts age out. No configuration decides “which accounts live here”; dispatch traffic does.

How jobs land on hosts that already have the volume. Account binding happens in the server’s dispatch match, so affinity is a scoring change there, not a Kubernetes scheduling change. Today’s match has no preference of any kind: a polling pod resolves to its fleet via its ServiceAccount (its node is never consulted), Jobs.pick_queued returns exactly one candidate (the strictly oldest queued job on the fleet, deterministically ordered so all concurrent pollers converge on the same row), and the winner is decided by a Postgres uniqueness in Claims.attempt. Affinity is three small additions to that flow:

1. Node identity at dispatch. The claim flow already reads and patches the polling pod (owner-label stamping), so its spec.nodeName is one field away.
2. The affinity signal. On every claim, upsert (node_name, account_id, last_run_at) into a small Postgres table, with rows pruned after 14 days (past any plausible volume lifetime under LRU). That record is the “this host recently ran this account, so it likely holds its volume” signal: a volume exists where the account’s jobs ran, and eviction staleness only costs the status-quo cold path, which also re-warms the volume. This dispatch-history inference is the v1 signal; hosts reporting actual volume inventory through the runner-session channel is the upgrade if the affinity hit rate shows history is too stale.
3. Top-K selection with an age tolerance. pick_queued returns the K oldest queued jobs instead of one (starting value: K = 20), and the server hands the polling runner the oldest job affine to its node only if that job’s enqueue time is within the age tolerance (starting value: 30s) of the queue head; otherwise the head. The tolerance is the precise operational meaning of the hard rule that affinity never delays a job: it bounds how long any job can be passed over and prevents starvation of accounts that hold no volume anywhere. Both values are configuration, tuned from the affinity hit rate and queue-latency telemetry.

A side effect improves dispatch contention: today every poller fights over the same single candidate and losers retry; per-node preference diverges candidates across hosts, reducing lost claim races, while the Postgres claim uniqueness remains the arbiter of correctness.

How the distribution stays fair when volumes cannot exist everywhere. Affinity creates a deliberate feedback loop: an account’s jobs concentrate on the hosts that hold its volume, which keeps fewer masters per account fleet-wide and each of them hotter. Two guards keep the loop healthy. First, saturation spillover: when an account’s affine hosts are busy, the job runs on any available host, organically creating a new volume exactly where demand outgrew capacity, so hot accounts widen their footprint and it shrinks again via LRU when demand recedes. Second, anti-monopoly by construction: an account has at most one master per host, size-capped by its provisioned image, so no account can crowd a host’s volume budget beyond its own cap; across accounts the watermark LRU keeps each host’s resident set proportional to actual recent demand. Small accounts are not starved: they warm a volume on first contact with any host and keep it as long as they keep running there. If history-based affinity proves too noisy at scale, the evolution is deterministic placement (rendezvous hashing of account onto k preferred hosts, with k scaled by account job volume), which needs no inventory signal and self-heals on host churn; that is a tuning decision, not an architecture change, because both variants are pure dispatch-scoring policies over the same volume substrate.

Observability for all three: per-host resident-volume count and bytes, quota free space, admission rejections (jobs run without a volume for space reasons), dispatch affinity hit rate (job landed on a host holding its account volume), promote/discard/dirty rates, eviction churn, and per-account cold-start rate.

Paving the way for generic volumes

This RFC ships one managed volume, but the substrate is the volume product spec #69 needs, so v1 makes three commitments that keep generic, user-declared volumes (DerivedData, SwiftPM caches, arbitrary paths) an additive step:

• Volumes are named from day one. Masters are keyed (account_id, volume_name) on disk and in the affinity table, with the Tuist cache as the reserved name (tuist-cache). Generic volumes are then new names, not a re-keying migration.
• The lifecycle is name-agnostic mechanism; the cache specifics are policy. Clone, attach, promote-or-discard, admission, watermark eviction, and rebuild know nothing about what a volume contains. Everything Tuist-cache-specific layers on top as per-volume policy: the CLI’s byte-budget LRU self-prune (generic volumes have no in-guest pruner, so their policy is a hard quota with the user owning the content) and the artifact-signing grant (meaningless for generic content). New volume types add policy, not lifecycle code.
• The plumbing is plural. The kubelet accepts a list of volume attachments per pod (v1 passes exactly one), tart run --disk composes for multiple disks, and the guest mounts by volume label against a small manifest of label-to-path mappings, with the mount gate waiting for all declared volumes. The dispatch affinity signal carries the volume name so per-volume affinity is a query change, not a schema change.

What generic volumes add later, deliberately out of scope here: the user-facing declaration surface (per-workflow volume names and mount paths), per-account quotas and billing for user-managed bytes, the semantics documentation (volumes are caches with last-writer-wins promotion and eviction, not durable storage), and the Linux/container flavor.

Platform scope: the Tuist cache volume is macOS-only for now. It ships on the macOS runner fleet (tart VMs, APFS images, block attachments); the Linux runner fleet keeps its current cache path unchanged. The Linux flavor arrives with the container-builder work in spec #69, where the same model maps onto Linux primitives (filesystem-level CoW and block attachment of the runtime there).

Kura’s role: unchanged

Jobs continue to upload cache writes to the remote as they do today, so the PN Kura remains the single source of truth; the volume is opportunistic warm state that can be deleted at any time without correctness impact. Dev machines and non-runner CI are unaffected. No replication paths, peering, bootstrap, or Kura configuration change at all. This preserves spec #75’s topology decisions with strictly less machinery.

Artifact signing: a server-signed grant scopes the payload to the account (the key is already ours)

Grounded in the EE implementation, the “MAC-bound signature” is narrower than it sounds. ArtifactSigner stores one extended attribute (tuist.cloud.metadata) on each cached artifact: an AES-CBC-encrypted JSON payload containing the machine’s MAC address, plus a deterministic ECDSA signature over that MAC string. Both the AES key/IV and the EC signing key are Tuist-owned constants compiled (obfuscated) into the closed-source binary, identical on every machine. Key ownership was never the gap; there is no per-machine key material at all. The machine binding lives entirely in the payload semantics: isValid decrypts the xattr and compares the embedded MAC against the current machine’s (ArtifactSignaturePayloadProvider.fetch()).

A raw environment-variable substitution would break the product boundary, so the substitution is gated on a credential. If the CLI simply read a scope value from the environment, validation would compare the artifact’s embedded payload against whatever the environment provides, and any actor (a third-party runner fleet, a laptop farm) could pick an arbitrary consistent value and make caches portable across their machines: the cross-machine bar would drop from “extract the obfuscated keys from the binary” to “set an env var”. Instead, dispatch delivers a server-signed grant: {scope: account-<id>, expires_at} signed by the Tuist server with a dedicated key pair (distinct from the artifact-signing key) whose public half is baked into the EE binary, the same trust root as today; rotation works by shipping the successor public key in the binary before the server switches, the standard key-roll. ArtifactSignaturePayloadProvider substitutes the grant’s scope for the MAC only after verifying the grant’s authenticity and expiry; no grant, an invalid grant, or an expired grant falls back to the MAC default, exactly like a dev machine, and verification is fully offline (no server round trip, so unreachability also degrades to MAC). The grant’s TTL is the job’s maximum lifetime plus a small margin, delivered over the same dispatch/JIT channel that already carries the account binding. The scope is per account, matching the volume’s tenancy: the volume is account-scoped, so a narrower per-project scope would fracture warmth inside one volume while project isolation is already carried by the cache keys themselves. Artifacts are signed with the scope (stable across jobs, so volume warmth persists) while grants rotate per job. Grant minting stays internal to dispatch in v1 and is deliberately not exposed on any public API; offering customer-infrastructure grants (the Namespace-style self-hosted case) would be an explicit product decision, not a side effect.

Leakage is assumed and priced in: runner jobs execute arbitrary user code that can read the grant from the environment, but a leaked grant expires within the job’s TTL, so sustaining cross-machine reuse outside Tuist runners would require continuously harvesting fresh grants from live runner jobs of one’s own account. Anyone in that position is an authorized account member who can already download the same artifacts through the remote cache API with their account token: no new threat class, and the bar for outsiders stays exactly where it is today (reverse-engineering the binary’s baked keys).

The failure mode the grant fixes is silent, not fatal: on a foreign-MAC VM every signed artifact fails validation, and CacheLocalStorage treats it as a cache miss (surfacing a misleading “requires using the Tuist-provided server” warning) and re-pulls from the remote. Two consequences follow. First, shipping the volume infrastructure without the EE change does not break jobs; it forfeits only the binaries hit, while the manifest and helper warmth still applies in full because those caches are not signed. Infra and the EE fix can therefore land and be observed independently. Second, any single-VM benchmark reuses one MAC and silently sidesteps the gate, so validation of the signing change specifically requires a cross-VM (fresh-MAC) probe. Both behaviors were confirmed by measurement (see Measured validation).

This is not a weakening. The current payload (the MAC address) is readable by any local process, so the existing signature is tamper evidence rather than a security boundary. A grant-attested account scope asserts “produced under this account’s runner identity”, which is a stronger, more meaningful claim than “produced on this machine”. Dev machines and third-party CI keep the MAC-derived default; nothing changes outside Tuist-dispatched runner environments.

Why a block device and not the existing virtiofs share

The mapper’s hot syscall is bulk directory metadata (getattrlistbulk), profiled during the cold-metadata phase. Virtiofs is weakest exactly there; a block-attached APFS volume gives local-speed metadata and reads, and keeps clonefile semantics inside the guest. The existing virtiofs share stays for what it is good at (tiny marker files: the cache endpoint marker today, the dirty marker with this RFC), or is retired with the on-host path.

Concurrency and tenancy

Two VMs of the same account on one host produce two branches; promotion is last-writer-wins among dirty, successful branches, and the loser’s warmth is simply not captured, which is acceptable for a cache (the dirty gate in the volume model prevents the worse case, a read-only branch clobbering a writer’s promote). Merging branches on promote (binaries merge trivially by content hash, manifests by key) would capture both jobs’ warmth and is a possible later refinement, deliberately not in v1: last-writer-wins is atomic, trivially correct, and the loss is bounded to one concurrent job’s delta. A job can only mount its own account’s volume (the account binding is dispatch-time, as today), and can therefore only poison its own account’s cache, which it can already do by uploading to the remote with its account token: no new threat class. Cross-account isolation is by construction (separate images), stronger than path-scoping inside a shared filesystem.

Measured validation (staging bench, 2026-07-10)

End-to-end validation on the staging bench rig (M2 mini, prod-shape VM, tuist fixture, 233 cached targets): five arms, five interleaved reps each, a fresh VM boot per rep. vol boots with a CoW clone of a warmed 20 GiB sparse APFS master image attached via tart run --disk and the cache root pointed at it. warm uses a warm root-disk cache dir with binaries cleaned, so it re-pulls (the best case any serving-side architecture can reach). cold starts from an empty cache dir (today’s one-shot runner). The pull-based arms ran at two serving distances: host-local, and across the real Scaleway PN vlan (0.8 ms RTT, 1 Gbps) from an identical Kura with the same dataset on the second fleet mini, which is shape-faithful to the EM data plane. Medians:

arm	generate 1 (pre-build)	generate 2	binary fetch	build	gen1 + build
vol (warm volume attached)	20.4s	12.5s	none	15.8s	36.2s
warm + PN Kura	21.3s	12.2s	~5s	14.8s	36.0s
cold + PN Kura (today’s one-shot runner)	39.6s	12.2s	~5s	14.4s	53.6s
warm + host-local Kura	19.4s	12.1s	~3s	14.5s	33.6s
cold + host-local Kura	37.3s	12.2s	~3s	14.2s	51.3s

Host-side clone of the master image: 27 to 42 ms. All arms ended every rep with all 233 artifacts and zero fetch failures. A 20 GiB sparse master comfortably held this fixture’s full cache directory (sizing data point).

What the numbers say:

• Against today’s real path (cold + PN), the volume is 49% faster pre-build (39.6s to 20.4s) and 32% faster through the build (53.6s to 36.2s). That gap is almost entirely the manifest, helper, and metadata warmth; on manifest-heavy customer projects (minutes of manifest compile versus this fixture’s ~17s) it grows accordingly.
• The pull is largely concurrency-hidden even at PN distance. Moving the serving point from host-local to the PN grew the fetch window from ~3s to ~5s but gen1 by only ~2s. Versus warm + PN, the volume is ~1s faster on gen1 and a wash on gen1 + build for this fixture (reads from the attached device cost ~1s of build). The volume’s per-job margin over any pull-based serving is therefore small on wall-clock at this artifact volume; what it removes unconditionally is the ~420 MiB of per-job PN egress and EM-node load, and the wall-clock margin scales with artifact volume.
• The residual 20.4s is generate-pipeline cost, not cache architecture. Timeline decomposition of the vol gen1: ~2.5s process start plus config and server handshake, ~4s SwiftPM package-graph load, ~7s graph transform plus hashing through cache-hit resolution, ~4s workspace and project generation, ~1s side-effect writes, ~2s synchronous run-metadata upload before exit. Zero fetch work. The same run repeated in the same boot (gen2) takes 12.6s from the same volume, so ~8s of gen1 is first-run-after-boot page-cache tax spread across all phases. Further reduction is CLI work, tracked as the follow-on CLI track below.
• Trust-gate probe (fresh MAC). A cloned VM (different MAC) booted with the same warmed volume: every artifact failed signature validation, the CacheLocalStorage warning fired, and gen1 silently re-pulled all 233 artifacts (22.7s, versus 20.4s same-MAC), re-signing them for the new machine. This confirms by measurement that without the EE payload change the volume still captures the full manifest and metadata win and nothing breaks, and that the EE change is what converts the remaining per-job re-pull (plus its PN egress) into local hits.
• Operational finding: the staging macOS fleet’s PN data plane is currently unreachable from the minis. The EM node was reprovisioned onto a different private network (172.16.0.6, which does not ARP from the minis’ vlan), and the readiness heartbeat is derived from cluster-state observation rather than fleet-side reachability, so dispatch keeps handing out an endpoint macOS jobs cannot dial. It went unnoticed because the islanded host-Kura staging experiment took over serving. Needs a separate fix (and argues for a reachability-based component in the heartbeat).

Follow-on CLI track (surfaced by validation, not part of this RFC)

Volumes take today’s per-job pre-build plus build from ~54s to ~36s on this fixture. The decomposition shows the next ~6s is CLI pipeline work, orthogonal to cache architecture but compounding with it:

• Cache the mapped SwiftPM external dependency graph (~2s warm, ~4s cold-boot per generate on this fixture). SwiftPackageManagerGraphLoader/PackageInfoMapper re-map ~90 packages’ PackageInfo into Tuist’s graph on every generate even though the mapping is deterministic in (workspace-state hash, manifest hashes, tuist version). Serialize it with the CachedManifestLoader pattern; the serialized graph then also rides the account volume, so runner VMs skip the mapping entirely. In progress.
• Fix the run-metadata upload tail (~2s observed, pathological). The CLI retried a 348 KB POST /api/analytics four times against a deterministic 403 (RetryProvider retries all errors, including non-retryable 4xx), synchronously before exit. Make 4xx terminal and the upload non-blocking.
• Parallel or incremental hashing and generation (the remaining ~11s core: ~7s transform plus hash, ~4s workspace generation). The largest remaining lever for every tuist generate everywhere, runner or not.

Scope

In scope: the per-account Tuist cache volume on macOS runner hosts only (kubelet plumbing for --disk, name-keyed image management, promote/discard with the dirty gate, admission, eviction, compaction, the quota-bounded cache volume on hosts), the dispatch-time volume-affinity scoring (server), the guest-side mount gate, cache-root pointing, and dirty marker (runner image), the byte-budget LRU prune (TuistCacheEE), and the grant-gated account-scoped signing (TuistCacheEE plus server-side grant minting in dispatch). The module cache is the first consumer of the volume substrate, which is built name-keyed and plural per “Paving the way for generic volumes”.

Out of scope, but aligned: the follow-on CLI track above; the user-facing generic-volume product (declaration surface, quotas, billing, semantics) that spec #69 describes; the Linux/container flavor; cross-host volume warmth via shared masters in object storage (a later evolution, and the natural convergence point with the container-builder volume distribution in spec #69); the REAPI batched-fetch work for the cold pull path (independent, still worthwhile for cold volumes and dev machines); grants for customer-hosted (non-Tuist) runner infrastructure, which would be a deliberate product extension.

Trade-offs

Advantages

• Attacks all three per-job cold costs, including the two (manifest compile, metadata phase) that no serving-side architecture can reach; measured pre-build within ~1s of the fully warm same-machine floor.
• No per-account host processes, which dissolves the macOS resource-governance problem (no cgroups) that was spec #75’s hardest gate; the host runs nothing new, it only manages files.
• Zero Kura changes; source of truth, upload path, and topology are untouched, and cold behavior equals today’s behavior.
• No new data plane in the CLI either: reads and writes are the existing local-cache code paths, and volume warming is the ordinary miss-fill behavior against a directory that persists.
• Removes the per-job binary transfer from the PN (~420 MiB/job on this fixture), relieving EM-node load and making job cost independent of artifact volume.
• Content growth is self-bounding: the CLI’s per-generate LRU prune (age today, byte budget with this RFC) rides the volume, so masters converge to each account’s working set, and an oversized working set degrades to a hot tier over the remote rather than failing.
• Host disk exhaustion is prevented by construction: an APFS quota fences the whole subsystem, admission control gates growth up front, and every space pressure degrades to the cold path, never to failed jobs.
• Affinity is a dispatch-scoring policy over the shared warm pool: no Kubernetes scheduling changes, and mispredictions cost only the status-quo cold path.
• Reuses proven mechanics: the reconciler’s existing image lifecycle, measured instant CoW cloning, and the runner account binding.
• Builds the spec #69 volume substrate (named, plural, policy-layered) with the module cache as its first consumer, converging the cache and compute roadmaps on one primitive.

Disadvantages

• Warmth is per host and per account in v1: a cold host still pays one status-quo job per account (mitigable later with shared masters in object storage).
• Warmth depends on dispatch affinity actually landing jobs on volume-holding hosts; a fleet much larger than per-account concurrency means some fraction of jobs run on non-affine hosts and pay the cold path (observable as the affinity hit rate).
• Builds read artifacts from the attached device at a measured ~1s penalty versus the root disk on this fixture; accepted, since moving artifacts to the root disk costs a cross-volume byte-copy (measured 3.2x VM penalty), which is strictly worse.
• Last-writer-wins promotion discards one concurrent job’s warmth occasionally.
• Disk footprint is per account times hosts, fenced by the host quota and per-master caps; needs admission, watermark eviction, and image-space reclamation (TRIM or periodic rebuild) from day one.
• Accounts whose working set exceeds the volume budget get partial warmth (hot tier) rather than full warmth; the remainder pulls from the PN as today.
• Manifest warmth churns with CLI releases (cache keys include the tuist version); binaries are unaffected.
• Requires an EE signing change plus server-side grant minting, and a runner-image change; the sparse-image hygiene is real operational surface (we hit sparse-image growth pathologies first-hand while benchmarking).
• macOS-only in v1: Linux runner jobs see no benefit until the spec #69 container flavor lands.

Alternatives considered

On-host Kura serving (spec #75, PR #11701)

A persistent per-account Kura on each host, VMs pull over the vmnet bridge. Measured: removes ~1 to 2s of wire from a ~4 to 5s pull, leaves the ~2.5 to 3.6s in-VM materialization floor, and does nothing for manifest or metadata cold state. Carries the hardest gates: per-account host processes needing cooperative resource governance on an OS without cgroups, peered replication wiring, and fan-out safety. Superseded by this RFC; the PR stays dark.

Clonefiled decompressed seed

Pre-place decompressed Binaries/ on the VM volume and CoW-clone per build. This RFC is that idea, productized and extended to the whole cache directory, with the signing fix making the trust exception unnecessary.

Raw environment-variable signing scope (no grant)

Let the CLI read the substitute payload directly from an env var. Rejected: validation would accept any consistent self-chosen value, so any third-party fleet could make caches portable across its machines, collapsing the cross-machine bar from “extract the binary’s obfuscated keys” to “set an env var”. The server-signed grant keeps the bar where it is today.

Hard affinity (jobs wait for volume-holding hosts)

Queue a job until a runner on an affine host is available. Rejected: it trades bounded, observable cache warmth for unbounded queue latency, and inverts the product’s priorities (a slower-starting job is strictly worse than a colder cache, which the remote path already covers). Affinity stays a scoring preference bounded by the age tolerance.

Virtiofs share instead of a block device

Mount a host directory into the VM. Rejected on measured grounds: the mapper’s hot path is bulk metadata syscalls, virtiofs’s weakest operation; and CoW semantics across the share boundary degrade to byte copies (measured 3.2x penalty).

Read-through / partially warm serving

Rejected in spec #75 and inherited here: a job never depends on partially replicated or partially warm state it cannot fall back from. The volume model satisfies this by construction, since a cold volume just means the status-quo remote path.

Cross-host shared masters (object storage) in v1

Pulling masters from object storage would warm new hosts instantly but adds a distribution path, invalidation, and egress costs before the basic model is proven. Deferred to an evolution once per-host masters are in production.

References

• Spec #69, Remote container builders: proposes cache volumes as a product; this RFC builds the macOS substrate with the module cache as first consumer.
• Spec #75, On-host Kura for the macOS runner fleet: superseded by this RFC; topology decisions inherited.
• PR tuist/tuist#11701, on-host Kura serving (Option A): stays dark, superseded.
• Namespace cache volumes and Blacksmith sticky disks: the commercial precedents for per-job CoW cache volumes attached to ephemeral runners.

P

pedro@tuist.dev Jul 12, 2026

I like the direction, and the cache-volume approach looks like a strong fit for the macOS runner fleet. I think the volume abstraction needs to be clarified before implementation, though.

The current design models a volume as a host-local master and makes dispatch affinity responsible for finding it. Namespace’s documented model treats the volume as a logical, versioned object with multiple physical replicas, private per-job forks, last-writer-wins promotion, and cache misses when a replica is not available on the selected host: https://namespace.so/docs/architecture/storage/cache-volumes

Can we define the generic volume contract in terms of:

• logical volume identity;
• generations and promotion;
• physical replicas;
• private per-job forks;
• cache misses and stale replicas;
• replication and recovery;
• access scope and authorization?

Host affinity can still be the first implementation strategy for local Mac storage, but it should be an optimization behind the volume service rather than part of the generic volume model. Tuist-specific behavior such as cache paths, pruning, Kura fallback, and artifact signing should remain in the Tuist cache adapter.

I would be comfortable approving a host-local version one if the specification explicitly states that it is a first implementation of this logical-volume model and defines how the interface evolves when cross-host replicas are introduced.