Hive Hive
Sign in

feat(cli): compress cache artifacts with LZFSE and bump cache version to 6

GitHub issue · Closed

Metadata
Source
tuist/tuist #11637
Updated
Jul 5, 2026
Domains
Cache
Details

Paired PR. This is the open-source half of a two-repo change. The archiver switch lived in tuist/TuistCacheEE#69, now merged; this PR carries the cache-version bump, dependency wiring, EE storage tests, and the submodule pointer bump. The submodule points at the merged EE main commit 650dc80.

What

Compresses remote binary/module cache artifacts with AppleArchive/LZFSE instead of storing them in uncompressed zips, and bumps CacheVersion (v5 -> v6) so the format change is safe across CLI versions.

This PR:

  • Bumps CacheVersion.current to version6 in CacheVersionFetcher.
  • Adds TuistAppleArchiver to the TuistCacheEE target (and its test target) in Module.swift.
  • Updates the TuistCacheEETests storage tests to build AppleArchive fixtures instead of store-zip fixtures.
  • Updates the CacheVersionFetcher unit test to expect version6.
  • Bumps the cli/TuistCacheEE submodule pointer to the merged EE main commit 650dc80.

Why

A customer reported cold cache downloads around 7 GB. The cache artifacts are archived with ZIPFoundation’s default compressionMethod: .none, i.e. stored uncompressed - every zip entry is “Stored 0%”. The payloads are Mach-O binaries that compress very well, so downloads carried essentially the raw xcframework bytes. Root cause: FileArchiver -> FileSystem.zipFileOrDirectoryContent -> FileManager.zipItem(...) never passes a compressionMethod, and ZIPFoundation defaults it to .none. This was an unexamined default, not a deliberate decision.

Validated results

Benchmarked the three archivers on a real 2.45 GB local cache (340 xcframeworks), reproducing the exact upload/download call paths (store = today’s ZIPFoundation, deflate = zip with compression, LZFSE = AppleArchive):

method size vs raw compress decompress
store (today) 100% 163 MB/s 304 MB/s
deflate (zip) 23.9% 48 MB/s 167 MB/s
LZFSE (this PR) 22.2% 277 MB/s 445 MB/s
  • ~76% smaller archives than store on a representative cache (2.45 GB -> ~0.54 GB), which is the download saving.
  • LZFSE compresses marginally smaller than deflate, ~6x faster to compress, and ~2.7x faster to decompress. Deflate was too slow to justify keeping the zip format (this matches the earlier decision to use TuistAppleArchiver over deflate for sharding), and since the format change forces a cache-version bump regardless, LZFSE’s format-change cost is already paid.
  • Every LZFSE archive round-tripped losslessly through decompress.

Why a cache-version bump

AppleArchive (.aar) is a different, Apple-only format that older, zip-only CLIs cannot decompress. Without a version bump they would resolve a v5 key, download an .aar, and fail to unarchive it. Bumping to v6 moves the new artifacts into a key namespace old clients never resolve, so they cleanly take a cache miss and rebuild instead. Both the legacy and regional binary/module cache are covered because both derive item hashes from CacheGraphContentHasher, which folds the version into the hash. This implies a one-time mandatory cache re-warm on upgrade, which is expected and accepted.

Concurrency-safety fix (validated in EE #69)

Review of the archiver rewrite caught a regression: the upload batch is keyed by (hash, name) and can legitimately hold two items that share a hash with different names (package products), but the rewrite archived every item to temporaryDirectory/<hash>.aar in a shared batch directory. Run concurrently, those two items raced on the same path and could upload a truncated or wrong artifact. The old FileArchiver isolated each item in its own temp directory. Fixed by naming each archive after its per-item signature-stripped staging directory (a fresh UUID), restoring per-item isolation. Download paths were already safe (each uses its own per-call temp directory).

Impact

  • ~76% smaller cold cache downloads for cache-heavy projects.
  • One-time cache re-warm on upgrade to the CLI carrying v6 (expected and accepted).
  • Lower egress on both the consumer and Tuist storage sides.

Validation

  • Benchmarked store/deflate/LZFSE on the real 2.45 GB cache (numbers above); every LZFSE archive round-tripped losslessly.
  • TuistCacheEEUnitTests builds green against both storages after the concurrency fix (xcodebuild build-for-testing -scheme TuistCacheEEUnitTests -> TEST BUILD SUCCEEDED).
  • Storage tests updated to build AppleArchive fixtures; CacheVersionFetcher test updated to version6.

How to test locally

  1. Check out this branch with the submodule at the paired EE commit (git submodule update --init cli/TuistCacheEE).
  2. TUIST_EE=1 tuist generate Tuist TuistCacheEE --no-open, then build the Tuist-Workspace scheme.
  3. Run the EE storage suites: TuistCacheEETests/CacheRemoteStorageTests and TuistCacheEETests/ModuleCacheRemoteStorageTests.
  4. End-to-end: warm a remote cache with this CLI, wipe the local cache, and confirm the pulled artifacts are .aar, decompress correctly, and the download is materially smaller than on main.

Coordination checklist

  • Merge tuist/TuistCacheEE#69.
  • Re-point cli/TuistCacheEE to the merged EE main commit (650dc80).
  • If this branch is rebased onto a newer main, keep the submodule at a commit whose EE tree matches the core it builds against.
Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments
T
tuist[bot] Jul 3, 2026

🛠️ Tuist Run Report 🛠️

Previews 📦
App Commit
Tuist e11d5fcf3
Tests 🧪
Scheme Status Cache hit rate Tests Skipped Ran Commit
TuistAcceptanceTests 7 % 118 0 118 65575cd7f
TuistApp 5 % 28 0 28 65575cd7f
TuistUnitTests 7 % 920 33 887 65575cd7f
Builds 🔨
Scheme Status Duration Commit
TuistAcceptanceTests 2m 28s 65575cd7f
TuistApp 3m 25s 65575cd7f
TuistUnitTests 2m 37s 65575cd7f
Bundles 🧰
Bundle Commit Install size Download size
Tuist 27c346e62 19.7 MBΔ -8.2 MB (-29.33%) 14.8 MBΔ -4.0 MB (-21.37%)