Manifest cache regression: cache is written but never reused

Thanks for the detailed report — the ruling-out work was really useful.

Our current suspicion matches yours: helpersHash. It’s the one key shared by both symptoms — it names the compiled helpers module directory (~/.cache/tuist/ProjectDescriptionHelpers/<hash>) and is embedded in every manifest cache entry, so if it’s unstable across runs you’d see exactly this: helpers rebuilt every run and every Manifests/1.<md5> entry rewritten.

We’ll try to reproduce on our side and work on it regardless, but two things from your machine would speed this up a lot:

1. Which hash field actually moves. No patched build needed:

cd ~/.cache/tuist/Manifests
dump() { for f in 1.*; do jq -c --arg f "$f" '{f:$f,h:.hashes}' "$f"; done | sort; }
dump > /tmp/before.txt
tuist generate --no-open
dump > /tmp/after.txt
diff /tmp/before.txt /tmp/after.txt | head

Honestly, “yes, helpersHash changed” / “no, it was something else” is the whole signal — you don’t need to paste anything. If you do paste, please keep the jq selector: the sibling manifest field in those files is your full manifest JSON base64-encoded.

2. Whether xcrun writes anything to stderr:

xcrun --sdk macosx --show-sdk-version >/tmp/sdk.out 2>/tmp/sdk.err
wc -c /tmp/sdk.out /tmp/sdk.err

A non-zero size on /tmp/sdk.err would confirm a specific theory we have. Again, just the byte count is enough — the contents can include paths with your username.

Also useful if it’s easy: does ls -1t ~/.cache/tuist/ProjectDescriptionHelpers gain a new directory after each run?

Thanks for the fast triage @esnunes — ran both checks (plus one more that I think narrows it down further).

1) Which hash field changes → none of them

Clean protocol, everything pinned to 4.203.1: rm -rf Tuist/.build/tuist-derived → tuist install → warm-up tuist generate --no-open (231s) → snapshot every ~/.cache/tuist/Manifests/1.* entry’s {tuistVersion, hashes} → measured tuist generate --no-open (149s) → snapshot again and diff:

entries: before=63 after=63 new_files=0 (22 entries written by 4.203.1)
CHANGED FIELDS across common entries: NONE — all hash fields identical

manifestHash / helpersHash / pluginsHash / disableSandboxHash are byte-identical before vs after the run, and no new cache filenames appeared.

2) xcrun stderr → empty

$ xcrun --sdk macosx --show-sdk-version 1>/tmp/sdk.out 2>/tmp/sdk.err
stdout: 26.2
stderr bytes: 0

Stable 26.2 across 3 consecutive invocations, so the stderr-pollution theory doesn’t hold on this machine (macOS 26.5.2 / Xcode 26.3).

3) New ProjectDescriptionHelpers entry per run → no

No new directory appeared, and the compiled libProjectDescriptionHelpers.dylib inside the existing entry has an mtime hours before both runs — so helpers were served from cache during this protocol. (The “helpers rebuilt every run” in my original report appears to have been contaminated by manifest edits and --verbose env changes between runs on my side — under this controlled protocol the helpers cache hits. Sorry for the misdirection.)

The part that looks like the actual bug

Even though every stored hash matches what the store path computes, the Manifests/1.* files were rewritten in place during the measured run (mtimes updated, content identical) and the run still took 149s recompiling every Project.swift. So the lookup path seems to fail before the hash comparison — a read/decode failure or a different key derivation on load vs store — rather than any hash being non-deterministic.

One more data point: a minimal project (helpers + Environment. access + 9-project workspace + local path package + empty Tuist/Package.swift) does not reproduce — warm generates are ~0.3s. The one thing I couldn’t add to the minimal repro is real resolved external dependencies (our project has 18 remote packages / 39 resolved), because a fresh tuist install on this machine fails with error: unknown option --cache-path; Did you mean --package-path? from the SwifterPM layer (happy to file that separately — warm ~/.cache/swifterpm masks it on the main repo).

Hi @esnunes Update: root cause found, and my original diagnosis in this issue is wrong.

The manifest cache is not broken. On 4.208.0, a warm tuist generate --no-open --verbose run does zero manifest dumps and zero ProjectDescriptionHelpers rebuilds — and still takes 159s. The regression is in resource-glob exclusion, and it is a complexity change, not a caching one.

(Same disclosure as the issue body: investigated with Claude Code under my supervision, everything measured on my machine, happy to re-run anything.)

Bisected boundary

175 releases between 4.157.4 and 4.208.0, binary search, 5 back-to-back tuist generate --no-open runs per candidate with nothing changed in between:

Last good: 4.169.0. First bad: 4.169.1. Still reproduces on the latest canary I had installed.

Where the time goes

Timestamping every verbose log line shows one single silent block; the second largest gap in the whole run is 0.2s:

Target FirebaseFirestoreTestingSupport ignored by product type
Minimum deployment targets supported by the selected SDKs: SDKDeploymentTargets(iOS: "12.0", ...)
>>> 127.0s with no output <<<
Transforming project <first project>: Deleting /Derived directory

All graph mappers together account for ~1.3s (StaticXCFrameworkModuleMapGraphMapper 0.571s, TargetsToCacheBinariesGraphMapper 0.528s, …), so the time is in package/target mapping.

I built 4.208.0 from source to get symbols. sample(1) over a 30s window inside that block, 20,843 samples, essentially one stack:

ResourceFileElement.from(manifest:generatorPaths:fileSystem:includeFiles:)   ResourceFileElement+ManifestMapper.swift:88
 └ globFiles #1 (_:excluding:)                                              ResourceFileElement+ManifestMapper.swift:44
    └ closure #1 in globFiles                                               ResourceFileElement+ManifestMapper.swift:45
       ├ AbsolutePath.contains(_:)                    10,632 samples (51%)
       │   └ UNIXPath.components.getter               Path.swift:623
       └ _ArrayBuffer._consumeAndCreateNew / Collection.split / Array.append

The change that did it

From 9303ffeb (“refactor(cli): replace FileHandler with FileSystem”, #10040), the only CLI commit in 4.169.1 that touches this path:

  files = try await fileSystem
      .throwingGlob(directory: .root, include: [String(path.pathString.dropFirst())])
      .collect()
-     .filter(includeFiles)
-     .filter { !excluded.contains($0) }
+     .filter { path in
+         !excluded.contains(where: { path.isDescendantOfOrEqual(to: $0) })
+     }
+     .concurrentFilter { try await includeFiles($0) }

excluded is a Set<AbsolutePath> that has already been expanded to concrete file paths a few lines above (excluded.formUnion(globs)). So the exclusion test went from one hash lookup per file to a linear scan of the whole set, and every comparison re-splits both paths into components via UNIXPath.components. That is O(files x excluded x depth) with an array allocation per comparison. #10957 (“avoid expanding recursive exclusion globs”) shrinks excluded, but the scan is still there, which matches this still being slow on 4.208.0 / 4.210 canary.

Fix and measurement

Walking the candidate’s parents against the set keeps the semantics and removes the scan:

-                        !excluded.contains(where: { path.isDescendantOfOrEqual(to: $0) })
+                        var candidate = path
+                        while true {
+                            if excluded.contains(candidate) { return false }
+                            if candidate.isRoot { return true }
+                            candidate = candidate.parentDirectory
+                        }

Measured with my own release build of 4.208.0, same machine, same workspace:

The generated project.pbxproj is byte-identical between the patched and unpatched self-built binaries (0 diff lines across all generated projects and workspaces), so this is a pure complexity fix. Happy to open a PR with this if you want it in that shape.

Project shape

114,366 files / 33,818 directories / 456 symlinks in the repo; workspace of 9 generated projects plus 39 resolved packages from Tuist/Package.swift (Tuist/.build is ~3.1 GB). Xcode 26.3, macOS 26.

Ruled out along the way

• Manifest / helpers cache — warm 4.208.0 does zero manifest dumps and zero helpers rebuilds, still 159s.
• URL.resourceValues(forKeys:) per entry in FileSystem’s Glob.directoryDestination — replacing it with lstat changed nothing (131ms vs 157ms globbing our source tree).
• FileSystem glob vs the old FileManager.enumerator traversal — 143ms vs 124ms on the same tree.
• Machine-level file-system slowness — raw lstat here is 2.5 µs, getattrlist 1.9 µs.