Registry sync clones and packages submodules that no target references, producing multi-hundred-MB archives and heavy /tmp pressure

Why is this needed?

Since #10256 (2026-04-13), a package whose tag carries a .gitmodules is routed away from GitHub’s zipball and into a full git clone with --init --recursive. The entire submodule tree is then zipped into the published archive. Nothing in that path asks whether any of that content is actually used, and nothing bounds how large the result may be.

Measured on a real clone of meta-llama/llama-stack-client-swift@0.0.58:

The submodule content is a transitive C++/Python dependency tree (llama-stack → executorch → 23 third-party submodules such as XNNPACK, abseil, eigen, sentencepiece). No target in Package.swift references any of it. The package’s targets resolve through the Sources//Tests/ convention, the generated openapi.yaml is checked in, and the submodule only feeds a regeneration script. SwiftPM would place that tree on disk during a source-control checkout but would never compile a byte of it.

Scoping note. All 8 currently published versions of this package were synced before the .gitmodules routing landed, so they are zipball archives of 34 KB–400 KB containing no submodule content at all, and SwiftPM has been resolving them successfully ever since. So this is not about archives already in the bucket. It is about what happens next time a package like this is synced:

• A new tag takes the clone path and would publish a multi-hundred-MB archive. (Today it instead fails permanently; #12232 fixes that, which is what makes this reachable.)
• A forced resync of a published version performs the full clone and zip, then hits ensure_checksum_change_allowed/5 and is discarded because the rebuilt checksum differs. The bandwidth and /tmp cost is paid in full; the upload is correctly refused.

Two costs follow:

1. Bucket and client cost for newly synced versions, downloaded in full by every resolving client.
2. Sync pod /tmp exhaustion, incurred even when the upload is refused, because the clone and zip happen first. /tmp is an emptyDir with sizeLimit: 16Gi in production and 4Gi in preview (infra/helm/tuist/values-preview.yaml), and swift_registry_release runs at concurrency 5 (server/config/runtime.exs). Peak per job here is ~1.3 GB of working tree plus the 546 MB archive ≈ 1.9 GB. Five concurrent jobs is ~9.5 GB — within production’s 16 Gi but with much less headroom than when such jobs aborted early. Preview’s 4 Gi is already tight for one and would be exceeded by two, evicting the pod.

The production values for this deployment carry the comment “If this OOMs again, lower queueConcurrency before raising memory”, so this pod has a history of resource pressure and the same knob governs /tmp.

Two pre-existing costs also become reachable at this scale: checksum_for_file/1 streams the archive in 2048-byte chunks (~280k reads for 546 MB), and inspect_archive/1 buffers the full unzip -Z listing (28k entries).

The decision can be made before cloning

A submodule only affects the build if its content is reachable from the resolved module graph — target paths, whether declared via path:/sources: or resolved through the Sources/<TargetName> convention. That is answerable from the manifest, and both inputs are already in hand at the right moment:

• parse_gitmodules_paths/1 already computes the submodule paths.
• fetch_manifests/3 already runs before fetch_source_archive/5 in the with chain in sync_release/7, so every Package.swift variant is already fetched when the clone-vs-zipball decision is made.

No new network calls are required. A prototype of the static check separates the two cases correctly:

meta-llama/llama-stack-client-swift@0.0.58
  submodules     : ['llama-stack']
  manifest paths : (none — convention-only: Sources/, Tests/)
  -> llama-stack: not referenced by any target path

SDWebImage/libwebp-Xcode@master
  submodules     : ['libwebp']
  manifest paths : ['include', 'libwebp/sharpyuv', 'libwebp/src']
  -> libwebp: REFERENCED by ['libwebp/sharpyuv', 'libwebp/src']

This is strictly better than deciding after the fact: for llama-stack-client-swift it removes the 1.3 GB download, the 546 MB archive, and the /tmp pressure entirely, and yields exactly the archive the registry has been serving successfully for months.

Steps to address the need

1. Add a manifest analysis that, given the already-fetched manifest contents and the already-parsed submodule paths, reports which submodules are referenced. Extract path-bearing arguments (path, sources, resources, exclude, publicHeadersPath) and paths embedded in settings (.headerSearchPath, .unsafeFlags(["-I", ...]), linkerSettings). Analyse every manifest variant, not just Package.swift. Treat anything not statically resolvable — computed paths, interpolation, loops — as referenced, so the analysis only ever errs toward cloning.
2. Gate the routing in fetch_source_archive/5 on the result: clone only when at least one submodule is referenced; otherwise take the zipball path. When a submodule is referenced, keep recursive initialization as today, since nested submodule needs cannot be resolved statically. Record the chosen strategy in metadata so the decision is auditable.
3. Keep a size ceiling as a backstop for the case where submodules genuinely are referenced and the tree is still enormous. Check after zip_directory/2 and before upload_source_archive/5, and record a verified skip (e.g. reason archive_too_large) so SyncWorker does not re-enqueue every 10-minute tick. Log package, version, and measured size so the ceiling can be tuned from real data. Falling back to the zipball is not appropriate here — by construction this branch is reached only when the submodule content is needed.
4. Tests in server/test/tuist/registry/swift/release_worker_test.exs: a package with an unreferenced submodule taking the zipball path and performing no clone; a package with a path:-referenced submodule still cloning; a manifest with a non-literal path being treated conservatively; and the size backstop recording a skip that survives a later sync tick.
5. Consider a pre-clone guard for the referenced-submodule case as a follow-up, since step 3 only triggers after the bytes are on disk — for example aborting when the working tree exceeds the limit mid-clone.
6. Independently, raise tmpSizeLimit for preview or drop its swift_registry_release concurrency to 1, so one large package cannot evict the preview sync pod.

Risks

• Manifests are arbitrary Swift, so text extraction cannot be complete. The conservative default (unparseable ⇒ referenced ⇒ clone) keeps this safe but means some large trees are still fetched.
• Build tool plugins can read any file inside the package directory, so a plugin could consume submodule content that no path argument names. Packages declaring plugins may warrant always cloning.
• Whether test-target paths are validated for non-root packages was not established; treating a test-only reference as “referenced” sidesteps the question.
• Dropping unreferenced submodules is a deliberate divergence from a source-control checkout, which would have those files on disk. It is not a build divergence — see the parity note in the comments, including the fact that this package is currently unresolvable from source control at all, while the registry serves it successfully.

The failure mode of getting this wrong is loud rather than silent: PackageBuilder throws invalidCustomPath (“invalid custom path ‘X’ for target ‘Y’”) for a missing declared path, or unknownTargets for a missing convention directory. That is still a broken consumer build, which is why the analysis must stay conservative.

Context

• Follow-up to #12232, which surfaced this by making these releases succeed instead of failing permanently.
• All sizes and behaviours above are reproducible from public repositories, the public registry endpoints, SwiftPM’s GitRepository.swift / PackageBuilder.swift, and the chart values in this repo.