feat(cli,server): per-shard test-product downloads + resilient shard download

What

Fixes the selective-testing shard-download timeout, makes each shard download only the products it needs, and hardens the upload path so very large bundles (hundreds of modules, multi-GB) upload reliably. Five changes:

1. Resilient download. The shard-bundle download in ShardService now retries (3× via RetryProvider), like the upload path already did. No timeout-value change — the global 90s timeoutIntervalForResource stays; retry is what rides out the transient slowdown that caused the failure.
2. Per-shard downloads. Products are split into one shared artifact (frameworks, dylibs, xctestrun — everything except .xctest) plus one artifact per module’s .xctest. get_shard returns download_urls (shared + the shard’s modules); ShardService downloads only those and merges them. Backward compatible: with no split artifacts it falls back to the legacy single bundle / download_url.
3. Module-aware suite bin-packing. Suite-granularity packing now groups a module’s suites onto one shard so #2 actually saves data — but only when it doesn’t hurt balance (oversized modules still split; a grouped packing >5% slower than optimal falls back to plain LPT).
4. Bounded upload memory. MultipartUploadArtifactService now keeps at most maxConcurrentParts (10) parts in flight, applying backpressure on the file read loop, so peak memory per upload is bounded by maxConcurrentParts * partSize regardless of artifact size. Previously the loop read every 10 MB part into its own buffer ahead of the uploads, so memory grew with the artifact and OOMed on multi-GB uploads. That is what caused a large build job’s upload to fail while a smaller one succeeded. Splitting the bundle (#2) already shrinks the largest object down to the shared artifact; this bounds the upload itself so it no longer scales with bundle size.
5. In-place per-module archiving. Each module’s .xctest is archived directly from the products directory via AppleArchiver.compress(subdirectory:relativeTo:to:), preserving its full path relative to the products root, instead of being copied into a staging directory first. A project with hundreds of modules would otherwise copy every multi-hundred-MB test bundle to a temp dir before compressing it, i.e. tens of GB of redundant disk I/O per build job. The archiver walks only the target subtree plus the ancestor directories leading to it, so sibling bundles in the same products directory are never read.

Why

A CI run failed downloading a 316 MB shard bundle with The request timed out, at exactly duration: 90230 ms — the wall-clock timeoutIntervalForResource = 90 cap firing on a transfer that was still progressing.

Measuring the same object from three vantage points (host SSH + a probe workflow inside a tuist-macos tart VM):

The 3.5 MB/s was not reproducible anywhere (every endpoint serves a single stream at 38–112 MB/s), so the timeout was a transient (most likely host NIC contention), and the 90s cap turned it into a hard failure → retry is the fix. Two alternatives were measured and rejected: parallel download is slower on the t3 endpoint CI uses (tail latency across Tigris backends), and endpoint pinning gives no single-stream speedup.

Per-shard downloads + the measured payoff

The bundle was one per-plan object every shard downloaded in full. Binaries are per-module (.xctest); everything else is shared.

The split is done client-side, once: the build job (tuist test --build-only) that has the products uploads N+1 objects (shared.aar + one modules/{name}.aar per module), each written exactly once. get_shard then hands each shard presigned URLs for shared + only its modules; shards only download, never upload. It’s per-module objects, not per-shard bundles, so a module several shards need (e.g. Generator after packing) is stored once and downloaded by each — no upload or storage duplication (storage stays ≈ the old single bundle). Splitting into shared + per-module artifacts (lzfse), measured on the real bundle:

• shared.aar: 175 MB (paid by every shard); per-module .xctest: ~24–35 MB each.

A shard downloads shared + only its modules — but under suite granularity the duration-only LPT bin-packer scattered every module across every shard (verified: both shards of the real plan touched all 5 modules), so the split alone saved nothing there. The module-aware packer fixes that. Simulated on the real plan (skewed module durations):

Zero balance impact — the oversized module splits (keeping balance) while small modules cluster; if modules were evenly sized (where affinity would unbalance shards), the tolerance gate falls back to plain LPT.

Validation

• Builds locally (TuistServer + TuistKit) via swift build --replace-scm-with-registry --force-resolved-versions.
• shards_test.exs 20/20; bin_packer_test.exs + shards 42/42.
• E2E archive round-trip with the real Apple Archive format: a single-module shard’s merged dir is diff -r-identical to the original minus the other modules; a full-shard merge is byte-identical to the original; framework symlinks preserved; no errors merging overlapping dirs.
• The shared artifact’s exclude pattern is .xctest/ (not .xctest): excludePatterns is a substring match, so .xctest would also drop the sibling .xctestrun that ShardService needs — covered by an AppleArchiver regression test.
• Throughput/size numbers measured directly against the real 316 MB bundle; module-spread/balance measured against the real plan’s suite durations from ClickHouse.

Notes

The OpenAPI CLI client (Types.swift) is hand-applied surgically, not via full mise run generate-api-cli-code: a full regen reorders the spec non-deterministically and breaks unrelated call sites (e.g. CreateCommandEventService: git_branch must precede id).

Follow-ups

Two further reductions were considered and deferred, both good candidates for their own change:

• Strip test-bundle binaries before archiving. An earlier strip -S (debug symbols) step was dropped: only ~6% on the bundle it was measured against, not worth the added surface (Mach-O detection, a strip pass, re-signing). On statically-linked test bundles the real lever is strip -x (local symbols), because the binary’s __LINKEDIT symbol table dominates its size. Measured on a real large test binary: ~40% smaller uncompressed / ~23% compressed. It can return behind a flag (with an ad-hoc re-sign, since stripping invalidates the code signature), and is the biggest remaining size lever, but carries enough correctness surface to warrant its own PR and validation.
• Single PUT for small artifacts. Each per-module object (tens of MB) currently runs a full multipart start / generate-url / complete round-trip; a plain presigned PUT below a size threshold would cut server round-trips across the many small module artifacts. The latency impact is small today, so this is low priority.

🤖 Generated with Claude Code