Problem
Suite-granularity shard plans were non-deterministically missing tests. On the same commit, enumeration discovered 44 of 69 targets one run and 12 the next (other bad runs landed at 1, 3, 5, 6, 10, 11, 12). Targets that weren’t enumerated were never added to the shard plan, so their tests silently never ran.
Root cause
XCTestEnumerator ran a single xcodebuild -enumerate-tests pass and parsed the human-readable text from streamed stdout. For a large test bundle that dump is huge, and a streamed pipe can truncate/interleave — dropping whole Target/Class entries. So the flakiness was mostly in how we read the output, not in xcodebuild computing a different set each time.
Changes
XCTestEnumerator now requests machine-parseable JSON written to a file (-test-enumeration-format json -test-enumeration-style hierarchical -test-enumeration-output-path) and parses that — no more stdout scraping. A missing/malformed output file now fails loudly instead of returning a silently-incomplete set.ShardPlanService anchors module completeness on the deterministic .xctestrun universe: bulk enumeration → per-target re-enumeration for any module the bulk pass dropped → a whole-module backstop for modules that still won’t enumerate. No module is ever silently excluded.ShardService translates the whole-module backstop sentinel back to a bare -only-testing <Module> on the runner.
Notes
- Swift Testing: verified suites enumerate by type name (e.g.
@Suite("display name") struct LoginSuite → …/LoginSuite/…), so they shard with valid -only-testing identifiers exactly like XCTest classes. The whole-module backstop should therefore rarely trigger.
- Not in this PR: suite-level historical duration matching (a separate balance-only fix) is intentionally left for its own PR.
Testing
XCTestEnumeratorTests, ShardPlanServiceTests, ShardServiceTests — green locally (tuist test TuistUnitTests).- Validated the JSON-to-file invocation against a probe package (Swift Testing type-name behavior; byte-stable output across repeated runs).
- Remaining external check: real-scale variance on a large iOS app (the probe is too small to induce stdout truncation) — confirm a suite-granularity plan covers all modules/suites stably across repeated runs before un-drafting.
🤖 Generated with Claude Code
T