Why
A suite-granularity shard plan could silently drop test targets. A test plan with 70 targets produced a shard plan covering only 12 — and the count swung run to run (as low as 2, as high as 69), so most targets’ tests simply never ran.
Root cause
The plan is built from xcodebuild -enumerate-tests, which boots each test bundle to discover its tests. When a runner can’t launch, xcodebuild still lists the target as a target node with no test children, and records the failure in the enumeration’s top-level errors array:
"errors": [ "UITests01-Runner encountered an error. (Underlying Error: Early unexpected exit, operation never finished bootstrapping …)" ],
"values": [{ "kind": "plan", "children": [
{ "kind": "target", "name": "UITests01" }, // present, ZERO children
{ "kind": "target", "name": "AppTests", "children": [ // booted fine
{ "kind": "class", "name": "LoginTests", … } ] } ]}]
XCTestEnumerator decoded only values and ignored errors, so a boot-failed target was indistinguishable from a genuinely empty one — ShardPlanService classified it “empty” and excluded it. No throw fired (the target is present, not absent, so the prior absent-only recovery never ran). The run-to-run variance is just which runners fail to launch each time.
Reproduced locally. A 20-target iOS fixture (10 unit + 10 UI tests, one scheme) enumerated 12× back-to-back: all 20 targets present every run, but the number returned present-but-empty swung 0 → 15 of 20 on identical inputs — the same pattern at small scale. Findings:
- The trigger is the test runner failing to bootstrap under boot contention (
"…Early unexpected exit, operation never finished bootstrapping…"). It hits unit-test targets too, not just UI tests.
- On a real boot failure the
errors array is populated (errors count ≈ empty count every run) → the classification below is sound against real failures, not just the deterministic stand-in.
- Re-enumerating a dropped target in isolation (
-only-testing <target>) recovers it cleanly (suites found, 0 errors) — one-at-a-time booting removes the contention.
A secondary gap surfaced in the same investigation: top-level Swift Testing @Test functions (no enclosing @Suite) enumerate as kind == "test" directly under the target, which the parser (class-only) also missed.
Fix
XCTestEnumerator returns a new XCTestEnumeration { targets, errors } that surfaces xcodebuild’s errors, and now collects a target’s direct class and test children (top-level @Test functions), using the enumerated names verbatim — both are valid -only-testing identifiers (verified empirically; free functions need the trailing ()).ShardPlanService re-enumerates, in isolation, every expected module the bulk pass produced no suites for — present-but-empty as well as absent. The isolated pass arbitrates:
- suites found → recovered into the plan;
- present-but-empty with no errors → genuinely empty → excluded;
- otherwise (errors, or still missing) → failure → throws
modulesFailedToEnumerate (logging the underlying xcodebuild error) instead of silently shipping an incomplete plan.
The throw is the hard guarantee: a flaky run now either recovers a complete plan or fails loudly (CI retries), never silently skips the bulk of the tests.
Tests
Updated the existing suite-granularity tests for the new return type/semantics, plus new coverage:
- enumerator: captures top-level
@Test functions (and still ignores nested class-test methods); surfaces errors.
- planner: a module reported empty-with-errors is recovered when an isolated pass succeeds; a module empty-with-errors on every pass throws.
All scoped TuistUnitTests (XCTestEnumeratorTests, ShardPlanServiceTests, ShardServiceTests) pass locally.
Alternatives considered
- Constrain xcodebuild’s parallelism (
-parallel-testing-enabled NO): tested on the repro loop — did not help (drops stayed 0→15/20). The contention is at the simulator runner-boot level, not xcodebuild’s worker cloning, so that flag isn’t the lever.
- The only effective concurrency control is targets-per-invocation — i.e. chunking the bulk call via
-only-testing. This PR’s recovery is exactly that at the limit (K=1, proven 100% reliable in the repro), applied reactively to the targets that failed. A proactive small-batch enumeration (cap K per call from the start) is a reasonable alternative that may scale better than bulk-then-recover for very large plans, at the cost of being slower on healthy ones; easy to switch to if preferred. The throw bound keeps either approach safe.
- Static enumeration (no boot) is the durable fix: reading the test metadata from the built
.xctest binaries (Mach-O ObjC metadata for XCTest + Swift Testing’s test-content section; MachOKit is already a dependency) discovers tests with zero runner boots, eliminating the bootstrap flakiness entirely and making this recovery machinery a thin safety net. Out of scope here; worth a follow-up.
🤖 Generated with Claude Code
T