Acceptance test runners are oversubscribed: 78 concurrent swift-frontend processes on a 6-core virtual machine

Why is this needed?

The acceptance test runner is oversubscribed by more than tenfold, so an acceptance test’s wall clock is mostly a function of what happens to be scheduled next to it rather than of the work it does. That already produces recurring red builds, and it is getting worse over time.

The evidence is a spindump that Xcode captured automatically when BuildAcceptanceTestSwiftPMPrebuiltMacro blew its four minute time limit in run 79b1dfbb, snapshotted at 2026-08-20 12:56:14 UTC.

The machine:

Hardware model:   VirtualMac2,1
Active cpus:      6
Memory size:      14 GB
Total CPU Time:   27.813s      (of a possible 30s across 6 cores in the 5s sample = 93% saturated)

What was running on those six cores at that moment: 1,199 processes, of which 78 concurrent swift-frontend, plus 51 git, 324 bash, 21 SWBBuildService, 9 swift-package and 8 ld.

How the fan-out happens

Four layers of parallelism stack, and only the outermost one is bounded:

2 shards                      --shard-total 2, the only bounded layer
  x 3+ test modules           parallelization: .enabled  (Project.swift:111)
    x N test cases            Swift Testing default, in process
      x nested xcodebuild     every acceptance test shells out to a real build
= 78 swift-frontend on 6 cores

Modules run in parallel. Project.swift:111 declares every acceptance target .testableTarget(target: .target($0.name), parallelization: .enabled). In the dump, three xctest workers all forked in the same second and were still alive 243 seconds later:

xcodebuild[48830]   forked 248s ago     the single `tuist test`
  xctest[50556]     forked 243s ago     TuistKitAcceptanceTests
  xctest[50557]     forked 243s ago     TuistAutomationAcceptanceTests   (the one that timed out)
  xctest[50558]     forked 243s ago     TuistCacheEEAcceptanceTests

Only three of the five acceptance targets had live workers at that instant, so three concurrent modules is what is proven, not necessarily the ceiling.

Cases run in parallel inside each module. xctest[50558] had three nested xcodebuild processes alive at once, forked 76s, 30s and 16s before the dump. That is three of its own test cases mid-build simultaneously.

Each case spawns a real build. An acceptance test is not a unit test; it shells out to xcodebuild, which starts a build service, which fans out to compilers. One test can be worth thirty-odd swift-frontend processes on its own.

Split by worker at the moment of the timeout:

The worker that failed held zero compiler processes. It had launched its build and was waiting on it while 53 compilers belonging to the other two workers consumed the machine. Its roughly 16 seconds of real work stretched past 240 seconds. Measured on an unloaded machine, that test is tuist install 13.2s, tuist generate 2.6s, xcodebuild build 6.8s.

Serializing a suite does not do what it looks like it does

This repo has already tried to mitigate this once, and the mitigation has never taken effect. GenerateAcceptanceTestiOSAppWithModuleMapPackages carries:

/// This integration test resolves a large external package graph and runs two Xcode builds. Serializing it prevents
/// the intermittent resource contention observed when it runs alongside the rest of the acceptance-test shard.
@Suite(.serialized)

.serialized serializes tests within a suite. It does not stop the suite running alongside everything else. That suite contains exactly one test, so the trait is a no-op, and one of the heavier contention sources in the shard is still running concurrently with everything.

Verified with a standalone Swift Testing package:

WINDOW SERIALIZED_SINGLE  0.00 -> 3.12     serialized suite, 1 test
WINDOW SIBLING_1          0.00 -> 3.12     ordinary sibling suite
WINDOW SIBLING_2         -0.00 -> 3.12
WINDOW SIBLING_3         -0.00 -> 3.12
WINDOW SER_MULTI_1        0.00 -> 2.12     serialized suite, 3 tests
WINDOW SER_MULTI_2        2.12 -> 4.12
WINDOW SER_MULTI_3        4.12 -> 6.12

The three tests inside a serialized suite run strictly back to back, but the serialized suite overlaps all four of its siblings. Anyone reaching for .serialized to fix contention needs to know this, because it is the obvious-looking fix and it silently does nothing across suites.

By contrast, the .serialized on DependenciesAcceptanceTests is correct and should stay. That suite has five tests that share SwiftPM’s global registry configuration, and the trait genuinely stops them overlapping each other.

The trend

Over nine days the suite became more concurrent, so runs finish sooner while each individual test waits longer:

Run wall clock is down 43% while per test wall clock is up around 38%. In that Aug 20 run, 20 tests took 180s or more and 6 took 240s or more.

Worth ruling out explicitly, because it is the obvious first guess: this is not swift-syntax being compiled from source. The string swift-syntax does not appear anywhere in the spindump, and the timed out worker held no compiler processes at all. The prebuilt macro path works and package resolution is not the bottleneck.

Steps to address the need

The goal is to stop running dramatically more concurrent compiler processes than the machine has cores. These are largely independent and can be combined.

1. Measure first. Add a step that records core count and load during an acceptance run so the effect of any change below is visible rather than guessed. A sysctl -n hw.ncpu plus periodic uptime in the job would make this observable on every run instead of only when a spindump happens to fire.

2. Cap the nested builds. This is the largest multiplier and the cheapest to bound, since it costs no extra machines. Each acceptance test shells out to xcodebuild and each of those independently decides how many compiler jobs to run, assuming it owns the machine. Passing a job limit (for example -jobs) to the xcodebuild invocations that acceptance tests spawn would bound the per test fan-out.

3. Bound the remaining parallelism knobs. Partially done: #12511 adds -maximum-parallel-testing-workers 2 to the sharded acceptance job, which caps how many test runner processes spawn and therefore how many modules run side by side. That leaves two knobs untouched. parallelization: .enabled at Project.swift:111 still declares every target parallelizable, and Swift Testing’s in process default still runs cases concurrently inside each runner. xctest[50558] above was running three concurrent nested builds on its own, so a worker cap alone does not bound the fan-out. The value of 2 is a starting point and should be tuned once step 1 makes load observable.

4. Fix the no-op serialization. GenerateAcceptanceTestiOSAppWithModuleMapPackages is annotated @Suite(.serialized) for a reason that the trait does not deliver. Either remove the annotation and the misleading comment, or achieve the intent some other way. Leaving it as is means a future reader believes a heavy suite is isolated when it is not.

5. Reconsider the shard and machine shape. The suite runs --shard-total 2 across 139 suites, so each machine takes roughly half. More shards would help proportionally, but it is a linear lever against a multiplicative problem: getting 78 compilers down to something six cores can run would take somewhere around 13 shards, meaning 13 macOS virtual machines per pull request. Larger machines than the current 6 core and 14 GB virtual machine may be the better trade.

6. The fork path has no sharding at all. .github/workflows/cli.yml:393 runs tuist test --platform macOS TuistAcceptanceTests with no shard flags, so external contributor pull requests run all 139 suites on a single six core virtual machine. That is strictly worse than the sharded path described above.

7. While in here, check the Xcode version. The runner executed /Applications/Xcode_26.6.app/... in this run even though .xcode-version pins 26.5. That may be intentional, but if the pin is meant to be authoritative then it is not being honoured, and acceptance tests are running on a different toolchain than the one we think we test against.

Related: #12511 removes the wall clock ceilings that this oversubscription was tripping. That change stops the false failures but does not reduce the oversubscription, which is why this is filed separately.

No GitHub comments yet.