Acceptance test runners are oversubscribed: 78 concurrent swift-frontend processes on a 6-core virtual machine

Why is this needed?

The acceptance test runner is oversubscribed by more than tenfold, which makes every acceptance test’s wall clock a function of what happens to be scheduled next to it rather than of the work it does. That is already producing recurring red builds, and it is getting worse over time.

The evidence comes from a spindump that Xcode captured automatically when BuildAcceptanceTestSwiftPMPrebuiltMacro blew its four minute time limit in run 79b1dfbb, snapshotted at 2026-08-20 12:56:14 UTC.

The machine:

Hardware model:   VirtualMac2,1
Active cpus:      6
Memory size:      14 GB
Total CPU Time:   27.813s      (of a possible 30s across 6 cores in the 5s sample = 93% saturated)

What was running on those 6 cores at that moment:

• 1,265 processes total
• 78 concurrent swift-frontend processes
• 51 git, 324 bash, 21 SWBBuildService, 9 swift-package, 8 ld

The fan-out that produces this is: one tuist test invocation starts three parallel xctest workers, each worker runs Swift Testing tests in parallel, and each of those tests shells out to its own nested xcodebuild, which starts its own build service and its own set of compiler processes. Confirmed by ancestry in the dump:

swift-frontend <- SWBBuildService <- xcodebuild <- xctest[50558] <- xcodebuild[48830] <- tuist[44592]

Split by worker, at the moment the failing test timed out:

The worker that failed was not doing heavy work. It had spawned its xcodebuild and was waiting on it, holding zero compiler processes of its own, while 53 compilers belonging to the other two workers consumed the machine. Its 16 seconds of actual work stretched past 240 seconds.

The effect is visible in aggregate too. Over nine days the suite became more concurrent, so the run finishes sooner while each individual test waits longer:

Run wall clock is down 43% while per test wall clock is up around 38%. In that Aug 20 run, 20 tests took 180s or more and 6 took 240s or more. That is fine for tests with no ceiling and fatal for any test that has one, which is what #12511 is cleaning up. But removing ceilings only hides the symptom: the suite is still running roughly 78 compiler processes on 6 cores, which wastes wall clock on context switching and memory pressure rather than getting work done.

For calibration, the same test’s real work measured locally on an unloaded machine is about 16 seconds: tuist install 13.2s, tuist generate 2.6s, xcodebuild build 6.8s.

Worth ruling out explicitly, because it is the obvious first guess: this is not swift-syntax being compiled from source. The string swift-syntax does not appear anywhere in the spindump, and the timed out worker held no compiler processes at all. The prebuilt macro path is working and Swift Package Manager resolution is not the bottleneck.

Steps to address the need

The goal is to stop running dramatically more concurrent compiler processes than the machine has cores. Any one of these would help, and they can be combined:

1. Measure first. Add a step that records core count and load during an acceptance run, so the effect of any change below is visible rather than guessed. The spindump above is a one off; a cheap sysctl -n hw.ncpu plus periodic uptime in the job would make this observable on every run.

2. Cap the nested builds. Each acceptance test shells out to xcodebuild, and each of those independently decides how many compiler jobs to run. Passing a job limit (for example -jobs) to the xcodebuild invocations that acceptance tests spawn would bound the per test fan-out instead of letting each build assume it owns the machine.

3. Cap test parallelism. The suite currently runs three xctest workers per machine, each running Swift Testing tests in parallel inside it. Reducing worker count, or bounding in process parallelism, trades a little throughput for far less contention.

4. Reconsider the shard and machine shape. The suite runs with --shard-total 2 across 139 suites, so each machine takes roughly half. Either more shards across more machines, or larger machines than the current 6 core and 14 GB virtual machine, would give the existing parallelism room to actually run.

5. While in here, check the Xcode version. The runner executed /Applications/Xcode_26.6.app/... in this run even though .xcode-version pins 26.5. That may be intentional, but if the pin is meant to be authoritative then it is not being honoured, and acceptance tests are running on a different toolchain than the one we think we test against.

Related: #12511 removes the wall clock ceilings that this oversubscription was tripping. That change stops the false failures but does not reduce the oversubscription, which is why this is filed separately.

No GitHub comments yet.