Stress-testing newly added tests before they merge

Summary

This RFC lands a stress gate: on a merge request, the tests that branch adds are rerun several times each, and the job fails if any of them disagrees with itself. Tuist detects flakiness today only after the fact, from aggregates over the default branch, which means the first person to pay for a flaky test is someone who did not write it, days later, in an unrelated pull request. The gate moves that detection to the one moment where the fix is cheap, while the author still holds the context that produced the test.

On the passive half of flaky-test management Tuist is already at parity with the market: detection from run history, muted and skipped quarantine states, monitors that fire on a test’s first run. The gap is the active half, and only one product ships it. This RFC closes that gap, and closes it with the piece that product deliberately leaves out, which is the block itself.

The part that needs an RFC rather than a pull request is where the seam sits, because this has to work for Xcode, for Gradle, and eventually for Bazel, and the naive version is written three times. The decision is a split: the server owns the newness verdict, the client owns the loop. “Which of these test cases has this project never run on its default branch” is a set difference over an identity the tests tables already share across build systems. “Run this exact set N times and decide the exit code” is -only-testing plus -test-iterations, includeTestsMatching plus forced re-execution, or --runs_per_test. Only the second half is per-build-system, and it is the half that is a few lines.

That split is not new. It is exactly the shape of quarantine and sharding, both of which already ship on both clients.

Motivation

A customer with a very large iOS suite already does this by hand: CI reruns every newly added test some number of times and blocks the merge request if any of those runs disagree. They want Tuist to own it, and the instinct is right, because the expensive half of the problem is the half they have to approximate and we do not.

Anyone can write the loop. What a homegrown version cannot do well is decide what “new” means. It has to infer it from the diff, and a diff does not know about XCTest cases inherited from a base class, Swift Testing’s @Test display names, per-argument identity on parameterised cases, or annotation-driven discovery on the JVM. Every one of those is a place where the homegrown gate quietly stresses nothing and reports green. Tuist is the only thing in the pipeline holding the default branch’s test history, which is the only source that answers the question directly rather than by approximation.

The retrospective path is not a substitute. check_new_test_cases/3 and the first_run events it feeds already tell us a test is new, and the flakiness monitors already tell us a test is unreliable, but the two facts arrive in that order, separated by however long it takes the test to misbehave on the default branch. By then the merge request is closed, the author has moved on, and the cost has been socialised across everyone whose pull request the test now fails.

The approach is also not speculative. Datadog reports that running new tests repeatedly identifies up to 75% of flaky tests before they merge. That is their number for their population, not ours, but it is the right order of magnitude to justify building the thing rather than arguing about whether repetition finds flakes.

Current state

• is_new is decided at ingestion. check_new_test_cases/3 reads the set of test case ids with CI runs on the project’s default branch over a trailing 90-day window (test_case_branch_presence) and marks everything outside it new. filter_first_run_test_case_runs/2 then isolates those runs to emit first_run events and webhooks. All of it happens after the tests have run, because the row it decorates is the one being written.
• With no default branch, nothing is new. check_new_test_cases/3 returns early and stamps is_new: false on every case. That is deliberate, not a gap: with no trusted branch there is no meaning to “already landed”.
• Quarantine is already a pre-run server fetch on both clients. The CLI asks the server for the muted and skipped identifier lists before it runs (fetchQuarantinedTests), excludes the skipped ones, and masks the muted ones afterwards (TestQuarantineService.markQuarantinedTests turns a failed summary back into a passed one when only quarantined tests failed). TuistTestQuarantine.kt does the same against the same endpoint, excluding skipped tests with Test.filter.excludeTestsMatching.
• Sharding is already a server-issued plan executed by two clients. CreateShardPlanService and ShardPlanService on the CLI, ShardsApi and TuistTestSharding.kt on Gradle, both applying the returned identifiers through -only-testing and includeTestsMatching. only_test_identifiers and skip_test_identifiers already ride on the test record.
• That plan is built from history, and new tests fall through it. CreateShardPlanParams documents that the suite inventory is read from the branch’s history, falling back to the default branch, and that at suite granularity the final shard is the catch-all. A suite the server has never seen has no estimate and lands in the catch-all by construction.
• The client already holds the per-case result set locally, with durations. The CLI parses the result bundle itself to build TestSummary before any upload (testSummary(resultBundlePath:)), which is how quarantine masking works without a server round trip. The Gradle plugin has the same set from its test listeners.
• Repetitions already ingest end to end. test_case_run_repetitions exists, the upload payload carries repetitions per case and per argument, and xcodebuild produces them via -test-iterations, -retry-tests-on-failure, -run-tests-until-failure and -test-repetition-relaunch-enabled. The CLI wires only the retry-on-failure shape, through the deprecated --retryCount.
• The only VCS API client is GitHub’s. create_check_run and update_check_run live in Tuist.GitHub.Client; Tuist.VCS.create_comment/1 gates on a refs/pull/ ref plus a GitHub App installation. Tuist.VCS knows GitLab only well enough to build a pipeline URL from a run id.
• Automated quarantine already refuses to touch never-validated tests. test_case_ids_with_successful_default_branch_run/3 exists so a case with no successful default-branch run stays ineligible, and its own documentation names the case it is protecting against: a brand-new test that has only ever run on a pull-request branch.

Prior art

The passive half is table stakes, and we are already at parity

Buildkite Test Engine detects flakiness by observing the same test producing different results on the same commit SHA, quarantines into muted (runs, failures soft-fail) and skipped (does not run), and offers workflow monitors including one that fires on a test’s first execution. That is, feature for feature, what Tuist already has: cross-run detection, the same two quarantine states, and first_run events feeding Tuist.Automations.Monitors.FlakyTestsMonitor. The resemblance is not a coincidence and it is worth stating plainly, because it means nothing in this RFC is about catching up.

Trunk Flaky Tests is explicitly passive: detection runs when uploads are processed, it does not rerun anything deliberately, and its New Test monitor (off by default) labels a test seen for the first time over a configurable grace period without classifying it. Develocity detects within a build (a test that fails then passes under the Test Retry plugin is classified flaky) and across builds sharing an input fingerprint. Both are retrospective, and both take the retry-on-failure shape rather than unconditional repetition.

None of the three reruns a newly added test on purpose, and none of them blocks a merge on the result.

The active half exists once, in Datadog Early Flake Detection

Datadog is the only product shipping the mechanism this RFC proposes, and its parameters are worth quoting exactly because they are a production-validated starting point rather than a guess. From the ddtrace sources (ddtrace/internal/ci_visibility/_api_client.py, ddtrace/testing/internal/retry_handlers.py):

• The library fetches a known tests list before the session; anything not in it is new.
• New tests are retried on a curve keyed on the duration of their first attempt: 5s: 10, 10s: 5, 30s: 3, 5m: 2 retries, and tests slower than five minutes are excluded entirely.
• A faulty_session_threshold (default 30, a percentage) disables the whole mechanism for a session when the share of tests reading as new exceeds it.
• Results are tagged (@test.is_new, @test.is_retry, @test.early_flake.enabled) and surfaced in their explorer.

Three things follow for this RFC. The duration-keyed retry curve is better than a flat N and is adopted below, because a suite’s flake risk and a suite’s wall-clock budget do not scale together. The faulty-session threshold is the right shape for the degenerate case and is adopted alongside an absolute cap, which it does not replace. And Datadog deliberately does not block on the result: Early Flake Detection tags, and blocking is a separate optional Quality Gates product. That is the one decision this RFC reverses, and the reason is in the failure-signal section: they can decouple detection from blocking because they sell the gate separately, and the customer asking us for this asked for the block.

Proposal

The loop belongs to the client, the verdict belongs to the server

Two concerns are entangled in “rerun the new tests”, and only one of them differs per build system. The newness verdict is a set difference over (name, suite_name, module_name), which is already neutral across Xcode and Gradle in the tests tables and would be neutral across Bazel the day Bazel ingests. The stress loop is a filter plus a repetition primitive, and it is genuinely different everywhere. Putting the verdict in the server means every new build system inherits it at no cost; putting the loop in the client means every new build system pays only for the part that actually differs, which is a filter translation.

The seam already carries this shape twice, which is the main reason to trust it. Quarantine is a server-computed set of identifiers, fetched before the run, applied by two independent clients. Sharding is a server-issued plan of identifiers, fetched before the run, applied by two independent clients. The stress gate is a third instance of the same contract, and the day Bazel joins, it joins all three at once.

A server-issued plan of what to stress cannot work, and the reason is structural rather than incidental. The shard plan derives its inventory from the branch’s history, and history is precisely what a new test lacks. The server can enumerate what it has seen; it cannot enumerate what it has not. So the candidate set has to originate on the client, and the server’s contribution is subtraction. Pointed that way, the same lookup that already backs check_new_test_cases/3 answers the question, and it answers it for every build system that writes into those tables.

Reusing the build system’s own retry mechanism as the gate is the trap worth naming, and it is the trap the whole rest of the market is in. Xcode’s -retry-tests-on-failure, Bazel’s --flaky_test_attempts, and the Test Retry Gradle plugin all rerun only what already failed. That excludes the entire population this gate exists to examine, which is tests that passed on their first attempt and would have failed on their fourth, and all three exist to hide flakiness in CI rather than surface it. The primitive the gate wants is unconditional repetition (-test-iterations, --runs_per_test), with the decision about what to repeat kept outside it.

The candidate set comes from the run that just happened, not from the diff

Three options were on the table: derive the candidate set from the branch diff, enumerate test cases before the run and ask the server, or run once and stress what that run revealed. Datadog’s known-tests-list model is a fourth and is treated in Alternatives, because it needs something neither of our clients has on the Xcode side.

Diff-based detection is rejected because it fails silently. Finding added tests in a diff means parsing test declarations out of source, per language and per framework: inherited XCTest cases that appear in no hunk, @Test display names that are not the function name, parameterised cases whose identity the source does not spell out, annotation-driven discovery on the JVM, a build-file layer on top for Bazel. Each of those produces an empty candidate set and a green gate, and a gate that can be silently empty is worse than no gate, because it is trusted.

Pre-run enumeration is rejected for v1 because neither client can do it yet. The CLI enumerates test modules from the .xctestproducts bundle for sharding, not individual cases; reaching case granularity needs an extra xcodebuild ... -enumerate-tests invocation after the build. Gradle has no comparable primitive at all. That is per-build-system work spent on both clients before the feature has earned it, which is the exact cost this RFC is trying to avoid paying three times.

The two-pass run is the choice, and its first pass is not a cost. CI runs the suite anyway. What falls out of that run, for free and at exact case granularity, is the set of cases that actually executed, already in the client’s memory because it parses the result bundle locally. The client sends those identifiers to the server, gets back the subset with no default-branch history, and reruns exactly that subset.

The first pass turns out to give two things rather than one, and the second is what makes the retry curve possible. Datadog keys its retry count on the duration of a test’s first attempt, measured rather than predicted. In a two-pass design that measurement is already in hand before the stress pass is planned, for every candidate, at no extra cost. A single-pass design has to decide the retry count before it knows how long the test takes. The candidate set is never approximated, it is observed, and so is its price.

The gate must not read its verdict back out of ingestion. is_new is stamped during ingestion, and on the result-bundle path ingestion is an asynchronous job the CLI enqueues by uploading. Waiting for that stamp would make a merge-request outcome depend on the processing queue draining, and that queue has wedged in production before. The verdict query answers from the same branch-presence data at request time. The runs still ingest and still get their is_new stamp; the gate simply does not wait for it.

“New” inherits its definition verbatim from check_new_test_cases/3, which is no CI run on the default branch in the trailing 90 days. Two consequences follow, and both err toward stressing something that is not new rather than skipping something that is: a test that last ran on the default branch more than 90 days ago reads as new, and a test that has only ever run locally reads as new. Diverging from that definition would be worse than either, because the gate and the dashboard’s new-test badge would then disagree about the same test in front of the same person.

The retry count is keyed on the duration the first pass already measured

A flat N is the wrong shape, because the value of another repetition is constant while its cost is not. Ten repetitions of a 200ms unit test is two seconds and worth having. Ten repetitions of a four-minute UI test is forty minutes and is how the gate gets switched off. The curve, not the number, is the decision.

Starting values, adopted from the one shipped implementation rather than invented: a candidate whose first-pass duration is at or under 5s gets 10 repetitions, under 10s gets 5, under 30s gets 3, under 5 minutes gets 2, and anything slower is excluded from the stress pass entirely and reported as excluded. These are configuration, tuned from the observed catch rate rather than re-litigated here. Ten repetitions catch a test that fails one run in ten roughly two thirds of the time in a single merge-request run, and because the gate runs on every push to the branch the probability compounds across a review cycle instead of being a single draw; two repetitions of a slow test is a much weaker filter, and that is the honest trade for not spending forty minutes.

All repetitions run rather than stopping at the first failure. Stopping early (-run-tests-until-failure) reaches the same verdict for less compute, but “failed 3 of 10” and “failed 1 of 10” are different conversations for the author, and on a bounded set the remaining repetitions are what make the report worth reading rather than merely obeying.

Each repetition gets a fresh process (starting value: relaunch enabled). The question is “will this flake in CI”, and in CI every run is a fresh process. Same-process repetition is cheaper and catches a real but narrower class, state leaking between repetitions of one test; it also lets a test pass ten times in one process and still flake once per CI run, which is exactly the false negative that would discredit the gate the first time it happened.

A stress pass never rebuilds. It reuses what the first pass built, which on Xcode is a test-without-building invocation against the same products and on Gradle is a re-execution of the same test task. That is what keeps repetition cheap: it is N test executions, not N builds.

The gate fails open, never expensive

Every degenerate case resolves to “run no stress pass and say so”, never to “stress everything”. The failure mode to design against is not a missed flake; it is a gate that turns a merge request into an unbounded compute bill on the day it is switched on.

• No default branch configured: the verdict is the one check_new_test_cases/3 already gives, which is that nothing is new. The gate reports that it found no default branch and exits zero, once, visibly.
• First run for a project, or a default branch with no CI history: every test reads as new. The primary guard is a faulty-session threshold (starting value: 30% of the cases that ran, matching the shipped default), above which the stress pass is skipped entirely and reported as skipped. A session where a third of the suite reads as new is not a session with a lot of new tests, it is a session whose history is wrong, and stressing on a wrong premise is worse than not stressing.
• A large suite with a genuinely large batch of new tests: this is the case a percentage alone does not catch, because 5% of fifty thousand tests is still two and a half thousand candidates. A candidate cap (starting value: 200 cases) bounds it, above which the gate stresses nothing and reports the count. The two guards do different jobs and neither replaces the other: the percentage catches a broken premise, the cap catches an expensive truth.
• A branch that adds nothing: no candidates, no second pass, no added wall clock. This is the common case, and it must cost exactly one request against a set the client already holds, which it does.

The failure signal is the exit code, because that is the version that ships

A non-zero exit code from the CLI is the v1 signal, and it needs nothing that does not already exist. The CLI already fails the run when tests fail, every CI provider blocks a merge on a failed job, and there is no VCS integration, app installation, webhook, or provider-specific code path anywhere on the path.

A check run is the version that ships to nobody. The only VCS API client the server has is GitHub’s, create_check_run is a GitHub check run, and the pull-request comment path gates on a GitHub App installation and a refs/pull/ ref. The customer asking for this is on GitLab. Designing the gate around a check run means designing a gate they cannot use and then waiting on VCS work that is not scoped anywhere. The check run is the upgrade once a GitLab client exists, and it changes nothing about the gate; it renders the same verdict in a second place.

Blocking is also the deliberate difference from the prior art, and it is the whole product. Datadog’s Early Flake Detection tags new tests as flaky and stops there; blocking the merge is a separate Quality Gates purchase. Splitting it that way is coherent for a platform that sells gates, and incoherent for us: a customer who is already blocking their own merge requests by hand is not asking for a better tag, and an exit code is the cheapest possible way to give them the block on the CI they actually run. Detection without the block is the feature Tuist already has.

The stress pass still uploads, because an exit code with no explanation sends the author back to rerunning CI, which is the behaviour this feature exists to remove. The upload is what makes the block explainable: which case, which repetition, which failure. Repetitions already have somewhere to land.

Those uploaded runs are repetitions of a deliberately chosen set, not independent observations, and have to be recorded as such. Ten forced repetitions of one test in one job are not ten organic CI runs, and letting the flakiness aggregates read them as such would let the gate manufacture the statistic it exists to protect. The flakiness monitors read per-case daily and rolling-window aggregates; a stress pass has to be legible to those as one gated observation rather than N. That is a requirement on how the pass is recorded, not a proposal about the row.

Quarantine is inherited, never created

A new test that is already muted cannot fail the gate, and that is the right answer with no new policy. Muting already means “this runs and its failure is masked”, enforced in TestQuarantineService.markQuarantinedTests and in the Gradle plugin. The gate inherits that mask rather than overriding it, so a team that already decided to tolerate a test does not have the decision reversed by a feature they enabled for a different reason. A skipped test never runs, so it never enters the candidate set and there is nothing to decide. The muted case is still stressed and still recorded, because a newly added muted test failing four of ten is exactly what a reviewer wants in front of them before the mute becomes permanent.

A test that fails the gate blocks; it is not auto-quarantined. The codebase already argues the opposite direction: test_case_ids_with_successful_default_branch_run/3 exists to keep cases with no successful default-branch run out of automated quarantine, naming the pull-request-only test as the case it protects. Auto-quarantining here would mute a test nobody has ever trusted and then let it land muted, converting the gate from “fix this before it lands” into “land it broken and forget”. Quarantining stays a human decision, and the existing --skip-quarantine remains the escape hatch for the run.

Cost, and why truncation is never silent

The stress pass touches only the tests the branch added, which is the smallest set in the suite on any given merge request, and the retry curve prices each of them by its own duration. For the ordinary merge request adding two or three fast unit tests, the added cost is seconds against a suite of thousands. The worst case is bounded by the faulty-session threshold, the candidate cap, and the per-test exclusion above five minutes acting together, and a wall-clock ceiling on the pass as a whole (starting value: 10 minutes) backstops whatever those three still let through.

A gate that quietly stresses twelve of forty candidates and reports green is worse than one that refuses. Every bound is therefore reported rather than applied silently, including the per-test exclusion, which is the one most likely to hide a slow flaky test from the very report that exists to surface it.

What each build system contributes, stated asymmetrically because it is asymmetric

• Xcode: the set is -only-testing, built from identifier lists the CLI already assembles for selective testing and sharding and already carries on the test record. The loop is -test-iterations with relaunch enabled. The results already have a home in test_case_run_repetitions. Both tuist test and tuist xcodebuild test route through TestService, so the gate lands once for both.
• Gradle: the set is Test.filter.includeTestsMatching, exactly what TuistTestSharding.kt already does with the shard plan. The loop is explicit re-execution, because Gradle’s up-to-date checks will otherwise decline to run the task a second time with unchanged inputs. That difference is the whole reason this RFC says the loop is the per-build-system part. The verdict query, the guards, the report, and the exit code are shared with Xcode.
• Bazel: our value there is the verdict and the gate, not the loop, and it is honest to say so. --runs_per_test already does unconditional repetition better than anything we would write, and --flaky_test_attempts is the primitive to stay away from for the same reason as Xcode’s. What Bazel lacks is the other half: with no test ingestion there is no default-branch history to subtract against, so the verdict has nothing to answer. Bazel is blocked on ingestion rather than on this feature, and this RFC’s obligation is to build nothing that has to be unbuilt when ingestion lands. Concretely: keep the verdict keyed on the (name, suite, module) identity the tests tables already share, and keep the loop behind the client, so Bazel support is a build_system value plus a filter translation. See spec #74.

What this gate cannot catch

Running a new test alone, repeatedly, catches self-contained nondeterminism and nothing else. A test that passes in isolation and fails after some other test has polluted shared state passes this gate, and so does a test that pollutes state other tests depend on. That is a real and common flake class, and the gate would be oversold if this RFC did not say so plainly. It is also a limitation the prior art shares, so it is not a reason to prefer anything else on the market. Catching it means running the new test inside its module’s ordering rather than alone, or shuffling, which multiplies the cost by the module rather than by the test. It is a follow-on with its own cost argument, not a v1 omission to paper over.

Scope

In scope: the request-time newness verdict on the server, using the same branch-presence definition as check_new_test_cases/3; the second pass in the CLI’s TestService, covering both tuist test and tuist xcodebuild test; the same second pass in the Gradle plugin, alongside the sharding and quarantine services it already has; the duration-keyed retry curve, the faulty-session threshold, the candidate cap, the per-test slow exclusion, and the reporting of every one of them; the exit-code gate; and recording the stress pass so it is explainable without inflating flakiness aggregates.

Out of scope, but aligned: Bazel, blocked on test ingestion (spec #74); the check run and the merge-request comment, blocked on a GitLab client and a rendering of the same verdict rather than a change to it; order-dependent flakiness; pre-run enumeration of test cases and the known-tests-list model, which become worth building only if the two-pass shape proves too slow and which need no server change when they do; and auto-quarantine of gate failures, which is a policy the codebase currently argues against.

Trade-offs

Advantages

• Catches a flaky test while its author still holds the context that produced it, rather than days later on the default branch in someone else’s pull request.
• Ships the half of flaky-test management the market mostly does not have, and ships the block that the one product with the loop sells separately.
• Adds no new seam. The verdict rides the same pre-run server fetch as quarantine and the same set-of-identifiers contract as sharding, both already implemented on both clients.
• The candidate set is observed rather than inferred, so the gate cannot be silently empty the way a diff-derived one can, and each candidate’s measured duration prices its own retry count.
• Independent of the ingestion pipeline’s health, because the verdict is answered at request time rather than read back out of a processing queue.
• Ships to a GitLab customer on day one, and to every other CI provider at the same time, because the signal is an exit code.
• Costs nothing on the overwhelming majority of merge requests, which add no tests.
• Reuses the repetition path end to end: xcodebuild produces repetitions, the payload carries them, the server stores them.
• Leaves quarantine policy exactly where it is in both directions: muted tests stay masked, and gate failures do not create new mutes.
• Starting values are inherited from a production-validated implementation rather than guessed, so the first tuning pass starts from a known-workable curve.

Disadvantages

• Catches only self-contained flakiness; order-dependent flakes pass the gate.
• Adds a second test invocation to the job, so job duration now varies with how many tests the branch adds. Bounded, but no longer constant.
• Repetition is a probabilistic filter, not a proof, and the filter is weakest exactly where tests are slowest: a four-minute test gets two repetitions and a rare flake in it usually lands.
• The 90-day, CI-only definition of “new” means a long-dormant test reads as new and gets stressed for no reason. Cheap, but it will be asked about.
• Sharded runs see only their own slice, so the gate is per shard rather than per merge request (see Open questions).
• Gradle needs the up-to-date check defeated explicitly, which is small but real per-build-system work with its own regression mode.
• Bazel gets nothing until test ingestion exists, so “works across build systems” is a claim about the seam rather than about shipped coverage.

Alternatives considered

Push the known-test set to the client and diff locally

The shipped model in Datadog Early Flake Detection: fetch the full set of known tests before the session, and treat anything the runner discovers that is not in the set as new, deciding it in-process as tests execute. It is a genuinely good design and it is single-pass. Rejected for v1 on client shape rather than on merit: it needs an in-process hook into the test framework, which their tracer has by instrumenting the runner and Tuist does not have on Xcode, because the CLI shells out to xcodebuild and reads the result bundle afterwards. It also moves a set proportional to the whole suite over the wire instead of one proportional to the branch, which is the wrong scaling for the very large suites this feature targets. Worth revisiting on Gradle specifically, where the plugin does have test listeners and could make the decision in-process.

Detect new tests from the branch diff

Parse the diff for added test declarations and stress those. Rejected because it is per-language and per-framework and wrong at exactly the edges that matter: inherited XCTest cases, @Test display names, parameterised identity, annotation-driven discovery. Every one of those failure modes produces an empty candidate set and a green gate, which is the worst possible output for something a team is relying on to block merges. Notably, no product in the prior art does it this way either.

Enumerate test cases before the run and stress in a single pass

Ask the build system for its test cases, send them to the server, then run once with the repetitions folded in. Rejected for v1 rather than on principle: it needs an enumeration primitive the CLI does not use today (-enumerate-tests after a build) and one Gradle does not have at all, so it front-loads per-build-system work onto both clients before the feature has proven itself. It also gives up the first pass’s duration measurement, which is what sizes each candidate’s retry count. The server side is identical either way, which is what makes this a clean upgrade if the two-pass shape turns out to be too slow.

Read is_new back from the ingested run

Upload the first pass, wait for it to be processed, then ask which of the resulting test case runs were flagged new. Rejected because it makes a merge-request outcome depend on the processing queue draining, and that queue has wedged in production before. The information is identical; the coupling is not.

Let the build system’s retry mechanism be the gate

Turn on -retry-tests-on-failure, --flaky_test_attempts, or the Test Retry Gradle plugin and treat a retried test as a gate failure. Rejected because those primitives only rerun what already failed, which excludes the entire population the gate exists to examine, and because their purpose is to hide flakiness in CI rather than surface it. This is also the shape Develocity’s within-build detection depends on, which is why that detection sees only flakes that happen to fail on their first attempt.

A flat repetition count for every test

Simpler to explain and simpler to configure. Rejected because the value of another repetition is constant while its cost is not, so any single number is either too weak for fast tests or too expensive for slow ones. The prior art started here and moved to a duration curve, which is a datapoint worth taking for free.

A server-issued stress plan, symmetric with the shard plan

Have the server compute the set to stress and hand it to the client the way it hands out shards. Rejected because it cannot be built: the shard plan is derived from the branch’s suite history, and a new test has none, so the server would be answering a question about tests it has never seen. The subtraction framing is the same seam pointed the only direction it can point.

A GitHub check run as the failure signal

Report the gate through create_check_run, which already exists. Rejected as the v1 signal because it is GitHub-only, the customer asking for this is on GitLab, and no GitLab client exists in the server. Kept as an upgrade, where it adds a rendering rather than changing the gate.

Tag the flaky test and let a separate gate decide, as the prior art does

Detect and label, and leave blocking to a distinct product surface. Rejected because it is the feature Tuist already has: first_run events, flakiness monitors, and quarantine already tag and label. The customer is not asking for a better label, they are asking for the merge to stop, and the exit code is the cheapest way to give them that on the CI they already run.

Auto-quarantine tests that fail the gate

Mute the offending test instead of blocking, so the branch can land. Rejected because it contradicts a policy the codebase already implements deliberately, and because it inverts the feature’s purpose, replacing “fix this before it lands” with “land it muted”.

Rollout

The gate is opt-in on the client and off by default, so a project that has not asked for one never gets one. Enablement then proceeds gate by gate:

1. Land the request-time newness verdict on the server. It is inert until a client calls it, and it is the piece both clients and eventually Bazel share.
2. Land the second pass in the CLI behind the opt-in flag, with the retry curve, the faulty-session threshold, the candidate cap, the slow exclusion, and their reporting. Xcode first, because TestService covers both entry points at once.
3. Enable on the requesting customer’s project and measure the two numbers that decide whether the inherited starting values transfer: the catch rate, meaning how often the gate blocks and whether the blocked tests were genuinely flaky, and the added wall clock per merge request.
4. Land the Gradle side once the verdict query is stable. It is the same client logic against the same endpoint, plus the forced re-execution.

Revisit the retry curve and both guards from that telemetry rather than from argument. The starting values come from a JVM, Python, Ruby and JavaScript population; an iOS suite of simulator-bound tests sits in different duration buckets, and the curve is the thing most likely to need moving.

Open questions

• Sharded runs. With a shard matrix, each job sees only its slice, so each shard gates on the new tests it happened to run, and the faulty-session percentage is computed per shard rather than per suite. Per-shard gating needs no fan-in, but no single job holds the whole picture, and “the merge request passed the stress gate” then means only that every shard did. Whether that is good enough or whether a fan-in step is needed is a product call, not something the codebase answers.
• Where the second pass is invoked. Whether tuist test runs it at the end of its own invocation, or whether it is a separate CI step the user wires up. The first is less to explain and less to misconfigure; the second keeps job duration predictable and composes better with a shard matrix.
• Whether the verdict is a new endpoint or a filter on the existing test-case listing. The listing is project-scoped and paginated and the query needs a body of candidate identifiers, which sits badly on a GET. Cheap either way, but it decides whether the Gradle plugin’s API surface grows.
• Whether stress-pass repetitions should feed cross-run flaky detection at all, or only render on the run that produced them. They are genuine evidence of flakiness, but they are also solicited evidence, and the flakiness monitors are threshold-driven.
• Whether the duration buckets transfer to iOS at all. The inherited curve gives a five-second test ten repetitions, and a large share of a simulator-bound iOS suite may sit above that boundary, in which case the effective default is three repetitions and the gate is much weaker than the prior art’s numbers suggest. This is measurable on the customer’s own data before anything ships and should be.
• The customer’s own numbers. What repetition count and what cap their homegrown version uses today, and what its catch rate looks like. That is the only datapoint available from an actual iOS suite, and it is worth asking for before shipping a curve borrowed from a different population.

References

• Spec #74, Kura REAPI Cache Instrumentation for Bazel: the Bazel direction whose test ingestion is the prerequisite for the verdict half of this gate on Bazel.
• Datadog Early Flake Detection: the only shipped implementation of the active loop. Parameters quoted above are from the ddtrace sources (ddtrace/internal/ci_visibility/_api_client.py, ddtrace/testing/internal/retry_handlers.py, ddtrace/internal/ci_visibility/api/_session.py) rather than the documentation, which omits them.
• Buildkite Test Engine, test state and quarantine: the muted/skipped model and the first-execution monitor that Tuist already matches.
• Trunk Flaky Tests detection: passive monitors, including a New Test monitor that labels without rerunning.
• Develocity flaky test detection: within-build detection built on retry-on-failure, plus cross-build detection on matching input fingerprints.
• server/lib/tuist/tests.ex: check_new_test_cases/3, filter_first_run_test_case_runs/2, test_case_ids_with_successful_default_branch_run/3.
• cli/Sources/TuistKit/Services/TestService.swift and cli/Sources/TuistKit/Services/TestQuarantineService.swift: the pre-run quarantine fetch and the post-run mask.
• cli/Sources/TuistKit/Services/Sharding/ShardPlanService.swift and gradle/src/main/kotlin/dev/tuist/gradle/TuistTestSharding.kt: the server-issued plan executed by two clients.
• gradle/src/main/kotlin/dev/tuist/gradle/TuistTestQuarantine.kt: the same pre-run fetch on the Gradle side.

No comments yet

Comments from contributors and members will appear here.