Postmortem #2 · Postmortem: test run ingestion outage from a masked ClickHouse migration, 19 August 2026

Postmortem: test run ingestion outage from a masked ClickHouse migration, 19 August 2026

Incident summary

On 19 August 2026 between 14:30 and 14:51 Coordinated Universal Time, every read and write of the test_runs ClickHouse table on tuist.dev returned an error. Test run ingestion from the command-line interface failed, and build run pages in the dashboard crashed.

The deployed server image declared two columns on the test_runs schema that did not exist in the database. The migration that should have added them had been silently skipped on every deployed environment because it shared a version number with an earlier migration that was already recorded as applied.

The defect was detected by continuous integration two hours and fifteen minutes before it reached production, with an error message naming the exact problem. Nothing acted on that signal.

Impact

• Production was affected for approximately 21 minutes, from the deploy at 14:30 to the remediation at 14:51:31 Coordinated Universal Time.
• Canary ran the same image and was remediated at 15:07:39 Coordinated Universal Time. The start of canary impact was not determined.
• Both the write path (test run creation from the command-line interface) and the read path (build run pages, latest-test lookups) failed. Any query touching Tuist.Tests.Test returned Ch.Error Code 47, UNKNOWN_IDENTIFIER.
• Server logs showed roughly nine errors per minute in the final minutes before remediation, spread across several customer accounts.
• Sentry recorded only 4 events under issue TUIST-4H5. The undercount is itself a finding: only the LiveView path reports to Sentry, so the failing API path was visible in logs alone.
• The number of failed requests was not established, and no data was lost. Runs that failed to record were absent rather than corrupted.

Detection

Detection worked. Response did not.

The main-push continuous integration run for the merge that introduced the collision (run 32251644536, 12:14 Coordinated Universal Time) failed with:

** (Ecto.MigrationError) migrations can't be executed, migration version 20260817120000 is duplicated

That is an unambiguous, fatal, correctly attributed message, printed two hours and fifteen minutes before the resulting image reached production. The first human awareness of the incident came from a Sentry alert at 14:35, after customer-visible failures had begun.

Timeline

All times Coordinated Universal Time on 19 August 2026 unless stated otherwise.

• 18 August, 15:52: pull request 12420 merges, adding priv/ingest_repo/migrations/20260817120000_create_test_case_runs_recent_window_per_case_mv.exs. Its version is recorded in schema_migrations on every environment.
• 18 August, 10:07: the Server workflow’s Test job runs for pull request 12403 and passes. This is the last time that job ran for that pull request, and it precedes the merge above by nearly six hours.
• 12:12:51: main-push continuous integration begins failing on an unrelated :chrome_not_found breakage in the Test job.
• 12:14:34: pull request 12403 merges on its 26-hour-old passing check, adding 20260817120000_add_caller_test_selection_to_test_runs.exs with the same version as the migration from 18 August, plus the Tuist.Tests.Test schema fields that depend on it.
• 12:14:41: the main-push run for that merge fails with the duplicate-version error. No alert is routed and no gate is blocked.
• 13:52: pull request 12480 merges, renumbering the migration to 20260819130000. This corrects future migrations but does not help environments already running the dependent code, and the image carrying it has not been built.
• 14:30:04: the production migrate Job for image sha-0b9ead317f38 completes. It applies nothing new, because the masked version is already recorded. The server rollout proceeds and production begins failing.
• 14:35:27: Sentry opens issue TUIST-4H5.
• 14:51:31: a one-off Job applies the missing DDL to production and records the renumbered version. Errors stop.
• 15:07:39: the same remediation is applied to canary, which required a just-in-time elevation for write access.

Root cause

Two pull requests chose the same hand-written migration timestamp, 20260817120000, in server/priv/ingest_repo/migrations/. Timestamps in this repository are picked by hand and rounded, which makes same-window collisions likely rather than remote.

Ecto’s duplicate-version check, Ecto.Migrator.ensure_no_duplication!/1, inspects only the list of pending migrations. On a database that had already recorded 20260817120000 from the first pull request, the second file counted as applied and the check never fired. Its ALTER TABLE was therefore skipped permanently and silently, on every deployed environment at once.

Tuist.Release.assert_all_migrations_up/1, which exists precisely to catch a half-migrated database, did not help. Migration status is keyed on version, so both files reported :up.

The result is a state the system has no way to notice: code that declares two columns, a database that lacks them, and a migration ledger that claims everything is applied.

Why the signal did not stop the deploy

Three independent gaps, each sufficient on its own to let this through.

1. The main branch does not require continuous integration to pass. The branch ruleset carries only deletion and non_fast_forward rules. Branch protection requires one review, code owner review, linear history, and conversation resolution, but has no required status checks at all. There is no merge queue, and no merge_group run has ever fired, even though merge_group: {} is already declared in .github/workflows/server.yml. A pull request therefore merges on whatever continuous integration ran against its own branch, however old, and whatever has landed on main since is never tested against it.
2. Deployment does not depend on continuous integration. .github/workflows/server-production-deployment.yml triggers on push to main with no dependency on the Server workflow’s conclusion. A red main does not delay or block a release.
3. A red main carries no information. Twelve of the last twenty main-push Server runs failed. The run that printed the migration error was already failing on an unrelated :chrome_not_found breakage that had started ninety seconds earlier and remained unfixed three hours later. One more red run was indistinguishable from the ambient state.

Contributing factors

• Migration versions are chosen by hand and rounded to the hour, rather than generated. Every colliding version so far ends in 120000.
• A collision between two pull requests exists only once both are on main, so neither pull request can see it in isolation. Without a merge queue there is no point in the process where the combination is evaluated before it lands.
• The same masking mechanism had already been observed once from branch deploys to staging, so this was a known failure mode rather than a novel one.
• Pull request 12480 is titled “renumber the duplicated ingest migration and guard against reoccurrence”, but contains only the rename: one file changed, zero insertions. The guard implied by the title was never written, and its apparent existence discouraged a second look.

Remediation

ecto_ch emits a bare ADD COLUMN with no IF NOT EXISTS (deps/ecto_ch/lib/ecto/adapters/clickhouse/migration.ex:328), so applying the data definition language by hand without recording the version would have made the next deploy’s migrate Job fail on a column that already exists, blocking the whole release cascade. Remediation therefore did both, per environment:

ALTER TABLE test_runs ADD COLUMN IF NOT EXISTS only_test_identifiers Array(String) DEFAULT [], ADD COLUMN IF NOT EXISTS skip_test_identifiers Array(String) DEFAULT []
INSERT INTO schema_migrations (version, inserted_at) VALUES (20260819130000, now())

This ran as a one-off bin/tuist eval Kubernetes Job copied from the live tuist-tuist-server-migrate pod specification, since kubectl exec does not work through the Pomerium gateway. ClickHouse needs no separate migration role: unlike Postgres, Tuist.Release swaps the database URL only for Tuist.Repo, so the Tuist.IngestRepo runtime credential already carries ALTER.

Two verification notes for anyone repeating this. A SELECT against system.columns from inside that Job returns nothing, because system tables are access-filtered for that user, so confirm with a plain SELECT of the new column instead. And kubectl logs timestamps on the server pods ran roughly four minutes ahead of the application’s own log timestamps, so judging whether errors stopped requires reading the timestamp inside the log line.

What went well

• The remediation was surgical and additive. Adding two columns with defaults is a metadata-only operation in ClickHouse, so there was no data rewrite and no rollback risk.
• Recording the migration version alongside the manual change meant the in-flight release pipeline needed no intervention and stayed green.
• Read-only production access was sufficient for the entire diagnosis, including confirming the columns were absent, before any write was proposed.

What went poorly

• A precise, fatal, correctly worded signal sat unread for two hours and fifteen minutes while the change it described rolled to production.
• The pull request merged on a check that was 26 hours old and predated the conflicting change.
• Canary was left broken for a further sixteen minutes after production was fixed, because write access there required a separate elevation that had not been requested in advance.
• A previous postmortem-style commit claimed a guard that was never implemented, which is worse than claiming nothing.

Prevention

The recommended primary change is the merge queue.