Postmortem #1 · Postmortem: Tuist Swift Package Registry incidents, July and August 2026

Postmortem: Tuist Swift Package Registry incidents, July and August 2026

Incident summary

Between approximately 23 July and 4 August 2026, the Tuist Swift Package Registry experienced a connected series of failures:

1. Package versions disappeared from registry metadata or alternated between different catalog states.
2. Recovery work restored or regenerated archive bytes under existing package versions, changing checksums that clients expect to be immutable.
3. Globally replicated object storage sometimes served different archive bytes from different regions.
4. Some archives were structurally invalid, had incorrect directory permissions, or lost signed framework bundle symlinks.
5. The sync service repeatedly exhausted its GitHub request quota, discarded work instead of retrying after reset, and previously exceeded its memory limit.

The events were related but not identical. The first failure came from competing catalog writers and stale full-catalog updates. The recovery then introduced a second class of failures because it changed archive identity without a safe, deterministic, and reversible repair process. Global object-cache inconsistency and archive construction defects prolonged the customer impact.

Three public incident windows were recorded:

• Incident 7: 27 July, 13:01 to 13:11 Coordinated Universal Time. The status incident was closed before all customer-visible failures were understood.
• Incident 8: 28 July, 12:59 to 17:21 Coordinated Universal Time.
• Registry users failures: 3 August at 22:04 through 4 August at 08:07 Central European Summer Time, approximately 10 hours.

The status windows understate the full user-visible period. The first confirmed customer report arrived at 08:42 Coordinated Universal Time on 27 July, and reports continued after incident 7 was closed.

Impact

Confirmed impact includes:

• One customer’s clean-resolution test failed for 20 of 25 packages in issue 12098.
• Multiple enterprise customers reported missing versions, checksum mismatches, directory permission errors, and archives that could not be extracted. Some teams paused registry rollout or disabled it for more than 40 developers.
• The 28 July recovery restored 5,687 archives across 538 packages.
• A later production census found checksum history changes for 22,113 versions across 1,643 packages. This is a lower bound on changed registry content, not a count of failed builds or affected customers.
• The census found 188 reachable but malformed versions across 18 packages. All 18 packages were regenerated.
• Approximately 26,200 other routine archive rewrites occurred between 24 and 31 July.
• Forty-seven large-archive variants were not covered by the initial scan.
• 34,857 versions created after the migration had no pre-migration baseline for comparison.
• The bucket had never enabled object versioning, so overwritten archive bytes could not be recovered.

The customer count is unknown. Swift Package Manager keeps trust-on-first-use fingerprints on each machine under ~/.swiftpm/security/fingerprints/<identity>.json. Only machines that resolved a version before its bytes changed and retained that fingerprint would fail with a fingerprint mismatch. Registry history therefore cannot identify every affected machine or developer.

There is no evidence of package source tampering by an external actor. The checksum changes came from Tuist’s own sync and repair paths.

Detection

We did not detect the first catalog regression ourselves. A customer opened issue 12098 on 27 July after cold resolutions failed.

Detection was weak because:

• Registry metrics were aggregate-only after 10 July. We could not query downloads by package, version, account, or checksum.
• We had no synthetic cold-resolution probe covering metadata, archive download, checksum equality, extraction, directory permissions, and signed bundle symlinks.
• We had no alert for a version disappearing from an existing package catalog.
• We had no alert for catalog checksums changing after publication.
• The cache deployment workflow had failed 38 consecutive times since 3 June, but that staleness did not trigger an operational alert.
• Rate-limit failures were logged but sync work was discarded, and there was no alert tied to lost coverage.

Timeline

Before the customer report

• 9 July: migration toward a standalone registry read service began while release publication still existed in more than one application.
• Around 23 July: the earliest known catalog drift appeared.
• 24 through 31 July: approximately 26,200 routine archive rewrites occurred, creating a much wider checksum-history exposure than the later targeted restore.

27 July

• 08:42 Coordinated Universal Time: a customer reported 20 failures across 25 packages in issue 12098.
• 13:01: incident 7 opened.
• 13:11: incident 7 closed after catalog entries were restored.
• Later that day: customers continued reporting missing versions and checksum mismatches in issue 12105. One package’s advertised checksum alternated roughly every ten minutes, showing that the registry had not reached a stable state.

28 July

• 12:59 Coordinated Universal Time: incident 8 opened.
• A bulk recovery restored 5,687 archives across 538 packages.
• Restored archives did not always match the bytes previously published for the same version. Clients with an existing trust fingerprint then rejected them.
• 17:21: incident 8 closed, although the repair consequences continued.

29 through 31 July

• The managed registry sync service was enabled as the intended sole writer in pull request 12136.
• Consistent archive reads, authoritative registry origin handling, and storage consistency headers were added in pull request 12120 and pull request 12166.
• Signed framework bundle symlink preservation was fixed in pull request 12139.
• The sync service’s production memory request and limit were raised after it had been terminated between 9 and 21 times per day for exceeding its one-gibibyte limit. See pull request 12188.
• The cache deployment pipeline was repaired in pull request 12187. That deployed code which still contained a legacy registry writer. The legacy writer was first gated in pull request 12190, then deleted in pull request 12192.
• A production repair census and runbook were committed in pull request 12194.
• Archive validation and safer forced resync behavior were introduced in pull request 12193.
• The first repair implementation silently failed to pass object metadata, so 72 jobs overwrote archives and then failed verification before catalog publication. Pull request 12197 fixed the missing option.
• A second production-only failure showed the returned header value could be a list rather than a string. Pull request 12198 normalized the value. The tests had stopped at a mocked storage boundary and did not exercise the complete upload, read-back, and publish contract.

3 through 4 August

• A third public incident opened after users reported directory permission errors and checksum mismatches.
• Tigris global object-cache replicas were found serving divergent archive bytes.
• The registry bucket was moved from global storage to a single region in Frankfurt.
• The public incident was resolved on 4 August at 08:07 Central European Summer Time.

11 August verification

Read-only production inspection found:

• The registry read service and managed sync service each had one ready replica.
• Both current pods had zero restarts after the morning rollout.
• The managed sync service had a one-gibibyte memory request, a four-gibibyte limit, and was using 981 mebibytes at the time of inspection.
• No legacy cache application deployment was present. The managed sync service is now the only registry catalog writer deployed in the Tuist namespace.
• The public availability endpoint returned success.
• Metadata and downloaded archive checksums matched for auth0.auth0_swift version 2.10.0 and adyen.adyen-3ds2-ios version 2.4.3, and both archives could be enumerated.
• The live deployment still had no production override for the default sync limit of 1,000 packages every ten minutes.
• From the current pod’s start at 07:58 through 14:20 Coordinated Universal Time, it logged 2,566 GitHub rate-limit failures and discarded nine scheduled sync passes. Consecutive passes at 14:10 and 14:20 were discarded. The container was healthy while the business process was losing coverage.

The live check proves current endpoint health for the sampled packages, not complete registry correctness.

Root cause

1. Catalog ownership was not structurally exclusive

The migration left two applications capable of publishing the same package catalogs to the same object-storage bucket.

The legacy cache writer read a full catalog through a ten-minute in-memory cache before acquiring the distributed lock. Even when the lock serialized the final write, a writer could publish a stale snapshot after another writer had added versions. The approximately ten-minute checksum and catalog flapping reported in issue 12105 matches this behavior.

Pull request 12192 removed the legacy writer instead of relying on configuration. This made single-writer ownership structural.

A later inventory in issue 12196 did not prove that every archive missing from a catalog was a lost update. Most remaining objects were invalid identifiers, incomplete release jobs, or already repaired content. That correction narrows the scope of the hypothesis but does not negate the directly observed stale-writer behavior during the July incident.

2. Transient failures were interpreted as durable absence

The sync path converted some manifest and permission failures into missing data and could permanently skip releases. Recovery behavior was improved in pull request 12095, but the incident showed that a transient upstream or storage failure must never authorize deletion or replacement of already-published registry state.

3. Recovery did not preserve published artifact identity

A package version is immutable from the client’s perspective. The repair process restored or regenerated source from upstream and published new bytes under an existing version. Several archive paths were non-deterministic, including symlink repacking and repositories with submodules. The same source could therefore produce a different checksum.

The bucket had no object versioning. Once an archive was overwritten, the previously trusted bytes were irrecoverable.

Issue 12195 tracks deterministic archive creation and remains open.

4. Replicated object storage did not provide the consistency the registry contract requires

During the August incident, global Tigris replicas returned different bytes for the same object. A client could fetch metadata from one consistent view and archive bytes from another, producing real checksum disagreement even after clearing a local fingerprint.

Consistent reads and writes were added in pull request 12107, pull request 12120, and pull request 12166. The bucket was then moved to a single Frankfurt region.

5. Operational safety depended on component health rather than correctness signals

The service could be ready in Kubernetes while not performing complete synchronization. It could also exceed memory limits, exhaust GitHub request quota, discard scheduled work, or run stale deployment code without a page.

The 11 August cluster inspection demonstrates that this gap remains: the pod is ready and has zero restarts, yet thousands of rate-limit failures and discarded sync passes are occurring.

Contributing factors

• The migration sequence enabled the new writer before removing the old writer’s code.
• Full-catalog read, modify, and write operations amplified the effect of stale reads.
• The recovery was executed before we had a complete inventory, immutable backups, deterministic packaging, or public-path validation.
• Production behavior differed from mocked storage tests twice in succession.
• Global object replication was incompatible with the registry’s immutable checksum contract under the observed cache behavior.
• Archive validation did not originally cover extraction, root-directory permissions, signed bundle symlinks, and submodule completeness together.
• Rate-limited jobs were discarded for a future catalog pass instead of being delayed until the quota reset.
• The deployment pipeline could remain broken for weeks without an alert.
• Aggregate telemetry prevented a precise blast-radius calculation and targeted customer communication.

What went well

• Customer reports included reproducible cold-resolution failures and helped identify that clearing local state did not always fix the issue.
• Eduardo’s full object census corrected early estimates and separated targeted restore activity, routine rewrites, malformed archives, dead duplicates, and versions without a baseline.
• The team moved from configuration-based writer ownership to deleting the legacy writer.
• Repair tooling gained extraction checks, read-back verification, and refusal to change a checksum without an explicit override.
• The public endpoint and the two sampled packages are healthy at the time of this draft.

What went poorly

• Incident 7 was closed before the registry had reached a stable state.
• The first response restored availability by changing immutable content, creating a larger and harder-to-measure follow-on incident.
• We could not roll back overwritten objects because versioning was disabled.
• Two repair hotfixes were needed because the complete production storage contract was not exercised before the repair.
• The original 31 July report stopped before the August incident and before the corrected census, so its impact numbers and conclusions were incomplete.
• We still cannot identify exactly which customers retained fingerprints for changed versions.
• A ready pod currently masks incomplete synchronization caused by rate limiting.

Completed mitigations

Prevention commitments

These owners and targets are draft commitments to confirm before publication.

Pull request inventory

Directly related recovery and hardening

• 12095: recover releases after transient manifest failures
• 12107: prevent metadata and archive regressions
• 12120: use consistent archive reads
• 12136: enable managed registry sync as the intended sole writer
• 12139: preserve signed framework bundle symlinks
• 12140: handle rate-limited sync work. This changed jobs to discard for a later catalog pass, which is insufficient and remains an open risk.
• 12166: make the registry origin authoritative and apply consistency headers
• 12187: restore the broken cache deployment pipeline
• 12188: increase sync-worker memory headroom
• 12190: gate legacy cache writers
• 12192: delete legacy cache writers
• 12193: reject invalid archives and make forced repair safer
• 12194: commit production repair scripts and the census workflow
• 12197: forward upload metadata during repair
• 12198: normalize storage header values
• 12205: expose GitHub request-quota metrics
• 12214: add request-quota dashboards
• 12232: improve submodule failure attribution

Adjacent changes reviewed but not treated as root causes

• 12077: strip package root directories improved client-side package handling.
• 12178: pin Colorizer as a registry dependency addressed a separate release problem.
• 12211: support symlinked root binary artifacts fixed a separate client-side SwifterPM behavior.
• 12231: work around an HTTP/2 Swift 6.2 Linux crash addressed a client and toolchain transport crash, not registry data corruption.

Evidence reviewed

• Pedro’s 31 July draft, registry-incidents-postmortem-2026-07.pdf, all 12 pages.
• Eduardo’s object census and final investigation, captured in issue 12191 after the original shared report link expired.
• Customer and engineering Slack threads from 27 July through 6 August.
• Registry-related merged pull requests and open follow-up issues from 21 July through 11 August.
• Read-only production Kubernetes deployment, pod, resource, event, and log inspection on 11 August.
• Public availability and checksum smoke tests on 11 August.

Remaining uncertainty

• We cannot calculate an exact number of affected developers because fingerprints are local to each machine.
• We cannot recover overwritten pre-incident bytes because object versioning was disabled.
• The 34,857 post-migration versions have no earlier baseline.
• The current public smoke test sampled two previously reported packages; it is not a full registry sweep.
• Historical Kubernetes events and logs for deleted pods are not retained in the live cluster view used for this investigation. Historical restart counts therefore come from the merged mitigation work and incident evidence, while the 11 August quota findings come directly from the current pod.