Hive Hive
Sign in

feat(server): gate production on an oldest-supported-CLI acceptance suite

GitHub issue · Closed

Metadata
Source
tuist/tuist #11097
Updated
Jun 24, 2026
Domains
Cache
Details

What changed

Adds a backward-compatibility acceptance suite to the production deployment cascade that runs the oldest CLI version we still support (3-month window, currently pinned to 4.155.0) against the freshly-deployed canary, and blocks production promotion on it.

  • e2e/module_cache_backward_compat.bats — using the pinned old binary: logs into canary, creates a throwaway project under the tuist org, warms the module cache with tuist cache, wipes the local cache (isolated via XDG_CACHE_HOME / XDG_STATE_HOME / XDG_CONFIG_HOME), then forces a remote pull with a focused tuist generate. The validation is behavioral: a cached xcframework can only be linked in the generated project if the signed cache-artifact download succeeded, so exit code + xcframework-linked proves an old CLI pulled from cache without a signature rejection (no brittle log-string matching).
  • e2e/fixtures/module_cache_app/ — minimal iOS app + one static framework, built only from long-stable ProjectDescription APIs so the old CLI can compile the manifest.
  • .github/workflows/server-production-deployment.yml — new backward-compatibility-acceptance job (needs canary, tuist-macos, server-k8s-canary env), added to production.needs. Pinned via MIN_SUPPORTED_TUIST_VERSION.
  • server/priv/docs/en/cli/compatibility.md (+ CLI docs sidebar) — a customer-facing Compatibility page under the CLI docs stating the 3-month support window. This is a public commitment, so it lives in the product docs (modeled on how tools like Stripe publish a versioning and support policy), not the internal handbook.

Why

The existing acceptance-tests job always uses the CLI built from HEAD, so it can never catch a server change that breaks an older CLI. That is exactly the cache-signature incident: the server stopped attaching the cryptographic signature to cache-artifact downloads (downloadCacheArtifact); newer CLIs had already stopped requiring it, but older CLIs still ran SignatureVerifierMiddleware and rejected every cache response, dropping cache hit rate to 0% and wiping out CI for customers pinned to an older release. This suite pins that floor and exercises the exact path that broke, blocking promotion if an older-but-supported CLI can’t pull from cache.

Version selection

The server’s minimum supported CLI version is @minimum_supported_cli_version in server/lib/tuist_web/plugs/warnings_header_plug.ex (currently 4.118.1; below it the server emits a deprecation warning and degrades server-side features). We pin the gate to 4.155.0 (released 2026-03-04, ~3 months back), comfortably above that floor and within the 3-month window. Both the auth login server flag (--path on older, --url on newer) and the project create --build-system requirement changed across the window, so the suite is written for the current (>= 4.155) interface; move the pin forward only.

How this was validated

The real gate cannot be dispatched without running the full deploy cascade, so the job was validated in isolation against the live canary via a temporary standalone workflow (added during review, now removed). Dispatched manually against 4.155.0, it ran green end-to-end:

# Using tuist binary: 4.155.0
# Logging in to https://canary.tuist.dev...
# Creating throwaway project tuist/bc-...
# tuist cache output: ... All cacheable targets have been cached successfully as xcframeworks
# tuist generate output: ...
ok 1 oldest supported CLI pulls the module cache from canary without a signature error

That ok 1 confirms the full chain on the pinned version: install 4.155.0 -> login -> project create -> tuist cache warm (built and uploaded the xcframework) -> wipe local cache -> tuist generate focused pull from canary -> exit 0 and the cached xcframework linked in the generated project.

Issues found and fixed while validating against canary:

  • auth login server flag flips across versions (--path vs --url); handled with a fallback.
  • project create requires --build-system non-interactively on >= 4.155 (older versions reject it), so the pin and the flag must agree.
  • test_helper load placement and XDG_CONFIG_HOME isolation (credentials live there, not under the cache dir).
  • Dropped the version-pinned Select Xcode step to match main’s macos-xcode-image change (it failed on fleet nodes missing the pinned Xcode).

Merge note

The gate is blocking (production.needs). While validating, canary was returning ~80% 500s on writes due to Postgres connection-pool exhaustion (DBConnection.ConnectionError, ~999 in 6h on canary, 0 on production), a separate canary incident tracked under its own task. Hold merge until canary is healthy so a flaky canary cannot block production promotion through this gate.

🤖 Generated with Claude Code

Comments
TA
tuist-atlas[bot] Jun 6, 2026

This feature is now available in xcresult-processor-image@0.12.0. Update to this version to use it.

TA
tuist-atlas[bot] Jun 6, 2026

The oldest-supported-CLI acceptance suite gate for production deployment is now available in server@1.207.0. Update to get this feature.