Hive Hive
Sign in

feat(cli): fail fast when the remote CAS cache is unavailable

GitHub issue · Closed

Metadata
Source
tuist/tuist #11624
Updated
Jul 5, 2026
Domains
Cache
Details

What

Adds a per-process circuit breaker for the remote CAS (compilation) cache, so a remote-cache outage degrades to a fast local-compile fallback instead of stalling the build.

  • CASCircuitBreaker (actor) — after N consecutive failures it opens; every subsequent load/save short-circuits to a local build instantly instead of re-discovering the outage. Re-probes after a short cooldown (half-open, single probe) so caching resumes if the backend recovers mid-build.
  • Health classification — a 404 miss (and a too-large save) mean the backend is reachable, so they reset the breaker; 5xx / timeout / connection / auth errors count toward opening it.
  • Short-timeout CAS session — a dedicated URLSession.tuistCAS with a 15s inactivity timeout (vs the shared 90s) so a hung backend surfaces in seconds; the resource timeout stays generous so a large but progressing artifact transfer isn’t cut off.

Wired into CASService.load/save; LoadCacheCASService/SaveCacheCASService thread the short-timeout session through the cache client.

Why

A build fires thousands of CAS load/save calls, many concurrently. When the remote cache is unavailable — a Kura pod roll, a box loss, a network blip — every call today independently waits for its own failure/timeout before the compiler falls back to a local build. On the CAS path there’s no retry (good) but also no shared failure state and a 90s timeout, so one outage becomes a per-request tax: with thousands of units the build slows to a crawl, or stalls entirely if the backend hangs, even though builds still succeed.

The breaker makes one unit’s discovery of the outage benefit all the others: the first few failures trip a shared switch, and the rest short-circuit to local compilation with no network round-trip. This is what makes the cache’s fail-open guarantee performance-safe, not just correctness-safe.

Investigation notes: the CAS load path already collapses a 404 miss and a 5xx into the same outcome = .error, which Xcode treats as “compile locally” (that’s how normal cold-cache misses work) — so fail-open is already correct. This PR only fixes how fast the fallback happens under a real outage.

Safety

  • Never changes build output. A short-circuited load behaves exactly like a cache miss (compiler builds locally); a short-circuited save reuses the existing “skip upload” path (the artifact is already built locally). The breaker only decides when to stop asking a backend that’s already failing.
  • Per-process (per build) — a new build starts fresh and re-probes; no stale state persists.
  • No config knobs (hard-coded threshold/cooldown, per project convention).

Tests

  • CASCircuitBreakerTests — the state machine: opens after threshold, success/miss reset the run, half-open allows a single probe after cooldown, probe failure re-opens, and the error classifier (miss/too-large = healthy; 5xx/auth/URLError = unavailable).
  • CASServiceTests (2 added) — a 503 run trips the breaker and the next load skips the remote entirely (asserts the remote stub is called once); a 404 miss keeps the breaker closed (remote still consulted).
  • swift build of TuistCAS and TuistCASTests is green locally.

Context

This is item #3 of the Kura production-readiness work: it de-risks the customer-plane 503 window during primary-pod rolls (a separate finding — the customer Service currently targets only the primary pod) by making the client tolerate any remote-cache unavailability gracefully.

🤖 Generated with Claude Code

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments
T
tuist[bot] Jul 2, 2026

🛠️ Tuist Run Report 🛠️

Tests 🧪
Scheme Status Cache hit rate Tests Skipped Ran Commit
TuistAcceptanceTests 88 % 145 0 145 45156be98
TuistUnitTests 89 % 3045 6 3039 45156be98
Flaky Tests ⚠️
  • TuistUnitTests: 1 flaky test (View all)
Test case Module Suite
parseTestWithCustomLabelXCResult() TuistXCResultServiceTests XCResultServiceTests
Builds 🔨
Scheme Status Duration Commit
TuistAcceptanceTests 2m 31s 45156be98
TuistUnitTests 3m 31s 45156be98