Hive Hive
Sign in

fix(server): keep Keygen out of hosted server startup path (hotfix)

GitHub issue · Closed

Metadata
Source
tuist/tuist #11271
Updated
Jul 5, 2026
Details

What changed

Tuist.License.assert_valid!/1 now skips license validation when the server runs in hosted mode (tuist_hosted?()), alongside the existing dev/test skips, and it no longer raises when the license service is unreachable (it logs and continues, still failing closed on a definitively invalid/expired/missing license).

Why — production outage (503)

tuist.dev was returning 503. Every tuist-tuist-server pod was in CrashLoopBackOff:

The license validation failed with the following error: %Req.TransportError{reason: :timeout}
Tuist.License.assert_valid!/1 lib/tuist/license.ex:177
Tuist.Application.start/2 lib/tuist/application.ex:57
Kernel pid terminated (application_controller) {application_start_failure,tuist,...}

Root cause

assert_valid!/1 is called unconditionally at the end of Tuist.Application.start/2 and raises on any failure, including a transport timeout reaching api.keygen.sh. The production cluster could not reach Keygen, so every pod boot re-validated (the Redis-cached license had expired), the raise aborted OTP application startup, the BEAM terminated, and nginx was left with no healthy upstream → 503.

Why prod specifically: since #11150 all server egress is SNAT’d through one dedicated stable egress IP. After hours of crash-loop hammering, Keygen/AWS rate-limited that shared IP — so the same validate-key call returns 200 from other IPs (laptop, canary, staging) but times out from prod.

Why this fix

Keygen licensing only applies to self-hosted / on-premise deployments. The hosted service (TUIST_HOSTED=1) should never have had a third-party license server as a hard, synchronous startup dependency. This change removes it from the hosted startup path entirely, and makes the check fail-open on transient transport errors so a self-hosted server is likewise never bricked by a Keygen blip.

Validation

  • Confirmed TUIST_HOSTED=1 on the running prod deploy, so the new guard fires.
  • Reproduced the crash signature from pod logs; the raise originates from the {:error, error} branch on a Req.TransportError timeout.

Deploy

Marked (hotfix) so it ships straight to production, bypassing the canary + acceptance cascade, to restore the site.

🤖 Generated with Claude Code

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments

No GitHub comments yet.