Hive
fix(server): keep Keygen out of hosted server startup path (hotfix)
GitHub issue · Closed
What changed
Tuist.License.assert_valid!/1 now skips license validation when the server runs in hosted mode (tuist_hosted?()), alongside the existing dev/test skips, and it no longer raises when the license service is unreachable (it logs and continues, still failing closed on a definitively invalid/expired/missing license).
Why — production outage (503)
tuist.dev was returning 503. Every tuist-tuist-server pod was in CrashLoopBackOff:
The license validation failed with the following error: %Req.TransportError{reason: :timeout}
Tuist.License.assert_valid!/1 lib/tuist/license.ex:177
Tuist.Application.start/2 lib/tuist/application.ex:57
Kernel pid terminated (application_controller) {application_start_failure,tuist,...}
Root cause
assert_valid!/1 is called unconditionally at the end of Tuist.Application.start/2 and raises on any failure, including a transport timeout reaching api.keygen.sh. The production cluster could not reach Keygen, so every pod boot re-validated (the Redis-cached license had expired), the raise aborted OTP application startup, the BEAM terminated, and nginx was left with no healthy upstream → 503.
Why prod specifically: since #11150 all server egress is SNAT’d through one dedicated stable egress IP. After hours of crash-loop hammering, Keygen/AWS rate-limited that shared IP — so the same validate-key call returns 200 from other IPs (laptop, canary, staging) but times out from prod.
Why this fix
Keygen licensing only applies to self-hosted / on-premise deployments. The hosted service (TUIST_HOSTED=1) should never have had a third-party license server as a hard, synchronous startup dependency. This change removes it from the hosted startup path entirely, and makes the check fail-open on transient transport errors so a self-hosted server is likewise never bricked by a Keygen blip.
Validation
- Confirmed
TUIST_HOSTED=1on the running prod deploy, so the new guard fires. - Reproduced the crash signature from pod logs; the raise originates from the
{:error, error}branch on aReq.TransportErrortimeout.
Deploy
Marked (hotfix) so it ships straight to production, bypassing the canary + acceptance cascade, to restore the site.
🤖 Generated with Claude Code
Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.
Start a Flight and preserve its objective, outcome, and session here.
No GitHub comments yet.