fix(infra): restore preview cleanup and add janitor

Metadata

Source

tuist/tuist #11468

Updated

Jul 5, 2026

Domains

Compute

Details

What changed

Restored the external preview sweep workflow so it checks out the repository before locked tool installation, installs the locked helm, kubectl, and jq versions, and parses namespace metadata as structured JavaScript Object Notation.
Added previewJanitor to the platform Helm chart as an in-cluster Kubernetes CronJob.
Moved the janitor cleanup logic into infra/helm/platform/files/preview-janitor.sh, rendered through a ConfigMap and mounted into the CronJob.
Enabled the janitor only in the tuist-preview platform overlay, with the preview worker taint toleration and a dedicated service account.
Added the preview platform overlay to Helm chart validation so the janitor resources are rendered in pull request checks.
Updated preview operations documentation to make the in-cluster janitor the primary cleanup loop and keep preview-sweep.yml as an external backstop.

Why

Preview cleanup should not depend only on a scheduled GitHub workflow. When that workflow broke during tool installation, expired preview namespaces stayed live and kept preview pods running. The cluster already has the expiration labels and the lifecycle resources, so the recurring cleanup loop belongs close to the resources it manages.

Keeping the cleanup script outside the CronJob manifest also makes it easier to review and test. The Helm template now describes Kubernetes resources, while the shell file contains the cleanup behavior.

Root cause

The immediate failure was in the external sweep workflow: locked mise installation ran without a checked-out repository, so the lockfile was missing. After that was fixed, the namespace parser had another bug where empty release labels shifted tab-separated fields and caused expired commit previews to be skipped.

The broader operational gap was that the only recurring cleanup loop lived outside the cluster. Any runner, workflow, or tool-installation failure could stop cleanup entirely.

Approach

The external sweep workflow is still fixed and remains scheduled as a backstop. The platform chart now also installs preview-janitor into the preview cluster. The janitor runs hourly, lists namespaces labeled tuist.dev/preview=true, and deletes namespaces whose tuist.dev/expires-at label is in the past. It also tears down stale state=creating namespaces after the same grace period the workflow used.

For each expired preview, the janitor deletes the matching KuraInstance, runs helm uninstall, and then deletes the namespace as the final backstop. Its permissions are disabled by default and enabled only in values-tuist-preview.yaml, because the job needs cluster-wide delete access for dynamic preview namespaces.

Impact

Expired preview pods should be cleaned up even if the scheduled GitHub workflow is unavailable. The GitHub workflow still provides an external force-run path while this change rolls out and gives us a second cleanup mechanism during the transition.

Validation

Parsed .github/workflows/preview-sweep.yml, .github/workflows/helm.yml, and the platform values files with Ruby’s parser.
Checked the preview sweep cleanup shell script with bash -n.
Ran shellcheck against infra/helm/platform/files/preview-janitor.sh.
Checked the standalone janitor script with sh -n.
Ran helm dependency update infra/helm/platform.
Ran helm lint infra/helm/platform -f infra/helm/platform/values-hetzner.yaml.
Ran helm lint infra/helm/platform -f infra/helm/platform/values-hetzner.yaml -f infra/helm/platform/values-tuist-preview.yaml.
Rendered the default platform overlay and verified it does not include preview-janitor.
Rendered the preview platform overlay, extracted the ConfigMap script, verified it matches infra/helm/platform/files/preview-janitor.sh, and checked it with sh -n.
Verified alpine/k8s:1.36.1 contains helm, kubectl, and jq, including under a read-only root filesystem with writable /tmp.
Exercised the mounted janitor script in the alpine/k8s:1.36.1 container with stubbed cluster commands, confirming it deletes expired and stale-creating previews while leaving live and malformed previews alone.
Ran kubeconform against the rendered preview platform manifests: 155 resources found in 1 file - Valid: 117, Invalid: 0, Errors: 0, Skipped: 38.
Ran git diff --check.
Ran preview-sweep.yml from this branch: https://github.com/tuist/tuist/actions/runs/28086706079
- Deleted preview-ondemand-tuist-testing-3.
Ran preview-sweep.yml again after the parser fix: https://github.com/tuist/tuist/actions/runs/28086860601
- Deleted preview-sha-318e74b and preview-sha-d28f4be.
Ran a final verification sweep: https://github.com/tuist/tuist/actions/runs/28087005742
- Reported No preview namespaces found.

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.

No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments

F

fortmarek Jun 25, 2026

Review finding:

[P2] infra/helm/platform/templates/preview-janitor.yaml grants the janitor ServiceAccount get/list/watch/delete on core secrets through a ClusterRole. Because that ClusterRole is bound cluster-wide, the CronJob can read Secrets in every namespace, not just the dynamic preview namespace it is cleaning. That means a compromised janitor pod or image could read platform/bootstrap Secrets as well as preview Helm release Secrets.

The namespace deletion backstop and Kura cleanup do not need Secret read access; only helm uninstall does, and only inside the preview namespace. Could we move the namespaced Helm permissions into per-preview RoleBindings created by preview-deploy.yml, or have this in-cluster janitor skip helm uninstall and leave that part to the external workflow? That keeps cleanup working without giving the janitor cluster-wide Secret read.