Resolves N/A
Add account-scoped object storage retention jobs so Tuist can reduce S3/Tigris usage without changing analytics metadata retention.
This PR adds application-level retention for tenant-owned binary artifacts in object storage:
- Adds a plan-aware retention policy for previews, build archives, test attachments, shard bundles, and cache artifacts.
- Adds a hosted-only daily scheduler that paginates accounts with follow-up scheduler jobs so retries resume at page boundaries.
- Adds one idempotent deletion worker per DB-backed artifact family: previews, build archives, test attachments, and shard bundles.
- Adds server-owned cache S3 retention workers for Xcode cache, Xcode module cache, and Gradle cache objects. These list S3 as the source of truth and delete only expired objects matching cache key shapes. Registry artifacts are excluded.
- Keeps DB-backed artifact selection bounded by small batches per account.
- Adds PostgreSQL/ClickHouse indexes and projections for the DB-backed retention access pattern, so workers avoid broad table scans as usage grows.
- Uses S3 bulk deletion in parallel 1000-key chunks with bounded concurrency and explicit S3 response checks.
Buckets and artifacts
| Bucket |
Configuration |
Artifacts stored |
Object key families |
Retention source |
| Tuist application bucket, or the account custom S3 bucket when fully configured |
Global bucket from TUIST_S3_BUCKET_NAME / secrets through Environment.s3_bucket_name/0; custom account bucket through Account.s3_bucket_name |
App preview bundles, preview icons, build archives, test run attachments, shard bundles |
<account>/<project>/previews/<app_build_id>.zip, <account>/<project>/previews/<app_build_id>.apk, <account>/<project>/previews/<preview_id>/icon.png, <account>/<project>/builds/<build_id>/build.zip, <account>/<project>/tests/runs/<test_run_id>/attachments/<attachment_id>/<file_name>, legacy <account>/<project>/tests/test-case-runs/<test_case_run_id>/attachments/<attachment_id>/<file_name>, <account_id>/<project_id>/shards/<shard_plan_id>/bundle.zip |
Database inserted_at for the matching preview, build, test attachment, or shard plan row |
| Xcode cache bucket |
TUIST_CACHE_XCODE_S3_BUCKET_NAME, S3_XCODE_CACHE_BUCKET, or cache secrets; falls back to the shared cache bucket when no dedicated Xcode bucket is configured |
Xcode compilation cache artifacts |
<account>/<project>/xcode/... |
S3 last_modified |
| Shared cache bucket |
TUIST_CACHE_S3_BUCKET_NAME, S3_BUCKET, or cache secrets |
Xcode module cache artifacts and Gradle cache artifacts |
<account>/<project>/module/..., <account>/<project>/gradle/... |
S3 last_modified |
Registry artifacts are not matched by the cache cleanup key filters and are not deleted by these workers.
Retention policy
The active or trialing account subscription determines the plan. Accounts without an active subscription fall back to Air. Air and Open Source share the same windows.
| Artifact family |
Air / Open Source |
Pro |
Enterprise |
| Cache artifacts, including Xcode compilation cache, Xcode module cache, and Gradle cache |
14 days |
30 days |
90 days |
| App preview bundles and preview icons |
60 days |
180 days |
365 days |
| Build archives |
30 days |
90 days |
365 days |
| Test run attachments |
30 days |
90 days |
365 days |
| Shard bundles |
7 days |
14 days |
30 days |
Cleanup strategy
DB-backed artifacts are cleaned from the hosted-only Oban cron at 02:30 daily. ScheduleExpiredArtifactsWorker pages accounts by id with a default page size of 500, bulk inserts per-account deletion jobs, and schedules a continuation job when another account page remains. Each per-account worker selects one keyset-paginated batch ordered by (inserted_at, id), computes object keys, deletes the blobs, and self-enqueues the next cursor when the batch was full. This lets a run walk the backlog without repeatedly selecting the oldest expired rows.
Cache artifacts are cleaned by hosted-only Oban crons at 03:00 for Xcode cache, 03:15 for Xcode module cache, and 03:30 for Gradle cache. Each worker lists one S3 page, filters by the expected key shape, batch-loads the matching account plans for that page, compares the object last_modified timestamp to the plan cutoff, deletes expired keys, and self-enqueues the next S3 continuation token when the bucket page is truncated. Objects whose account handle no longer resolves to a known account are skipped instead of defaulting to the shortest plan window.
Cleanup deletes only binary blobs from object storage. Metadata rows such as previews, build runs, test runs, test attachments, and shard plans remain available for analytics, dashboards, and data exports. Tuist does not persist a per-artifact purge ledger; retention status is derived from the artifact timestamp and the current account plan at cleanup time.
All object deletion uses S3 multi-object deletion in chunks of 1000 keys with bounded concurrency. Deletion errors are returned to Oban so failed jobs retry instead of silently succeeding.
How to test locally
mix format lib/tuist/environment.ex lib/tuist/storage.ex lib/tuist/shards.ex lib/tuist/oban/runtime_config.ex lib/tuist/storage/retention_policy.ex lib/tuist/storage/expired_artifacts.ex lib/tuist/storage/cache_artifact_retention.ex lib/tuist/storage/workers/delete_expired_build_archives_worker.ex lib/tuist/storage/workers/delete_expired_preview_artifacts_worker.ex lib/tuist/storage/workers/delete_expired_shard_bundles_worker.ex lib/tuist/storage/workers/delete_expired_test_attachments_worker.ex lib/tuist/storage/workers/delete_expired_xcode_cache_artifacts_worker.ex lib/tuist/storage/workers/delete_expired_xcode_module_cache_artifacts_worker.ex lib/tuist/storage/workers/delete_expired_gradle_cache_artifacts_worker.ex lib/tuist/storage/workers/schedule_expired_artifacts_worker.ex priv/repo/migrations/20260529100000_add_artifact_retention_indexes.exs priv/ingest_repo/migrations/20260529100000_add_artifact_retention_projections.exs test/test_helper.exs test/tuist/storage_test.exs test/tuist/oban/runtime_config_test.exs test/tuist/storage/retention_policy_test.exs test/tuist/storage/cache_artifact_retention_test.exs test/tuist/storage/workers/delete_expired_artifact_workers_test.exs test/tuist/storage/workers/delete_expired_cache_artifact_workers_test.exs test/tuist/storage/workers/schedule_expired_artifacts_worker_test.exs
mix compile --warnings-as-errors
mix test test/tuist/storage/cache_artifact_retention_test.exs test/tuist/storage/retention_policy_test.exs
mise run security
aube install --frozen-lockfile
mix test test/tuist/storage/workers/delete_expired_cache_artifact_workers_test.exs test/tuist/storage/workers/delete_expired_artifact_workers_test.exs test/tuist/storage/workers/schedule_expired_artifacts_worker_test.exs test/tuist/storage/cache_artifact_retention_test.exs was attempted earlier, but the local test DB failed before tests ran with ERROR 42P01 (undefined_table) relation "users" does not exist.