Hive Hive
Sign in

fix(server): speed up xcode cache widgets

GitHub issue · Closed

Metadata
Source
tuist/tuist #11170
Updated
Jun 24, 2026
Domains
Kura
Details

What changed

  • Denormalized project_id onto ClickHouse cas_outputs and populated it for new CAS output writes, seeds, and test fixtures.
  • Added an in-place ClickHouse migration that adds project_id, backfills existing rows with a dictionary-backed ALTER TABLE ... UPDATE, and materializes a project-ordered projection.
  • Added proj_by_project_operation_inserted_at ordered by (project_id, operation, inserted_at) so project-scoped CAS analytics get a prunable access path without replacing the hot table.
  • Reworked CAS widget analytics to aggregate download/upload metrics from cas_outputs in one project-scoped query instead of issuing separate queries for each widget.
  • Updated the Xcode cache LiveView to reuse the combined CAS analytics result and run the remaining hit-rate analytics concurrently.
  • Updated server/data-export.md for the new stored analytics field.

Why

The /tuist/tuist/xcode-cache page mounted quickly, but its widgets spent 20+ seconds loading because each CAS widget queried cas_outputs separately and joined through build_runs to filter by project. Since project_id only lived on build_runs, ClickHouse could not prune cas_outputs before the join and each widget scanned the full table.

Root cause

The CAS analytics query shape filtered by br.project_id after joining from cas_outputs, so the most selective project predicate was unavailable to the cas_outputs table scan. The LiveView then amplified the cost by issuing independent CAS widget queries for transfer, latency, and throughput.

Why this migration shape

A shadow-table rewrite plus EXCHANGE TABLES is unsafe for a hot ingest table unless CAS output writes are paused: rows can be written while the copy is running, and a row-count check only detects some races after the expensive work has already happened. This PR therefore keeps cas_outputs as the canonical table and uses an in-place ClickHouse mutation for the backfill.

The migration still does real historical backfill work. It creates a temporary dictionary over build_runs, updates project_id on existing rows where it is still 0, then materializes a projection ordered by (project_id, operation, inserted_at). New application writes include project_id directly, so rows written by the updated code do not depend on the dictionary.

Impact

Project-scoped CAS analytics can use denormalized project_id for historical rows once the mutation completes and for new rows immediately after the application writes them. The dashboard also avoids redundant CAS queries by deriving transfer, latency, and throughput widgets from one CAS analytics result while fetching hit-rate analytics concurrently.

Validation

  • MIX_ENV=test mix ecto.reset
  • mix test test/tuist/builds/cas_output_test.exs test/tuist/builds/analytics_test.exs:2780 test/tuist/builds_test.exs:1519 test/tuist_web/live/xcode_cache_live_test.exs
  • git diff --check

Note: local mix deps.get currently wants unrelated patch bumps for cowboy, cowlib, and deep_merge; those lockfile changes were not included in this PR.

Comments
TA
tuist-atlas[bot] Jun 10, 2026

The Xcode cache widgets performance improvements are now available in version xcresult-processor-image@0.14.1. To use these optimizations, update to this version.