Hive Hive
Sign in

fix(server): relay test-run completion broadcast via Oban so dashboards refresh

GitHub issue · Closed

Metadata
Source
tuist/tuist #11526
Updated
Jul 5, 2026
Domains
Testing
Details

Purpose

A finished test run could stay stuck on the “processing” spinner in the web UI even though ClickHouse already showed status=success — the run looked perpetually unprocessed until the user manually refreshed the page.

Root cause

The test-run detail LiveView (TestRunLive) shows the spinner purely off @run.status == "processing" and only clears it when it receives a {:test_created} PubSub message — there is no polling. That broadcast is emitted from inside Tests.create_test/1, which for most runs executes on the xcresult-processor. The processor runs as an isolated, non-clustered BEAM node (no TUIST_CLUSTER_DNS_SERVICE / RELEASE_DISTRIBUTION, on the Tart NAT network, not in the tuist-tuist-server-headless endpoints), and Tuist.PubSub uses the default PG2 adapter, which only delivers across Erlang-clustered nodes. So the completion broadcast was emitted on a node that can’t reach the web tier and the LiveView never updated.

What changed

1. Relay the broadcast through Oban (Tuist.Tests.Workers.BroadcastTestCreatedWorker). Tests.create_test/1 now enqueues the broadcast on the :default queue instead of broadcasting in-process. The :default queue is consumed only by web pods (the build- and xcresult-processor fleets each run a single dedicated queue), so the job always runs on a clustered web node and the PG2 broadcast reaches every subscribed LiveView. The payload (%Test{}) and topic ("<account>/<project>") are unchanged, so all subscribers behave exactly as before — only the node that emits the broadcast moves.

2. Poll fallback in TestRunLive. While a run is processing, the LiveView re-checks every 5s and self-heals even if the broadcast is ever lost (cluster/Redis blip, crash, a client that missed the message). It stops polling once the run reaches a terminal status.

Why this approach

The processor is NAT-isolated by design and reaches Postgres/Oban over Tailscale; it shares the Oban/Postgres backend with the web tier but cannot reach the in-cluster tuist-tuist-redis ClusterIP. Relaying through Oban reuses infrastructure the processor already depends on — no new network exposure, no Tailscale ACL change, and no need to switch the global PubSub transport to Redis (which would otherwise be required to bridge the node boundary). The poll fallback is defence-in-depth and also helps clients whose WebSocket reconnects.

Impact

  • Test-run dashboards update on their own when a run finishes processing on the external processor (the common case), instead of appearing stuck until a manual refresh.
  • No behavioural change for any PubSub subscriber — the broadcast payload/topic are identical.
  • One extra lightweight Oban job (:default queue) per test-run creation.

How to test locally

cd server
mix test test/tuist/tests/workers/broadcast_test_created_worker_test.exs \
test/tuist_web/live/test_run_live_test.exs
  • The worker test asserts the broadcast lands on the project topic and is a no-op for a missing run.
  • The TestRunLive test asserts a processing run clears its spinner via the poll even when no broadcast is delivered.

Notes for reviewers

  • An unrelated plug_cowboy 2.8.1 → 2.9.0 bump was present in server/mix.lock in the worktree and is intentionally not included in this PR.
  • Follow-up: build runs likely share the same cross-node isolation (the build processor TUIST_MODE=processor is also non-clustered); if there is an analogous completion broadcast it would want the same relay.
Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments
P
pepicrft Jun 29, 2026

I think the durable fix is to make run completion a persisted-state problem first, and treat the push notification as a best-effort accelerator.

I would avoid clustering the macOS processor nodes with the web nodes for this. It would couple a specialized parsing fleet to the live web cluster for the sake of a dashboard refresh, and it would bring extra operational surface: Erlang distribution credentials, node discovery, network reachability, and remote process messaging across two very different runtime roles. That feels like the wrong boundary.

My preferred shape would be:

  • Keep processor nodes isolated.
  • Make TestRunLive poll while the run is in any transient state, both processing and in_progress.
  • Stop polling only after a terminal state is visible, ideally with one delayed follow-up refresh so ClickHouse buffered rows have time to land.
  • Keep Phoenix Publish-Subscribe (https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html) as an optimization when the event originates on the web cluster.
  • If we still want push from the processor side, use an explicit web-side notifier or outbox instead of a one-off broadcast relay hidden inside Tests.create_test/1.

There are also two concrete issues in this implementation:

  • The relayed payload is a %Tuist.Tests.Test{} struct, but several aggregate dashboards match {:test_created, %{name: "test"}}, so they still ignore the event.
  • Sharded runs can wait in in_progress, but the detail page only subscribes and polls for processing, so those pages can still stay stale.

So I would remove the Oban relay from this path and make the LiveViews resilient to missed notifications. That keeps the processor boundary clean and makes the dashboard correct even when messages or jobs are delayed.