Hive
fix(server): don’t block deploy on validated-on-branch backfill
GitHub issue · Closed
Hotfix for #11626. The validated-on-branch backfill blocked the production deploy past its migration-hook timeout and failed it.
What happened
#11626’s migration ran a synchronous partition-by-partition backfill scanning the raw test_case_runs fact table (5.81B rows / 9 partitions, ~8 min and 360M rows for the first partition alone). That’s a ~1.5–2 hr job; it blew past the deploy’s 60-min migration-hook window, blocked the rollout, and failed the deploy (run 28616040900). New pods never rolled, so the fix isn’t live.
Fix
Create the marker table + MV only (sub-second). The MV forward-fills every new run, so the table converges as tests run on the default branch — and the CPU win (the validation query reading the marker table instead of scanning raw runs) is independent of the historical backfill. Historical rows get backfilled out-of-band after deploy, off the critical path.
The migration never applied (it timed out mid-backfill, so it’s not recorded in schema_migrations), so editing it re-runs cleanly: CREATE TABLE IF NOT EXISTS keeps the partial rows, CREATE MATERIALIZED VIEW IF NOT EXISTS starts forward-fill.
Correctness note
Until a test case runs again on the default branch it may be treated as “unvalidated”, which only makes automated flaky-quarantine more conservative (skips action), never less. The gap self-heals as the default branch runs, and closes fully once the out-of-band backfill lands.
Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.
Start a Flight and preserve its objective, outcome, and session here.
No GitHub comments yet.