fix(server): count each run once in rolling flaky/reliability alert evaluation

fix(server): de-duplicate re-inserted runs in rolling flaky/reliability alert evaluation

What changed

The rolling-window path of the flaky-test automations (FlakyTestsMonitor) now collapses the per-test-case recent-runs aggregate to one entry per run before computing a rate. Previously it summed raw array entries and divided by a fixed slot count, so a run that appears more than once in the aggregate was counted more than once.

Applies to all three rolling monitors: flakiness_rate, flaky_run_count, and reliability_rate, across both the bucketed MV fast path (test_case_runs_recent_{100,250,500,750}_per_case) and the 1000-entry fallback (test_case_runs_recent_per_case).

Why

test_case_runs is a ReplacingMergeTree, and flaky detection re-inserts a run to set is_flaky after ingestion (the commit-based and hash-based detectors both do this). The recent-runs materialized views record every physical insert, so a single logical run lands in the rolling aggregate 2-4 times. Crucially the duplicates are concentrated on flaky/failed runs (a passing run is never re-marked), so they:

• inflate flakiness_rate / flaky_run_count (numerator over-counts flaky runs), and
• deflate reliability_rate (the fixed-size denominator stays while extra failure entries fill the window),

for exactly the runs a threshold reacts to. A pre-existing code comment acknowledged the duplication but assumed it was “bounded noise ≤1%”; that is wrong because the noise is not uniform, it is the signal.

Observed impact

On a customer project (reliability_rate < 97% and flakiness_rate ≥ 2%, both rolling-75), a burst on 2026-07-02 auto-skipped 615 distinct test cases and muted others that were genuinely healthy. Verified examples:

• A 100%-reliable test (8 failures in 15,195 runs, all old) was skipped: its 8 recent flaky failures appeared 24 times in recent_successful_runs, dragging the windowed reliability below threshold.
• A 0.1%-flaky test was muted: at mute time its true last-75 window held exactly 1 flaky run (1.33%), but the run was counted 3× → 4%, over the 2% threshold.

The fix

rolling_triggered_test_case_ids_from_recent_runs now:

1. ARRAY JOINs the recent-runs array,
2. groups by (test_case_id, run_key) where run_key is the run’s ran_at (toUnixTimestamp64Micro(ran_at) for the big table, -tupleElement(entry,1) for the negated-key buckets), keeping max(flag) so a run that was ever marked flaky / ever succeeded is represented once with the correct flag,
3. takes the latest size distinct runs via ORDER BY run_key DESC LIMIT size BY test_case_id,
4. computes the rate as sum(flag) * 100.0 / count() (or sum(flag) for flaky_run_count) over those distinct runs.

This is a read-time correction; it does not require re-backfilling the aggregates or changing the MV schema, and it keeps the same bounded per-test-case scan (no full test_case_runs walk).

Validated against production data

The exact generated query for both code paths was run read-only against the production aggregates: the two wrongly-skipped tests now evaluate to 100% reliability over 75 distinct runs, and the wrongly-muted test to 0% flakiness, matching their dashboard rates.

Tests

TDD: added three tests to flaky_tests_monitor_test.exs that re-insert a single run the way flaky detection does (same id + ran_at, inserted multiple times) and assert:

• flakiness_rate does not mute a healthy test (20% real vs 60% double-counted),
• de-duplication still keeps the flaky mark, so a genuinely flaky run is counted once (fires at a 15% threshold),
• reliability_rate does not skip a healthy test (90% real vs 70% double-counted).

De-dup headroom (bucket selection)

De-dup collapses re-inserted runs after the bounded aggregate is merged, so a window equal to its source bucket (e.g. the default rolling_window_size = 100, which reads the size-100 bucket) could yield fewer than size distinct runs once duplicates consume slots. The duplicates are ongoing (the live MV re-fires on every is_flaky re-mark, ~1.4 copies per flaky run — verified on same-day production data), so this does not age out.

Fix: recent_runs_source now picks a bucket strictly larger than the window (size < bucket), guaranteeing headroom. This is a no-op for every rolling alert in use today (all size = 75, already reading the 100 bucket); only windows ≥ 100 read one tier up, and only windows ≥ 750 reach the 1000-entry aggregate. Added a boundary test that would collapse to 100% flaky under a size-equal bucket and reads correctly (~1%) with the larger one.

Run identity note: de-dup keys on ran_at (microsecond), not the run id, which the aggregates do not carry. Two distinct executions sharing a ran_at would merge; measured 0 such collisions across busy test cases (incl. a parallel-testing one). Carrying id would need re-backfilling the aggregates and is deferred unless collisions appear.

Scope / follow-up

The last_days (calendar-window) path aggregates daily run_count / successful_run_count / flaky_run_count, which are also incremented by re-inserts. The effect there is milder (the denominator grows too) and out of scope for this fix; tracked as a follow-up. A separate cleanup could also de-duplicate at write time (dedupe the flaky re-mark before it reaches the MVs) so the stored aggregates stay clean.

No GitHub comments yet.