Hive Hive
Sign in

fix(server): stop reporting expected webhook delivery failures to Sentry

GitHub issue · Closed

Metadata
Source
tuist/tuist #11631
Updated
Jul 5, 2026
Domains
Atlas
Details

What changed

Tuist.SentryEventFilter.before_send/1 now drops Sentry events whose original exception is an Oban.PerformError tagged with Tuist.Webhooks.Workers.DeliveryWorker, plus a new test suite for the server-side filter.

Why

Two Sentry issues have been accumulating from webhook deliveries to a single customer endpoint:

  • TUIST-14Q: DeliveryWorker failed with {:error, %Req.TransportError{reason: :closed}} (323 events, escalating)
  • TUIST-10D: DeliveryWorker failed with {:error, "HTTP 400"} (~186k events in a week)

Root cause

These are not bugs in our delivery path. The customer’s receiving server either returns HTTP 400 or closes the connection. The worker already handles this correctly: every attempt is recorded in the ClickHouse webhook_delivery_attempts table (which powers the endpoint dashboard), and Oban retries on the RFC schedule (1m/5m/30m/2h/8h/24h, 7 attempts). The only defect is that TuistCommon.ObanTelemetry reports every failed attempt to Sentry as an Oban.PerformError, so one misbehaving customer endpoint produces hundreds of thousands of non-actionable error events.

Why this solution

  • Oban requires an {:error, reason} return to drive the retry schedule, so the failure can’t be silenced at the worker without losing retries. Filtering has to happen at the Sentry reporting layer, and Tuist.SentryEventFilter is the existing home for “expected, non-actionable errors”.
  • The filter is scoped narrowly: Oban.PerformError only wraps error-tuple returns from perform/1, which for this worker are exactly the expected delivery outcomes (non-2xx, transport errors, SSRF-blocked URLs, deleted endpoints). A genuine bug inside the worker (e.g. a JSON encoding crash) raises its own exception type, bypasses Oban.PerformError entirely, and is still reported.
  • Adding Oban.PerformError to the global ignore list was rejected as too broad: it would hide real failures from every other worker. The clause matches on the oban_worker tag that TuistCommon.ObanTelemetry stamps on each event, so other workers’ failures keep flowing to Sentry.

Impact

Sentry stops receiving an event per failed delivery attempt to customer-controlled endpoints. Delivery-failure visibility remains where it belongs: the per-endpoint dashboard (ClickHouse attempt rows) and the Oban PromEx metrics.

Validation

  • mix test test/tuist/sentry_event_filter_test.exs (5 tests: drops delivery-worker PerformErrors, keeps other workers’ PerformErrors, keeps raised exceptions from the delivery worker, existing ignore behavior unchanged)
  • mix test test/tuist/webhooks/workers/delivery_worker_test.exs (11 tests, unchanged, all pass)
  • mix credo lib/tuist/sentry_event_filter.ex clean

Fixes TUIST-14Q Fixes TUIST-10D

🤖 Generated with Claude Code

Flights

Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.

New Flights are paused Configure model inference, GitHub, and a sandbox provider to start another Flight. Existing results remain available below.
No Flights yet

Start a Flight and preserve its objective, outcome, and session here.

Comments

No GitHub comments yet.