Hive
fix(server): stop reporting expected webhook delivery failures to Sentry
GitHub issue · Closed
What changed
Tuist.SentryEventFilter.before_send/1 now drops Sentry events whose original exception is an Oban.PerformError tagged with Tuist.Webhooks.Workers.DeliveryWorker, plus a new test suite for the server-side filter.
Why
Two Sentry issues have been accumulating from webhook deliveries to a single customer endpoint:
- TUIST-14Q:
DeliveryWorker failed with {:error, %Req.TransportError{reason: :closed}}(323 events, escalating) - TUIST-10D:
DeliveryWorker failed with {:error, "HTTP 400"}(~186k events in a week)
Root cause
These are not bugs in our delivery path. The customer’s receiving server either returns HTTP 400 or closes the connection. The worker already handles this correctly: every attempt is recorded in the ClickHouse webhook_delivery_attempts table (which powers the endpoint dashboard), and Oban retries on the RFC schedule (1m/5m/30m/2h/8h/24h, 7 attempts). The only defect is that TuistCommon.ObanTelemetry reports every failed attempt to Sentry as an Oban.PerformError, so one misbehaving customer endpoint produces hundreds of thousands of non-actionable error events.
Why this solution
- Oban requires an
{:error, reason}return to drive the retry schedule, so the failure can’t be silenced at the worker without losing retries. Filtering has to happen at the Sentry reporting layer, andTuist.SentryEventFilteris the existing home for “expected, non-actionable errors”. - The filter is scoped narrowly:
Oban.PerformErroronly wraps error-tuple returns fromperform/1, which for this worker are exactly the expected delivery outcomes (non-2xx, transport errors, SSRF-blocked URLs, deleted endpoints). A genuine bug inside the worker (e.g. a JSON encoding crash) raises its own exception type, bypassesOban.PerformErrorentirely, and is still reported. - Adding
Oban.PerformErrorto the global ignore list was rejected as too broad: it would hide real failures from every other worker. The clause matches on theoban_workertag thatTuistCommon.ObanTelemetrystamps on each event, so other workers’ failures keep flowing to Sentry.
Impact
Sentry stops receiving an event per failed delivery attempt to customer-controlled endpoints. Delivery-failure visibility remains where it belongs: the per-endpoint dashboard (ClickHouse attempt rows) and the Oban PromEx metrics.
Validation
mix test test/tuist/sentry_event_filter_test.exs(5 tests: drops delivery-worker PerformErrors, keeps other workers’ PerformErrors, keeps raised exceptions from the delivery worker, existing ignore behavior unchanged)mix test test/tuist/webhooks/workers/delivery_worker_test.exs(11 tests, unchanged, all pass)mix credo lib/tuist/sentry_event_filter.exclean
Fixes TUIST-14Q Fixes TUIST-10D
🤖 Generated with Claude Code
Investigate, reproduce, or fix this item in an isolated repository. Each Flight preserves its outcome and agent session.
Start a Flight and preserve its objective, outcome, and session here.
No GitHub comments yet.