fix(oban): tolerate transient peer connection failures

What changed

Hive now configures Oban to use a local database peer implementation that preserves the peer loop when a transient database connection failure exhausts the election transaction. The peer keeps the existing database-backed leadership behavior, but logs exhausted transactions and schedules the next election instead of crashing.

Production database connections also enable client socket keepalives. On Linux, Hive now sets the same short keepalive cadence on the application socket that it already requested from Postgres through startup parameters.

Why

Sentry issue HIVE-G showed DBConnection.ConnectionError: ssl send: closed coming from the Oban peer election path. That is operational noise, but it also means the process responsible for database-backed Oban leadership can exit after a transient connection close.

Root cause

The peer election runs inside an Oban repository transaction. When the underlying encrypted database socket is already closed, the transaction can exhaust its retry path and surface through rollback handling. Oban then raises out of the peer process instead of treating the failed election as temporary.

Hive already had Postgres-side keepalive parameters, which help the database notice abandoned clients. It did not also enable client socket keepalives, so a long-idle application-side connection could stay in the pool until the next checkout discovered that the socket was dead.

Approach

The new Hive.Oban.Peers.Database keeps Oban’s database peer semantics, including the oban_peers table, leader notifications, election scheduling, and the built-in peer’s retry: 1 transaction budget. The difference is that election and termination transactions also use on_exhausted: :log.

When an election transaction still exhausts, the peer now drops local leadership before scheduling the next election. That avoids a stale leader continuing to run leader-only plugins after another replica has already claimed the expired peer row.

This custom peer is intentionally temporary. It mirrors Oban.Peers.Database from Oban 2.23 with the smallest local difference needed for this failure path. We should remove it and return to the built-in Oban peer once Oban handles exhausted database peer election retries without terminating the peer process.

The runtime database config enables keepalive: true for production sockets and adds Linux raw socket options for a shorter probe cadence. Non-Linux environments keep the portable keepalive option without the Linux-specific raw values.

Impact

There is no migration and no operator action required. Production should report fewer dead idle database socket errors, and a remaining transient close should no longer terminate the Oban peer.

A replica that cannot verify leadership temporarily stops acting as leader. That can delay leader-only work until the next successful election, but it avoids duplicate leaders. The local peer still creates maintenance risk because it mirrors Oban internals. The removal condition is explicit in the module documentation so future Oban upgrades have a clear cleanup path.

Validation

• mix format --check-formatted config/runtime.exs config/config.exs lib/hive/oban/peers/database.ex test/hive/oban/peers/database_test.exs
• mix compile --warnings-as-errors
• mix test test/hive/oban/peers/database_test.exs
• Production-config smoke check for Hive.Repo socket options
• mix test
• mix credo
• Follow-up documentation check: mix format --check-formatted lib/hive/oban/peers/database.ex
• Follow-up documentation check: mix compile --warnings-as-errors
• Follow-up retry-budget check: mix format --check-formatted lib/hive/oban/peers/database.ex test/hive/oban/peers/database_test.exs
• Follow-up retry-budget check: mix test test/hive/oban/peers/database_test.exs
• Follow-up retry-budget check: mix compile --warnings-as-errors
• Stale-leadership fix: mix format --check-formatted lib/hive/oban/peers/database.ex test/hive/oban/peers/database_test.exs
• Stale-leadership fix: mix test test/hive/oban/peers/database_test.exs
• Stale-leadership fix: mix compile --warnings-as-errors
• Stale-leadership fix: mix test
• Stale-leadership fix: mix credo