fix(oban): rescue orphaned executing jobs

What changed

This adds Oban.Plugins.Lifeline to the application Oban configuration with a 30 minute rescue window.

It also adds a focused configuration test that asserts the Lifeline plugin stays enabled with that rescue window.

Why

The hive-domains-evolution-worker Sentry monitor had been missing check-ins since June 24. Production had one Hive.Domains.EvolutionWorker row stuck in executing, owned by a pod that no longer existed.

Root cause

Hive.Domains.EvolutionWorker is intentionally configured as a singleton through Oban uniqueness over incomplete jobs. When an old node left job 92445 in executing, every future scheduled run conflicted with that row and no fresh cron job could be inserted.

Approach

Oban.Plugins.Lifeline is the built-in recovery path for jobs left in executing after a node disappears. A 30 minute rescue window is longer than the recent normal domain evolution runtime, while still short enough to avoid a multi-day monitor outage.

The production row was manually cancelled during the incident response, and the next scheduled run inserted successfully at 09:30 Coordinated Universal Time. Sentry now receives check-ins again. The worker is currently failing for a separate Fireworks account suspension, but it is no longer missing scheduled check-ins.

Impact

Future orphaned executing jobs are rescued automatically instead of blocking singleton workers indefinitely. The normal singleton behavior for active jobs remains unchanged.

Validation

• mix test test/hive/oban_config_test.exs
• mix precommit, 797 tests passed and Credo found no issues
• Verified production after cancelling job 92445: there are zero incomplete Hive.Domains.EvolutionWorker rows, and Sentry recorded the 09:30 production check-in.

No GitHub comments yet.