Building background jobs that fail well

Jobs that leave the request-response cycle make an application feel fast, but they also create a new set of promises. A user may no longer be waiting for the work to finish, yet the system still has to finish it—or explain clearly why it could not.

Start with the delivery contract

“Run this later” is not a useful contract. Decide whether a job may run once, at least once, or at most once. Most practical queues provide at-least-once delivery, which means every handler must be safe to execute more than once.

const previous = await jobs.findByIdempotencyKey(key)
if (previous?.completedAt) return previous.result

return processJob(input)

The idempotency key is not an implementation detail. It is part of the boundary between the producer and the worker.

Retries need a budget

Retrying immediately can turn one failing dependency into a system-wide incident. Exponential backoff creates space for recovery, while jitter prevents thousands of workers from retrying on the same schedule.

Permanent failures should move to a dead-letter queue with enough context to investigate and replay them safely. A dead-letter queue is useful only when somebody can understand what entered it.

Make progress observable

Track the transition between queued, running, retrying, completed, and failed states. Emit structured events at those boundaries and carry the same correlation identifier through the API, broker, and worker.

Reliability comes less from avoiding failure than from making every failure bounded, visible, and recoverable.