Background Jobs & Async Workers

Move slow work off the request path: job queues, at-least-once delivery and idempotent consumers, retries with backoff + jitter, dead-letter queues, the transactional outbox, and why exactly-once is mostly a myth.

Why async work exists

Some work is too slow or too unreliable to do inside an HTTP request: sending email, transcoding video, generating a report, calling a flaky third party. Block the request on it and you tie up a connection, hit timeouts, and lose the work if the call fails. The fix is to enqueue a job and return immediately; a pool of workers processes the queue out-of-band.

request ──▶ API enqueues job ──▶ [ queue ] ──▶ worker pool ──▶ side effect
            (returns 202)         (Redis/SQS/         (retry, ack)
                                   RabbitMQ/Kafka)

Where this lands by level

SDE-1: push work to a queue and process it in a worker. SDE-2: at-least-once delivery, idempotency, retries/backoff, and a DLQ. SDE-3: the transactional outbox, ordering guarantees, and the exactly-once discussion. Core at every backend-heavy shop — Amazon, Uber, Stripe, DoorDash — and a frequent follow-up to any HLD write path.

The delivery guarantee you actually get

Almost every queue gives at-least-once delivery: a message can be delivered more than once (a worker crashes after doing the work but before ack'ing; the broker redelivers). So the rule that governs everything else:

Make consumers idempotent. Processing the same message twice must equal processing it once.

async function handle(job: { id: string; userId: string; amountCents: number }) {
  // Idempotency key = job.id. If we've seen it, no-op.
  const fresh = await store.markProcessedIfNew(job.id);
  if (!fresh) return;                 // duplicate delivery → safe no-op

  await chargeCard(job.userId, job.amountCents);
}

Retries, backoff, and the dead-letter queue

Retry transient failures (network blips, 503s) — but with exponential backoff + jitter, or a thundering retry herd hammers the failing dependency in lockstep.
Cap the attempts. After N failures, move the message to a dead-letter queue (DLQ) instead of retrying forever. The DLQ is your "broken jobs" inbox — alert on it.
Visibility timeout: while a worker holds a message it's hidden from others; if the worker dies, it reappears for redelivery. Size it longer than the job's worst-case runtime, or you get duplicate concurrent processing.

The transactional outbox

The classic bug: you write to the DB and enqueue a job as two separate operations — the process crashes between them, and now your data and your queue disagree (charge recorded, receipt email never queued — or vice versa).

The outbox pattern fixes it: in the same DB transaction as your business write, insert a row into an outbox table. A separate relay (or CDC on the table) reads committed outbox rows and publishes them to the queue. The DB transaction makes the write and the "intent to publish" atomic; the relay makes publishing at-least-once.

Exactly-once is (mostly) a myth

You can't truly guarantee a side effect happens exactly once across crashes — networks drop acks. What you can build is at-least-once delivery + idempotent processing, which is effectively exactly-once from the outside. If someone says "exactly-once," ask where the idempotency key lives.

Scheduling

Cron / scheduled jobs: time-triggered (nightly reports). Beware running the same cron on N app servers — use a leader lock (see distributed-locks).
Delayed jobs: "run this in 1 hour" — delayed queues or a scheduled-time field the workers poll. (For the streaming flavor, see messaging-and-queues.)