Design a Payment Retry System

Why this is really a correctness problem

A payment call crosses a network that lies: it can succeed while the response is lost, fail cleanly, or hang ambiguously. Naive retries then either double-charge the customer or lose the payment. So a "retry system" is not a for loop with sleep — it's a small state machine wrapped around two guarantees: every attempt is idempotent, and retries back off without stampeding. Get those two right and the rest is bookkeeping.

Clarify scope first: "One payment, many attempts against a flaky gateway. I'll design the per-payment retry lifecycle, idempotency, the backoff policy, and where exhausted payments go (DLQ). Out of scope: the ledger and reconciliation, though I'll note where they hook in — OK?"

The lifecycle is a state machine

Model the attempt explicitly so illegal moves are unrepresentable: you can't "retry" a Done payment, and a non-retryable rejection skips straight to the dead-letter state. Walk a transient failure that recovers, then try an illegal event and watch the machine refuse it:

• Pending → Sending (send): claim the work, attach the idempotency key.
• Sending → Done (ok): the gateway confirmed success.
• Sending → Backoff (fail): a transient error (timeout, 5xx, network) — schedule a retry after a delay.
• Backoff → Sending (retry): the delay elapsed; try again with the same idempotency key.
• Sending → Dead (reject): a permanent error (4xx like "card declined") — retrying can never help, so don't.
• Backoff → Dead (exhaust): max attempts reached → dead-letter for a human or a separate recovery flow.

The two hard parts

1. Idempotency across retries

Because delivery is at-least-once (you retry on ambiguity), the effect must be at-most-once. Every payment carries a unique idempotency key, generated once and reused on every retry. The gateway's idempotent endpoint either applies the charge once or returns the original result — so a retry after an ambiguous timeout is safe (the same Stripe pattern as the ATM and REST idempotency). Locally you journal the key before sending, so a crash mid-flight still knows the attempt is in flight.

2. Exponential backoff with jitter

Retrying immediately hammers an already-struggling gateway. Back off exponentially — base * 2^attempt — capped at a ceiling. Then add jitter (randomness): without it, a thousand payments that failed in the same outage all retry at the exact same instant and re-create the spike — a thundering herd. Full jitter: sleep = random(0, min(cap, base * 2^attempt)).

Entities & responsibilities

• RetryEngine is the orchestrator — it owns no policy detail; it asks the injected RetryPolicy and drives the state machine.
• RetryPolicy (Strategy) decides delay, retryability and the attempt cap.
• PaymentGateway (port) is the boundary to the outside world; a StripeGateway/MockGateway adapter implements it (and tests use the mock).
• IdempotencyStore makes retries safe; Scheduler defers a Backoff attempt without blocking a thread; DeadLetterQueue catches the exhausted.

Think it through like the interview

Practice — level up

A payment retry engine is idempotent at-least-once delivery wrapped in a state machine. These drills train the pieces.