All WordPress HTML Templates Forms & Webhooks AI & Tools
Forms & Webhooks

Retries and Backoff: Designing a Delivery Policy That Survives Outages

A practitioner’s guide to webhook retry policy: exponential backoff with jitter, retryable status codes, retry budgets, circuit breakers and dead letters.

A real working moment illustrating the theme of an article about webhook retry. Wide 16:9 banner, one strong focal point, magazine editorial quality, authentic and unstaged.

A customer endpoint went dark for 43 minutes last year. Nothing dramatic: their load balancer was swapping certificates and the health check flapped. By the time it came back, our queue held just over 9,000 events for that one tenant. We released them. Their newly-recovered app server fell over in about eleven seconds, and the outage that should have ended at 14:43 ended at 15:20 instead.

That was our fault, not theirs. We had exponential backoff. What we didn’t have was jitter, a concurrency cap per destination, or any notion of a retry budget. A webhook retry policy that only answers “how long do I wait before trying again?” is maybe a third of a real delivery policy.

Here’s the version we ship now, with the numbers, the edge cases and the parts that still bite.

Key Takeaways

  • Exponential backoff without jitter synchronises every queued event into a stampede the moment a consumer recovers. Use full jitter or AWS-style decorrelated jitter, and cap any single delay at 1 to 6 hours.
  • Retry on 408, 429, 5xx, TLS failures, DNS failures, connection resets and timeouts. Do not retry 400, 401, 403, 404, 410, 422. A 410 Gone should disable the subscription outright.
  • Expire by wall-clock age (24 to 72 hours is the industry norm) rather than attempt count alone, and keep a per-destination retry budget around 10 percent of successful deliveries so one dead consumer cannot starve the queue.
  • Every attempt must carry the same event ID and idempotency key plus an incrementing attempt counter, otherwise the receiver has no way to deduplicate and your at-least-once guarantee becomes their data corruption bug.
  • Honour Retry-After even when it is longer than your own schedule, re-resolve DNS between attempts, and keep request timeouts tight (3 to 5 second connect, 10 second total) so a hung consumer cannot pin your workers.

The five decisions a delivery policy actually makes

Writing the retry loop is the easy part. The policy is the set of answers to these five questions:

  1. Is this failure retryable? A 422 is not going to become a 200 in four hours.
  2. When is the next attempt? Base delay, growth factor, jitter distribution, per-attempt cap.
  3. When do we give up? Max attempts, max age, or both. Which one fires first matters more than either number.
  4. How much of our capacity may retries consume? Globally and per destination.
  5. What happens after we give up? Dead letter, auto-disable, alert, replay window.

Most teams ship answers to 2 and 3 and discover 1, 4 and 5 during an incident. The expensive failures live in 4.

A real working moment that fits an article section about

Exponential backoff is table stakes. Jitter is the part people skip

Plain exponential backoff is delay = base * factor^attempt, capped. With base 5s and factor 2 you get 5s, 10s, 20s, 40s, 1m20, 2m40, 5m20, 10m40, 21m20, 42m40, 1h25, 2h50, then a 6 hour cap. Nineteen attempts lands you at roughly 48 hours of total effort, which is about where Shopify sits before it drops a subscription, and comfortably inside Stripe’s three-day window. Those are reasonable anchors because consumers have been trained by them.

The problem is that deterministic delays preserve the structure of the incident. If 9,000 events all failed inside the same 43 minute window, their retry times stay clustered forever: every one of them is doing the same arithmetic. You get a Dirac spike at recovery.

Full jitter fixes it by throwing away the schedule and keeping only the ceiling:

const BASEMS = 5000;
const CAP_MS  = 6  60  60 * 1000; // 6h ceiling on any single delay

// Full jitter: pick uniformly from [0, exponential ceiling].
// Spreads a recovered backlog across the whole window instead of one spike.
function fullJitter(attempt) {
  const ceiling = Math.min(CAPMS, BASEMS * 2 ** attempt);
  return Math.floor(Math.random() * ceiling);
}

// Decorrelated jitter (AWS): grows from the previous actual delay,
// so late attempts stay spread without collapsing to near-zero waits.
function decorrelated(prevDelay) {
  const lo = BASE_MS;
  const hi = Math.min(CAPMS, Math.max(prevDelay  3, BASEMS  3));
  return Math.floor(lo + Math.random() * (hi - lo));
}

We use decorrelated jitter for webhook delivery and full jitter for internal RPC. The reason is operator-facing: with full jitter, attempt 9 can legitimately fire 400 milliseconds after attempt 8, which looks like a bug when a customer reads their delivery log and makes support conversations worse. Decorrelated keeps a visible floor.

One caveat. Jitter is for aggregate protection, and it costs individual latency. If you deliver low-volume, high-urgency events (payment confirmations to a single merchant), a fixed early schedule of 10s, 30s, 2m is better for the first three attempts, then switch to jittered exponential. We do exactly that: the first three attempts are tight and deterministic, everything after is jittered.

Classifying failures: the table that prevents most incidents

This is the cheapest high-value thing in the whole system, and it is almost always wrong in a first implementation because people write if (!res.ok) retry().

  • Retry: 408, 425, 429, 500, 502, 503, 504, plus transport-level failures (DNS NXDOMAIN or SERVFAIL, connection refused, connection reset, TLS handshake failure, read timeout).
  • Do not retry: 400, 401, 403, 404, 405, 406, 409 in most designs, 413, 415, 422. These are contract failures. Retrying them burns budget and fills the consumer’s logs with noise they will eventually bill you for in support time.
  • Retry, then disable: 410 Gone. Treat it as the consumer saying “this subscription is dead”. One attempt, then deactivate and notify.
  • Special case: 301/302/307/308. Follow a single redirect for GET-like semantics, never follow a cross-origin redirect for a signed POST. Signatures are computed over a URL and body; a redirect silently invalidates the first and may drop the second. We reject off-host redirects and surface them as a configuration error.

A subtlety on 401 and 403: if the consumer uses short-lived bearer tokens that your platform refreshes, 401 becomes retryable exactly once, after a forced token refresh. Model that as a distinct outcome, not as a generic retry, or you will loop.

Honour Retry-After, including when it hurts

Retry-After can be delta-seconds or an HTTP-date, and consumers send both. If a receiver tells you to come back in 900 seconds, your 40 second schedule is wrong and theirs is right. Clamp it to something sane (we clamp to 24 hours) and log when you clamp.

function retryAfterMs(header) {
  if (!header) return null;
  const secs = Number(header);
  if (Number.isFinite(secs)) return Math.min(secs * 1000, 86400000);
  const date = Date.parse(header);           // HTTP-date form
  if (Number.isNaN(date)) return null;
  return Math.min(Math.max(date - Date.now(), 0), 86400000);
}

const wait = retryAfterMs(res.headers.get('retry-after')) ?? decorrelated(prev);

If you also operate the receiving side, the mirror of this is emitting Retry-After on your own 429s so senders back off correctly instead of hammering you. We covered the sender-friendly side of that in rate limiting your form endpoint without blocking real users.

Retry budgets and circuit breakers

Retries are extra load generated precisely when the system is least able to take it. That means you need a hard ceiling independent of your schedule. Two mechanisms, both necessary:

Retry budget. Per destination, track successes and retries over a sliding 60 second window. Allow retries up to about 10 percent of successes, with a small constant floor (say 10 retries/minute) so a cold endpoint can still recover. The 10 percent figure comes out of the Google SRE book’s overload chapter and has held up well for us. When the budget is exhausted, retries are deferred, not dropped: push nextattemptat further out and move on.

Circuit breaker. After N consecutive hard failures (we use 20, or any single 5xx streak lasting 5 minutes), open the circuit for that destination. While open, stop dialling entirely and let events accumulate with their deadlines intact. Probe with a single request every 30 to 60 seconds. On success, go half-open and release the backlog at a throttled rate.

That last clause is the one that saved us. Do not drain a backlog at full speed. We release at min(normal_concurrency, 2) for the first 60 seconds after recovery, then ramp. A recovered consumer is almost always a cold consumer with empty caches and an unwarmed connection pool.

-- Claim due work without two workers grabbing the same row.
-- SKIP LOCKED is the whole trick; without it you serialise on the hot rows.
UPDATE deliveries d
   SET status = 'inflight', lockedat = now(), attempt = attempt + 1
 WHERE d.id IN (
   SELECT id FROM deliveries
    WHERE status = 'pending'
      AND nextattemptat <= now()
      AND endpointid NOT IN (SELECT endpointid FROM open_circuits)
    ORDER BY nextattemptat
    FOR UPDATE SKIP LOCKED
    LIMIT 50
 )
RETURNING d.*;

Note that nextattemptat is computed from database time, not worker time. Worker clock drift of even a few seconds across a fleet produces delivery logs that appear to go backwards, and customers will open tickets about it.

At-least-once is a promise you make to yourself and a problem you hand to them

Any retry policy that can retry after a timeout has already given up on exactly-once. A 10 second read timeout does not mean the consumer didn’t process the event; it means you don’t know. We measured this on one high-volume tenant: roughly 0.4 percent of timed-out requests had in fact been processed successfully. Those are duplicates by construction.

So make deduplication possible:

POST /hooks/orders HTTP/1.1
X-Event-Id: evt_01HZ8F3QX2K9
X-Idempotency-Key: evt_01HZ8F3QX2K9        # stable across every attempt
X-Delivery-Id: dlv_01HZ8F7T1A44            # unique per attempt
X-Attempt: 4
X-Event-Timestamp: 1767225600
X-Signature: t=1767225600,v1=9f2c...        # recomputed per attempt

The idempotency key must not change between attempts. Obvious, and we have still shipped a bug where it was derived from the delivery row rather than the event row. The signature timestamp, by contrast, must change, because it defends against replay and consumers reject stale timestamps outside a tolerance window (300 seconds is the convention).

Ordering is the other casualty. The moment you retry independently, event 5 can arrive before event 4. If a consumer needs order, the honest options are: a sequence number plus a version field in the payload so they can discard stale writes, or per-key serialisation where a failing event blocks its own partition and nothing else. We have shipped both. Per-key serialisation is simpler to explain and far more dangerous, because one poison event stalls a key indefinitely. Always put a max-age escape hatch on it. The receiver-side counterpart to all of this is in how to build a webhook receiver that does not lose events, and if the payload shape itself changes over time, versioning webhooks without breaking consumers covers the field-level rules.

The endgame: dead letters, auto-disable and replay

Expire on age first, attempts second. If your schedule says 19 attempts but a consumer returns 503 in 200 milliseconds every time, you will burn all 19 in a few hours and give up during an outage that lasts a day. Conversely, if the consumer times out at 10 seconds each time, 19 attempts spread across the full 48 hours. Age is the guarantee customers actually care about: “we try for 48 hours”. Attempts are an implementation detail.

After expiry:

  • Write the event to a dead letter store with the last 3 response bodies truncated to 2KB each. Not all of them. Nobody has ever read attempt 7’s body, and storing full bodies for every attempt was the single largest line in one client’s database bill.
  • Auto-disable the subscription after a sustained failure period, not a failure count. We use 72 hours with zero successes, and we email at 1 hour, 24 hours and on disable. Disabling at “N consecutive failures” punishes low-volume tenants unfairly: 19 failures for a tenant sending two events a day spans ten days.
  • Expose replay. A button, an API, a date-range bulk replay. Replay must reuse the original event ID and idempotency key, or you have just invented a duplicate-injection tool. Rate-limit replay harder than live traffic; it is the single most common way customers DDoS themselves.

Make all of this visible rather than explaining it in support tickets. A per-endpoint delivery log with attempt timestamps, status codes and the next scheduled attempt removes most of the guesswork, and a public health view removes the rest. We wrote up that side separately in building a status page for your webhook deliveries.

Timeouts are part of the policy

Separate connect and total timeouts. We use 5 seconds to connect and 10 seconds total. Generous read timeouts (30 to 60 seconds) are the most common cause of worker pool exhaustion we see, because a consumer under load holds every connection open until it dies, and your delivery system stops delivering to everyone. Tell consumers to acknowledge in under a second and process asynchronously, then enforce it with the timeout.

Two more things. Do not cache DNS across retries beyond the record TTL, and do not reuse a keep-alive connection to a host that just returned 503. Plenty of outages are resolved by a DNS failover or a load balancer swap, and a pinned connection or a stale resolver cache will keep you talking to the corpse.

When not to build this at all

All of the above is maybe 1,200 lines of code plus a queue, a scheduler, a breaker and an admin UI. It is worth it if delivery is part of your product. It is not worth it if you have three forms on a marketing site and want submissions in Slack. For that, point the form at a hosted endpoint like WebForms and let someone else own the queue, the retries and the dead letters. We have built the full machinery three times and used the hosted route perhaps thirty.

The honest dividing line: if you cannot name the person who will be paged when the delivery queue backs up, you should not own the delivery queue.

Frequently Asked Questions

How many webhook retry attempts are enough?

Think in hours, not attempts. A 24 to 72 hour window is the de facto standard, which works out to roughly 15 to 20 attempts with a base of 5 seconds, a factor of 2 and a 6 hour cap. Set the attempt count as a safety ceiling and let the age limit be the real expiry rule.

Full jitter or decorrelated jitter for exponential backoff?

Full jitter gives the best spread and the lowest total work under contention, so use it for internal service calls. Decorrelated jitter keeps a visible minimum gap between attempts, which matters for customer-facing delivery logs where a retry 300 milliseconds after the previous one looks broken. Either one beats no jitter by a wide margin.

Should a 429 response count against the retry budget?

Yes, and it should also respect Retry-After ahead of your own schedule. A 429 means the consumer is explicitly asking for less traffic, so continuing to spend budget on it is counterproductive. Treat repeated 429s as a signal to lower per-destination concurrency, not just to wait longer.

How do I stop one broken consumer from delaying everyone else?

Isolate work by destination: a per-endpoint circuit breaker, a per-endpoint concurrency cap, and a retry budget scoped to that endpoint. A single global FIFO queue is the design that keeps failing, because the dead endpoint’s retries sit at the head of the queue and block healthy traffic. Partition by endpoint ID and claim rows with FOR UPDATE SKIP LOCKED.

Is it safe to retry a request that timed out?

Only if the consumer can deduplicate, which means you must send a stable idempotency key on every attempt and document that they should store it. A timeout tells you nothing about whether the work happened, so some fraction of those retries will be duplicates. If duplicates would cause real damage and the consumer cannot dedupe, send once and dead-letter instead.

If you only change one thing this week, add jitter and a per-destination concurrency cap. Those two together turn a recovery stampede into a gentle ramp, and they take about an afternoon. The retry budget, breaker and replay API can follow once you have a delivery log good enough to show you which endpoints actually need them.

dead letter queue webhook delivery decorrelated jitter vs full jitter exponential backoff with jitter retry budget circuit breaker webhooks webhook idempotency key retries webhook retry webhook retry policy design which HTTP status codes to retry