Idempotency for Webhooks: Handling Duplicate Deliveries Properly

A client’s checkout flow once refunded the same order three times in eleven seconds. The payment provider had delivered one charge.refunded event, retried it twice because our handler took too long to respond, and the handler cheerfully processed all three. Nothing was broken in the usual sense. Every line of code did exactly what it was written to do.

That’s the thing about webhook idempotency: nobody designs a duplicate-handling bug. You inherit it the moment you accept a POST from a system you don’t control, because every serious webhook provider guarantees at-least-once delivery and not one of them guarantees exactly-once. Stripe, Shopify, GitHub, Slack, Twilio, Paddle: all of them will send you the same event twice under the right conditions, and those conditions happen more often than you’d like.

The question isn’t whether you’ll get a duplicate webhook. It’s whether the second copy quietly does nothing, or sends a second invoice.

Duplicates are the design, not the failure

Distributed systems can give you at-most-once or at-least-once delivery. Exactly-once delivery across a network you don’t own is marketing, not engineering. Providers choose at-least-once because losing a payment_intent.succeeded is far worse than sending it twice, and they push the dedupe responsibility to you. That’s the correct trade. It just needs to be acknowledged in your code.

The concrete ways you get a second copy:

• Your 2xx got lost. You processed the event, the response never arrived, the provider retried. This is the common one and it’s invisible in your logs unless you look for it.
• You were slow. Handler took 7 seconds, the provider’s timeout was 5, it marked the delivery failed and queued a retry. Your side-effects already ran.
• You returned 500 after doing work. Charged the card, then the email service threw, then your framework returned a 500. The retry charges again.
• Fan-out and config drift. Two endpoints registered against the same topic, or staging pointed at production after a database clone. Both happen.
• Manual replay. Somebody hits “resend” in the dashboard to debug something unrelated.

Only one of those is a provider quirk. The rest are yours.

Choosing an idempotency key that actually identifies the event

An idempotency key is whatever string lets you say “I have definitely seen this exact delivery before.” Pick it from the provider, in this order of preference:

1. The event ID. Stripe’s event.id (evt...), Paddle’s eventid, Shopify’s X-Shopify-Webhook-Id, GitHub’s X-GitHub-Delivery, Slack’s event_id. Stable across retries, unique per event. Use this.
2. A composite business key when the provider sends no ID: {resourcetype}:{resourceid}:{status}:{timestamp}. Deliberate and explainable, if ugly.
3. A SHA-256 of the raw body. Last resort.

The body hash deserves a warning, because many guides recommend it freely. Two genuinely distinct events can have byte-identical payloads: a subscription.updated that fires twice for different reasons, an inventory webhook that sends the same quantity after two separate adjustments, a form submission with no unique ID. Hash-based dedupe silently drops the second real event and you’ll never get an error about it. If you must hash, hash body plus the delivery timestamp header, and keep the dedupe window short.

Also: namespace the key by source. ('stripe', 'evt123'), not 'evt123'. You will add a second provider.

Webhook idempotency is a storage problem, not a logic problem

Almost every broken implementation we’ve reviewed looks like this:

// Don't. Two concurrent deliveries both pass the check.
const seen = await db.query('select 1 from webhookevents where eventid = $1', [id]);
if (seen.rowCount) return res.sendStatus(200);
await db.query('insert into webhookevents (eventid) values ($1)', [id]);
await doTheWork();

The window between the SELECT and the INSERT is small. It is not zero, and retries arrive in bursts, often within the same second. Let the database enforce uniqueness instead:

create table webhook_events (
  id           bigserial primary key,
  source       text        not null,
  event_id     text        not null,
  event_type   text        not null,
  payload      jsonb       not null,
  received_at  timestamptz not null default now(),
  claimed_at   timestamptz,
  processed_at timestamptz,
  attempts     int         not null default 0,
  last_error   text,
  unique (source, event_id)   -- the whole idempotency guarantee lives here
);

create index on webhookevents (processedat) where processed_at is null;

app.post('/hooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
  let event;
  try {
    // Verify before trusting anything, including the event ID you dedupe on.
    event = stripe.webhooks.constructEvent(req.body, req.headers['stripe-signature'], SECRET);
  } catch {
    return res.sendStatus(400); // bad signature: never retry this
  }

  const { rowCount } = await db.query(
    `insert into webhookevents (source, eventid, event_type, payload)
     values ('stripe', $1, $2, $3)
     on conflict (source, event_id) do nothing`,
    [event.id, event.type, event]
  );

  if (rowCount === 0) {
    res.set('X-Duplicate', '1');     // makes duplicates measurable in logs
    return res.sendStatus(200);      // 200, not 409: the provider should stop retrying
  }

  await queue.add('process-webhook', { source: 'stripe', eventId: event.id });
  res.sendStatus(200);
});

One insert, one branch, no race. Note the order: verify the signature before you write anything, otherwise anyone who can guess your endpoint can poison your dedupe table with fabricated event IDs and cause you to drop the real ones.

Redis is a cache, not your source of truth

SET key 1 NX EX 86400 is atomic and fast, and it’s a fine pre-filter in front of a high-volume endpoint. It is not a dedupe store. Keys get evicted under memory pressure, a failover to a replica can lose the last second of writes, and the Redis write isn’t in the same transaction as your business write. Use Redis to skip a database round trip on obvious repeats, and the unique constraint to be correct.

Separate receipt from processing

The single highest-value change you can make is to stop doing work inside the HTTP request. Receive, verify, persist, enqueue, return 200. Everything else happens in a worker you control the timing of.

This fixes three problems at once. The provider stops retrying because you’re slow. A failure in step four of your processing no longer re-runs steps one through three. And the retry story becomes yours: your own backoff, your own dead-letter queue, your own replay button.

The worker then needs its own idempotency, because two workers can pick up the same job. Claim atomically:

update webhook_events
   set claimed_at = now(), attempts = attempts + 1
 where source = $1
   and event_id = $2
   and processed_at is null
   and (claimedat is null or claimedat < now() - interval '5 minutes')  -- reclaim crashed workers
returning payload;

No rows returned means someone else owns it or it’s already done. Return, don’t retry. Set processed_at in the same transaction as your business writes so a crash between the two can’t mark an unprocessed event complete. If a side effect has to live outside that transaction (an email, an outbound API call), use the transactional outbox pattern: write the intent to a table inside the transaction, let a separate loop deliver it.

Out-of-order delivery breaks more things than duplicates

Deduping perfectly still leaves you exposed. Providers retry failed deliveries with backoff while continuing to send new ones, so a subscription.updated from 10:00:01 can land after the one from 10:00:04. Your dedupe table says both are new. Both are. One is stale.

Two defences, and you want both.

Guard writes with a monotonic field. Keep the source system’s timestamp on your row and refuse to go backwards:

update subscriptions
   set status = $1, plan = $2, sourceupdatedat = $3
 where third-party_id = $4
   and (sourceupdatedat is null or sourceupdatedat < $3);

Zero rows affected is a success here, not an error. You got an older truth and declined it. Use the provider’s own field (event.created, updatedat in the payload) and never your own receivedat, which tells you about network luck, not about ordering.

Treat the webhook as a notification, not as data. For anything where correctness matters more than latency, use the event as a trigger to re-fetch current state from the provider’s API, then reconcile. Stripe explicitly suggests this, and it dissolves the ordering problem entirely: whichever event you process last, you end up with the live state. The cost is an extra API call and rate-limit pressure, so reserve it for money, entitlements and access control, and take the payload at face value for analytics and notifications.

Pair that with a nightly reconciliation job that lists objects changed in the last 24 hours and fixes drift. Webhooks will fail eventually: an endpoint disabled after too many failures, a deploy window, a certificate expiry. Reconciliation is the safety net that turns a delivery outage into a delayed sync instead of a support ticket.

Your outbound side effects need idempotency keys too

Inbound dedupe protects your database. It doesn’t protect the third party you call during processing. If your worker crashes after creating a refund but before committing, the retry creates a second refund unless you pass a key.

// Derive the key from the inbound event, so a retry reuses the exact same key.
await stripe.refunds.create(
  { charge: payload.data.object.id },
  { idempotencyKey: refund:${event.id} }
);

Deriving the key from the inbound event ID is the whole trick. A random UUID per attempt gives you nothing. Note Stripe clears idempotency keys after 24 hours, so a retry three days later will go through as a new request. That’s one more reason your worker’s retry budget should be hours and not days.

For services with no idempotency support (most transactional email APIs, Slack incoming webhooks), dedupe on your side: a sentnotifications table with a unique constraint on (eventid, channel, recipient), checked before you send. Crude, effective, and it means you can answer “did we email this customer twice” without reading provider logs.

Doing this in WordPress without footguns

The pattern translates, but two WordPress habits will bite you. Don’t use transients or the object cache as a lock: on a site with Redis or Memcached, a cache flush during a deploy wipes your dedupe state, and on a site without persistent caching your transient is in wp_options with no atomicity guarantee. Don’t do the work inside the request either, because a plugin somewhere will add 400ms and push you past the provider’s timeout.

// On activation: a real unique index is the only thing doing the dedupe.
$wpdb->query( "CREATE TABLE IF NOT EXISTS {$wpdb->prefix}webhook_events (
  id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  source VARCHAR(32) NOT NULL,
  event_id VARCHAR(191) NOT NULL,
  payload LONGTEXT NOT NULL,
  received_at DATETIME NOT NULL,
  processed_at DATETIME NULL,
  PRIMARY KEY (id),
  UNIQUE KEY sourceevent (source, eventid)
) {$wpdb->getcharsetcollate()}" );

// In the REST callback, after signature verification:
$inserted = $wpdb->query( $wpdb->prepare(
  "INSERT IGNORE INTO {$wpdb->prefix}webhook_events
   (source, eventid, payload, receivedat) VALUES (%s, %s, %s, %s)",
  'stripe', $eventid, wpjsonencode( $body ), currenttime( 'mysql', true )
) );

if ( 0 === $inserted ) {
  return new WPRESTResponse( array( 'duplicate' => true ), 200 );
}

// Single-run cron event, scheduled a few seconds out.
wpschedulesingleevent( time() + 5, 'mypluginprocesswebhook', array( $eventid ) );
return new WPRESTResponse( array( 'queued' => true ), 200 );

Keep the 191 character column length if you’re on utf8mb4 with a legacy index limit. Register the route with a permission callback that verifies the signature rather than returning true; that’s covered in detail in custom REST API endpoints in WordPress. If WP-Cron is unreliable on the host (it often is on cached sites with low traffic), switch to a real system cron hitting wp-cron.php every minute, or use Action Scheduler.

Status codes, retention and proving it works

Return 200 for a duplicate. Not 409, not 304. A 4xx tells some providers the delivery failed and tells others to disable your endpoint after enough of them. Neither is what you want for an event you have correctly ignored. Return 400 only for a signature or parse failure, which is genuinely unretryable. Return 500 when you want a retry and mean it.

Retention: your dedupe records must outlive the provider’s retry window by a comfortable margin. Stripe retries for up to three days in live mode, Shopify across roughly 48 hours and nineteen attempts, GitHub only once. Thirty days of rows is cheap, answers most support questions, and gives you replay material. Partition or prune monthly rather than deleting row by row, and keep the payload column if your volume allows. The ability to replay a real event into staging is worth more than the disk.

Then instrument it. Count duplicates as a metric, not a log line. A sudden spike in duplicate webhook deliveries almost always means you got slow, which means some of those duplicates are racing your own in-flight processing. Alert when the duplicate rate for an endpoint exceeds 2 percent of deliveries in an hour. Exercise the behaviour locally before it ships: fire the same payload twice, out of order, and with a mid-processing crash, as described in testing webhooks locally. If your test suite never sends the same event twice, you don’t have idempotency. You have an untested hypothesis.