How to Build a Webhook Receiver That Does Not Lose Events

A client’s Stripe integration looked fine for eight months. Then their database had a slow Tuesday, the connection pool saturated, and the endpoint started returning 500s. Stripe retried, as it does, for three days. On day four the retries stopped and 41 subscription events were simply gone, including two cancellations that kept billing customers who had already left.

Nobody noticed for three weeks. The endpoint returned 200 again as soon as the database recovered, the dashboard went green, and the hole in the middle was invisible because there was no record of what had been attempted. That’s the failure mode worth designing against: not the endpoint being down, but the endpoint being down quietly and the provider giving up before you look.

A reliable webhook receiver is not a controller method. It’s an intake pipeline with a durable buffer in the middle, and the HTTP handler is the least interesting part of it.

The handler has exactly two jobs

Verify, persist, acknowledge. Every line of business logic inside the request cycle is a new way to breach the provider’s timeout, and those timeouts are tighter than people expect. Shopify cuts the connection at 5 seconds. Most others sit between 10 and 30 seconds, and none of them care that your CRM API was slow today.

Here’s the whole handler in Express. Note the raw body parser, which is the part everyone gets wrong.

import express from 'express';
import crypto from 'node:crypto';
import { pool } from './db.js';

const app = express();

// Raw bytes, not parsed JSON. HMAC is computed over the exact payload sent.
app.post('/hooks/stripe',
  express.raw({ type: 'application/json', limit: '2mb' }),
  async (req, res) => {
    const sig = req.get('stripe-signature') || '';
    if (!verify(req.body, sig, process.env.STRIPEWEBHOOKSECRET)) {
      return res.sendStatus(401); // 4xx: do not retry, this will never succeed
    }

    let eventId;
    try {
      eventId = JSON.parse(req.body.toString('utf8')).id;
    } catch {
      return res.sendStatus(400);
    }

    try {
      await pool.query(
        `INSERT INTO webhookevents (provider, eventid, headers, payload, received_at)
         VALUES ($1, $2, $3, $4, now())
         ON CONFLICT (provider, event_id) DO NOTHING`,
        ['stripe', eventId, JSON.stringify(req.headers), req.body]
      );
    } catch (err) {
      req.log.error({ err, eventId }, 'webhook persist failed');
      return res.sendStatus(503); // 5xx: please retry, we lost it
    }

    res.sendStatus(200);
  });

Three deliberate choices in there. The signature failure returns 401 because a retry will produce the identical result, so there’s no point burning the provider’s retry budget. The persistence failure returns 503 because a retry is exactly what you want. And the duplicate insert is swallowed by ON CONFLICT DO NOTHING, because providers resend events constantly and you still owe them a 200.

If you’re on PHP, the equivalent trap is reaching for $POST instead of filegetcontents('php://input'). In WordPress, registerrestroute hands you a parsed WPRESTRequest, and $request->getbody() gives you the raw string you actually need for HMAC.

Why the raw body matters more than it sounds

Signature verification runs HMAC over the literal bytes the provider sent. Parse that JSON and re-serialise it and you’ve changed key order, whitespace, unicode escaping, and float formatting. The signature no longer matches. People then “fix” this by disabling verification in production, which is a far worse outcome than the original bug.

Buffering proxies cause the same class of problem from a different direction. If Cloudflare, nginx or an AWS ALB sits in front of you and strips or normalises headers, your signature header may arrive lowercased, duplicated, or missing entirely. Log the full header set on every delivery during the first week of a new integration. Providers occasionally document a header name that differs from what actually arrives over the wire, and the only way to find out is to look.

The verification details, timestamp tolerances and replay windows are covered separately in Webhook Security: Signature Verification, Replay Attacks and Secrets, so this piece stays on durability.

The durable buffer: write before you work

The table is the whole design. Everything else is plumbing.

CREATE TABLE webhook_events (
  id            bigserial PRIMARY KEY,
  provider      text        NOT NULL,
  event_id      text        NOT NULL,
  headers       jsonb       NOT NULL,
  payload       bytea       NOT NULL,   -- raw bytes, exactly as received
  received_at   timestamptz NOT NULL,
  status        text        NOT NULL DEFAULT 'pending',
  attempts      int         NOT NULL DEFAULT 0,
  nextattemptat timestamptz NOT NULL DEFAULT now(),
  locked_until  timestamptz,
  last_error    text,
  processed_at  timestamptz,
  UNIQUE (provider, event_id)
);

-- Partial index: the worker only ever scans pending rows that are due.
CREATE INDEX idxwebhookdue ON webhookevents (nextattempt_at)
  WHERE status = 'pending';

Store the payload as bytea, not jsonb. You want to replay the original bytes through verification months later when you’re arguing with a provider about whether an event was ever sent. Add a jsonb generated column if you need to query inside the payload.

The unique constraint is your deduplication primitive and it’s doing real work. Providers deliver the same event twice more often than their docs suggest, usually when a response is slow enough that they time out after you’ve already committed. The constraint turns that into a no-op. Deduplicating the effects of an event, which is a different and harder problem, is covered in Idempotency for Webhooks.

When the event has no ID

Some providers send no stable identifier. Build one: SHA-256 over the raw body plus the provider’s timestamp header, truncated to 32 hex characters. It’s not perfect. Two genuinely identical events sent in the same second will collapse into one. But for most payloads (which contain an order ID, a resource ID, or an updated-at value) collisions are vanishingly rare.

Draining the queue without a broker

Below roughly 500 events per minute, a database-backed queue beats SQS or Redis on operational simplicity and loses nothing meaningful. SKIP LOCKED is what makes it safe for concurrent workers.

UPDATE webhook_events
SET status = 'processing',
    locked_until = now() + interval '5 minutes',
    attempts = attempts + 1
WHERE id IN (
  SELECT id FROM webhook_events
  WHERE status = 'pending' AND nextattemptat <= now()
  ORDER BY id
  FOR UPDATE SKIP LOCKED
  LIMIT 20              -- small batches keep the lock window short
)
RETURNING id, provider, payload;

A reaper query resets rows whose locked_until has passed back to pending. That covers the worker being OOM-killed mid-process, which will happen.

Backoff on failure should be exponential with jitter. Without jitter, a downstream API that returns 503 for thirty seconds will have every one of your failed events retrying in lockstep forever.

function nextDelaySeconds(attempts) {
  const base = Math.min(2 ** attempts, 3600);      // cap at 1 hour
  return Math.round(base * (0.5 + Math.random())); // +/- 50% jitter
}

After 12 to 15 attempts, move the row to status = 'dead' and alert. A dead letter queue that nobody looks at is a landfill. Make it a dashboard row with a count, visible on the same screen as your error rate.

Serverless changes the rules

On Lambda, Vercel or Cloudflare Workers, work scheduled after you return the response is not guaranteed to run. The container can be frozen the instant the response is flushed. A floating promise to “process in the background” gives you a lossy receiver, and you won’t find out until traffic spikes.

Two correct patterns. Use the platform’s explicit mechanism (ctx.waitUntil() on Workers, waitUntil from @vercel/functions), or have the function write to a real queue and return. API Gateway can put the payload straight into SQS with no Lambda at all, removing your code from the critical path entirely. Fewer moving parts in the path that must not fail.

Serverless also hurts you via connection limits. Three hundred concurrent invocations each opening a Postgres connection will exhaust the pool, and your handler starts returning 503 under exactly the load spike that matters. Use a pooler (PgBouncer, RDS Proxy, Supabase’s pooled port) or don’t use a database in the handler at all.

Ordering is not your problem to solve

Developers spend a lot of effort trying to process webhooks in order. Stop. HTTP delivery over a retrying transport gives no ordering guarantee, and a subscription.updated can and will land before the subscription.created that caused it.

Make handlers order-independent instead. Three mechanisms cover nearly everything:

• Version guards. If the payload carries a sequence number or updatedat, compare before writing and discard anything older than your current state. One WHERE updatedat < $new clause solves most of it.
• Fetch, don’t trust. On receipt of invoice.paid, call the provider’s API for the current invoice rather than acting on the embedded snapshot. Costs a round trip, eliminates a class of bugs.
• Upsert on create. Treat created events as upserts so a late arrival doesn’t duplicate a record or throw.

The related question of what you put in the payload in the first place, and how to change it later without breaking consumers, is in Payload Design: Versioning Webhooks Without Breaking Consumers.

Proving webhook reliability, not assuming it

Error rate is the wrong primary metric. A receiver that accepts everything and silently never processes it has a 0% error rate. Track these instead:

• Age of oldest pending event. Alert above 5 minutes. This is the single most useful webhook alert you will ever configure.
• Dead letter count, as an absolute number, alerting on any increase rather than on a percentage.
• Delivery-to-processed p95 latency, computed from the provider’s own timestamp header rather than your received_at, so network and provider-side queueing are visible.
• Daily reconciliation. Pull the provider’s event list via API (Stripe, Shopify and GitHub all expose one) and diff it against your event_id column. This catches events that never arrived at all, which no amount of receiver-side instrumentation can.

That reconciliation job is the one people skip. It’s twenty lines of code and it’s the only check that covers DNS failures, WAF blocks and expired certificates on your own endpoint.

If you’re receiving form submissions rather than platform events, a hosted endpoint like WebForms already does the durable intake, retry and fan-out to Slack, Zapier or your own webhook, which is usually a better use of an afternoon than rebuilding it.

Test the failure path, not the happy path

Every receiver works when the database is up. Write tests for the cases that actually bite:

1. Same event delivered twice concurrently. Assert exactly one side effect.
2. Database unavailable during insert. Assert a 5xx is returned, not a 200.
3. Worker killed between claim and commit. Assert the reaper requeues it.
4. Payload with a tampered byte. Assert 401 and no row written.
5. A 10,000-event replay dumped in at once. Watch pool usage and downstream rate limits.

For number five, provider CLIs make this straightforward: stripe trigger payment_intent.succeeded in a loop, or replay yesterday’s rows through your worker in a staging environment.

Three things not to do

Don’t process inside a database transaction that stays open while you call an third-party API. You’ll hold locks for the duration of someone else’s outage.

Don’t delete processed rows immediately. Keep them 30 to 90 days, then archive. The cost of a few gigabytes is nothing against the ability to answer “did we receive that?” with certainty.

Don’t put the receiver behind the same autoscaling group as your marketing site. A traffic spike on one should not cause event loss on the other. Separate deployment, separate scaling policy, separate alerting.