Building a Status Page for Your Webhook Deliveries

A client messaged us at 11pm: “your webhooks stopped working.” They were right, sort of. Their endpoint had started returning 502s roughly four hours earlier, our delivery worker had retried, backed off and eventually parked 1,900 events in a dead letter queue, and nobody on either side knew because the only record of any of it was a log line in CloudWatch that nobody was watching.

Fixing the delivery was twenty minutes of work. Proving what happened took three hours of grep. That gap is the entire argument for building proper webhook monitoring before you need it, and for putting a delivery status view in front of the people consuming your events rather than making them email you.

This is the version we actually build now, after shipping it four or five times. Postgres, one attempts table, keyset pagination, and a very deliberate decision about what the public status page is allowed to say.

Three pages, not one

People say “status page” and mean different things. Separate them early. They have different access rules, different retention needs, and different definitions of “healthy”.

The internal delivery log is for you. Full payloads, every endpoint across every account, replay anything, filter by error class. This is the forensic tool.

The per-consumer delivery status view lives in your customer dashboard. It shows only their endpoints, with payloads redacted according to your masking rules, and a replay button scoped to a single message. This is the page that stops the 11pm message.

The public status page is aggregate and reports on your system only. If a customer’s receiver is down, your public page stays green. Getting this wrong is a classic mistake: we have seen a provider’s status page flip to “degraded” because one large consumer deployed a broken release, which then triggered incident processes at four other companies who were fine. Be strict about the boundary.

The data model that makes every query easy

Two tables and a partitioning plan. Messages are logical events. Attempts are what physically happened on the wire.

create table webhook_messages (
  id              uuid primary key,          -- stable across all retries; this is the webhook-id header
  endpointid     uuid not null references webhookendpoints(id),
  event_type      text not null,
  payload_version text not null,             -- see payload versioning
  payload         jsonb,                     -- null when offloaded to object storage
  payload_ref     text,                      -- s3 key for payloads over 32KB
  enqueued_at     timestamptz not null default now(),
  state           text not null              -- pending | delivered | exhausted | cancelled
);

create table webhook_attempts (
  id             bigint generated always as identity,
  message_id     uuid not null,
  attempt_number smallint not null,
  started_at     timestamptz not null,
  duration_ms    integer,
  status_code    smallint,                   -- null when no response was ever received
  errorclass    text,                       -- dns | tls | timeout | connreset | http4xx | http5xx | redirect_blocked
  response_body  text,                       -- truncated at 2048 bytes
  triggeredby   text not null default 'auto',   -- auto | manualreplay | backfill
  primary key (id, started_at)
) partition by range (started_at);

create index on webhookattempts (messageid, attempt_number);
create index on webhookattempts (startedat desc, id desc);

Monthly range partitions on startedat are not premature optimisation here. Attempts grow at roughly 1.3 rows per message once you account for retries, and at 2 million events a day you will want drop table webhookattempts202601 rather than a DELETE that runs for six hours and bloats the heap.

Note what is not in the attempts table: the payload. Storing it per attempt multiplies your largest column by your retry count for zero information gain. A retry sends the same bytes.

Instrumenting the delivery worker

The attempt row must be written in the same transaction that updates the message state. Write them separately and you will eventually get a crash between the two, producing either a phantom retry or an attempt that never appears in the logs. We have debugged both.

import { request } from 'undici';

const BODY_CAP = 2048; // enough for a JSON error or a stack line, not a full HTML error page

function classify(err) {
  const c = err.code || err.name;
  if (c === 'ENOTFOUND' || c === 'EAI_AGAIN') return 'dns';
  if (c === 'UNDERRHEADERSTIMEOUT' || c === 'UNDERRBODYTIMEOUT') return 'timeout';
  if (c === 'ECONNRESET' || c === 'ECONNREFUSED') return 'conn_reset';
  if (String(c).startsWith('ERRTLS') || c === 'CERTHAS_EXPIRED') return 'tls';
  return 'unknown';
}

export async function attemptDelivery(msg, endpoint, attemptNumber) {
  const body = JSON.stringify(msg.payload);
  const ts = Math.floor(Date.now() / 1000);
  const startedAt = new Date();
  const t0 = process.hrtime.bigint();

  let statusCode = null, errorClass = null, responseBody = null;

  try {
    const res = await request(endpoint.url, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        'webhook-id': msg.id,                 // consumers dedupe on this across retries
        'webhook-timestamp': String(ts),
        'webhook-signature': sign(msg.id, ts, body, endpoint.secret),
        'user-agent': 'Acme-Webhooks/1.4'
      },
      body,
      headersTimeout: 5000,
      bodyTimeout: 10000,
      maxRedirections: 0                      // a 302 to an attacker-controlled host is an SSRF vector
    });
    statusCode = res.statusCode;
    responseBody = (await res.body.text()).slice(0, BODY_CAP);
    if (statusCode >= 500) errorClass = 'http_5xx';
    else if (statusCode >= 400) errorClass = 'http_4xx';
    else if (statusCode >= 300) errorClass = 'redirect_blocked';
  } catch (err) {
    errorClass = classify(err);
  }

  const durationMs = Number((process.hrtime.bigint() - t0) / 1000000n);
  const delivered = statusCode !== null && statusCode >= 200 && statusCode < 300;

  // single transaction: attempt row + message state + next retry schedule
  await commitAttempt({
    messageId: msg.id, attemptNumber, startedAt, durationMs,
    statusCode, errorClass, responseBody, delivered
  });

  return { delivered, statusCode, errorClass, durationMs };
}

Two details that matter more than they look. maxRedirections: 0 keeps a compromised or sloppy consumer from bouncing your signed request somewhere internal. Measuring with hrtime rather than Date.now() matters once you start charting p95 latency: wall clock jumps during NTP correction produce negative durations and a very confusing graph.

If your queue and retry logic are not yet solid, that is the prerequisite. We wrote up the receiving half of this problem in building a webhook receiver that does not lose events, and most of the durability reasoning applies symmetrically to the sending side.

Webhook monitoring that catches the failures that matter

Global success rate is a bad alarm. At 50,000 deliveries an hour spread over 800 endpoints, one endpoint failing 100% of the time moves your aggregate by a fraction of a percent. Nobody pages on that. The customer notices in four hours.

Roll up per endpoint instead:

select
  m.endpoint_id,
  count(*)                                                   as attempts,
  count(*) filter (where a.status_code between 200 and 299)   as ok,
  percentiledisc(0.95) within group (order by a.durationms) as p95_ms,
  max(a.startedat) filter (where a.statuscode between 200 and 299) as lastsuccessat,
  mode() within group (order by a.errorclass)                as toperror
from webhook_attempts a
join webhookmessages m on m.id = a.messageid
where a.started_at > now() - interval '1 hour'
group by m.endpoint_id;

Then alert on two conditions only. First: consecutive failures on one endpoint crosses 10, which catches hard breakage fast. Second: no successful delivery for 15 minutes on an endpoint that averaged more than one delivery a minute over the previous day, which catches the silent cases where volume dropped to zero and nothing is technically erroring. The second rule is the one that earns its keep.

For your own health, track queue age: p99 seconds from enqueuedat to the first attempt’s startedat. If that number is 1.2s, you are fine regardless of how many consumers are returning 500s. If it climbs to 90s, you have a real incident. That metric is what your public status page should be driven by.

The consumer-facing delivery status view

Keep it to one table and one detail panel. Columns: timestamp, event type, status pill, attempt count, duration. Expanding a row reveals request headers, the redacted payload, and the truncated response body.

Paginate with keyset, never OFFSET. Page 400 of an offset query on a partitioned attempts table will time out, and someone will hit page 400 while hunting for a delivery from last Tuesday.

-- cursor = (laststartedat, last_id) from the previous page
select a.*, m.event_type
from webhook_attempts a
join webhookmessages m on m.id = a.messageid
where m.endpoint_id = $1
  and (a.started_at, a.id) < ($2, $3)
order by a.started_at desc, a.id desc
limit 50;

Dense log tables are also where responsive layouts fall apart. You cannot sensibly hide the status column on mobile. The collapse-to-cards pattern works but needs the header repeated per row; we covered the trade-offs in responsive data tables. If you are building the dashboard on Canvas, the stacked table variant plus a sticky filter bar gets you most of the way without custom CSS.

The copy-as-curl button

This is the highest leverage feature on the page. Generate a runnable curl for any attempt so the consumer can reproduce it against their own staging environment:

function toCurl({ url, headers, body }) {
  const h = Object.entries(headers)
    .filter(([k]) => k !== 'webhook-signature')   // stale signatures fail timestamp tolerance anyway
    .map(([k, v]) =>   -H '${k}: ${v.replace(/'/g, "'\\''")}')
    .join(' \\\n');
  return curl -X POST '${url}' \\\n${h} \\\n  -d '${body.replace(/'/g, "'\\''")}';
}

Omit the signature deliberately and say so in the UI. A 40-minute-old timestamp will be rejected by any correct verifier, so handing over a signature that cannot work produces a support ticket instead of preventing one. Put a link next to the button explaining how to re-sign for local testing, which follows the same logic described in webhook signature verification and replay protection.

Replay semantics, redaction and retention

Replay must not create a new message. It appends an attempt to the existing message with triggeredby = 'manualreplay' and sends the same webhook-id, so a consumer with correct deduplication treats it as the duplicate it is. If you mint a fresh id on replay, you have handed every customer a tool for double-charging their users. Consumers that dedupe properly, as described in idempotency for webhooks, will be fine either way, but plenty do not.

Rate limit bulk replay. One customer clicking “replay all 1,900” will otherwise saturate your worker pool and delay everyone else’s live traffic. Cap it at 200 messages per request and queue them at normal priority, behind new events.

On redaction, apply a field mask at write time, not read time:

<?php
// Masked at ingest so the raw value never reaches the webhook_logs table.
const MASKED_PATHS = ['customer.email', 'customer.phone', 'card.last4'];

function maskPayload(array $payload, array $paths): array {
    foreach ($paths as $path) {
        $ref = &$payload;
        $parts = explode('.', $path);
        $last = array_pop($parts);
        foreach ($parts as $p) {
            if (!isset($ref[$p]) || !is_array($ref[$p])) { continue 2; }
            $ref = &$ref[$p];
        }
        if (isset($ref[$last])) { $ref[$last] = '[redacted]'; }
        unset($ref);
    }
    return $payload;
}

Retention that has held up for us: full payloads for 7 days, attempt metadata for 30, hourly aggregates for 13 months. The aggregates are what you need when someone asks “was this failing in March”, and they cost almost nothing. If your event payloads include anything that falls under a data processing agreement, 7 days of payload storage is also a much easier conversation with a customer’s legal team than 90.

What the public page should actually say

Four numbers, updated every 60 seconds, each with a 30-day sparkline: p99 queue age, deliveries attempted in the last hour, percentage of attempts that received any HTTP response within timeout, and current incident state. That third metric is the honest split between your failures and theirs. A consumer returning 503 counts as a successful delivery attempt by your system.

Serve it from a static file or a cached JSON endpoint on infrastructure that does not share a database with the thing it reports on. A status page that goes down with the API is theatre. If you have no spare infrastructure, a small static page with a JSON fetch and a form endpoint from WebForms for incident subscriptions is a perfectly respectable thirty-minute build.

One caveat on all of this. If you send fewer than a few thousand webhooks a day to under twenty endpoints, the attempts table plus a basic internal filter view is genuinely enough. Skip the aggregates, skip the public page, keep the alert on consecutive failures. The architecture above earns its complexity somewhere north of a million deliveries a month, and building it on day one is a good way to ship nothing.