The Webhook Retry Runbook for Reliable Delivery

How do you design webhooks for redelivery? #

Treat every webhook like a delivery driver who keeps knocking until someone signs. That is literally the contract. Providers retry, events duplicate, order scrambles, bursts arrive during incidents.

Every webhook endpoint will receive duplicates, out of order events, and bursts during provider incidents. The runbook response is a fixed shape: verify the signature, persist the event, acknowledge fast, process asynchronously, and make handlers idempotent by event ID. Slow or stateful request handlers turn one failure into a redelivery storm.

Keep two dashboards: delivery health with attempts, latency, and failure rate by endpoint, and business outcomes with orders fulfilled and entitlements granted. When they diverge, the bug sits in the handler, not the network.

How do you acknowledge fast and process asynchronously? #

The request handler has one job: decide accept or reject in milliseconds. Verify the HMAC signature over the raw body, check the timestamp tolerance window, persist the raw event with its provider event ID, and return success. Everything else, entitlement updates, emails, provisioning, runs in a background worker reading from a queue.

Stripe documents this shape clearly: receive events, verify signatures, and return a fast success response, with slow work handled outside the request (https://docs.stripe.com/webhooks). GitHub describes the same push model where deliveries expect prompt responses and get retried on failure (https://docs.github.com/en/webhooks). Paddle frames its webhook events as notifications your app must accept quickly and reconcile (https://developer.paddle.com/webhooks/overview).

Slow handlers cause the exact failures they fear. A handler that calls three downstream APIs before responding will time out under load, and the provider will redeliver, doubling the downstream calls. Fast ack plus queue converts redelivery from a correctness threat into a duplicate the idempotency check absorbs.

// Minimal webhook handler shape
export async function POST(req) {
  const raw = await req.text();
  if (!verifyHmac(raw, req.headers)) return Response.json({ ok: false }, { status: 400 });
  const event = JSON.parse(raw);
  const fresh = await storeEventOnce(event); // unique on provider event id
  if (fresh) await enqueue('webhook.work', { id: event.id });
  return Response.json({ ok: true }); // always fast
}

Signature verification deserves care. Read the raw body before any JSON parsing, since parsing can alter whitespace and break the digest. Compare digests in constant time. Reject events outside the tolerance window even when the signature matches, because captured payloads stay replayable forever without that check. Stripe documents signature verification with timestamp tolerance as the expected defense (https://stripe.com/docs/webhooks/best-practices).

How do you handle retries, backoff, and the dead letter queue? #

Providers retry on their own schedule, but your outbound webhooks and your internal reprocessing need explicit policy. Retry transient failures like timeouts, 5xx responses, and rate limits with capped exponential backoff plus jitter. Do not retry validation failures, signature mismatches, or missing entity errors on a timer. Those need human or code fixes, and timers just burn quota.

A workable schedule looks like this: retry after seconds, then minutes, then tens of minutes, with a cap around a day and jitter so a fleet of failures does not march in lockstep. Jitter matters more than people think. Without it, every failed event from an outage retries at the same second and recreates the outage.

After a bounded number of attempts, park the event in a dead letter queue with full context: payload, headers, attempt history with status codes, and the handler version that failed. Review the dead letter on a cadence, fix the root cause, and replay from the queue rather than asking the provider to resend blindly. Alert on dead letter growth rate in addition to depth. A sudden spike means a deploy broke the handler. Depth tells you about yesterday. Growth rate tells you about right now.

Try it: JSON Formatter and Validator

How do you make handlers idempotent by event id? #

Every handler must answer one question: what happens if this event processes twice. The answer should be nothing the second time. Key processing on the provider event ID with a uniqueness constraint, check durable state before applying side effects, and design state transitions so replays converge, moving pending to active and never active back to pending.

Separate delivery identity from event identity. One logical event can arrive with several delivery IDs across retries. Store both, dedupe on the logical ID, and log the delivery ID for tracing. Return success for already processed events rather than errors. Errors invite more redelivery of work already done.

-- Dedupe on the logical event id, trace on delivery id
CREATE TABLE webhook_events (
  event_id TEXT PRIMARY KEY,
  delivery_id TEXT NOT NULL,
  payload TEXT NOT NULL,
  status TEXT NOT NULL DEFAULT 'received'
);

State machine design does quiet work here. If fulfillment moves an order from pending to active, a replayed event finds active and stops. If your code instead toggles or appends, replays corrupt. Convergent transitions plus a uniqueness constraint make at least once delivery safe by construction.

How do you monitor without drowning in noise? #

Track attempts, success rate, ack latency percentiles, and queue depth per endpoint. Alert on sustained failure rate changes and dead letter growth, not single failures. Correlate spikes with deploys. Most webhook incidents follow a handler change, not a provider outage.

For billing webhooks specifically, add a reconciliation job. Poll the provider periodically and compare against locally recorded entitlements. The poller catches anything the push path missed, and the push path keeps latency low. Speed plus a backstop beats either alone. Paddle and Stripe both document event types and list APIs that make this reconciliation straightforward (https://developer.paddle.com/webhooks/overview).

One more habit. Log the handler version with every processed event. When a bad deploy slips through, you can name every event it touched and replay exactly those, instead of guessing from timestamps.

Build the endpoint boring. Verify, persist, ack, queue, dedupe, reconcile. The driver keeps knocking. Your door should open fast and remember every knock.

What are the trade-offs? #

Design for redelivery wins when providers promise at least once. Stripe and Paddle retry on timeouts and 5xx, so verify HMAC over the raw body, persist the event, ack in milliseconds, work async, retry transient failures with capped backoff plus jitter, and park the exhausted in a dead letter queue keyed by provider event id.

It loses when billing logic lives inside the request handler or monitors drown in noise. Slow endpoints cause redelivery storms. Pick the alternative shape: fast ack, queued work, reviewed dead letters.

Who this is for (and who should skip it)? #

This runbook helps builders making webhook endpoints survive redelivery, ordering drift, and bursts. If you own billing or provisioning via webhooks, the fast ack plus queue plus idempotency shape is load bearing.

Skip deep webhook infrastructure if you consume no webhooks and you reconcile by polling. That path stays simple until a provider requires push for lifecycle events.

One limit to know. Validation and signature errors should not retry on a timer, since repeats change nothing until code or config is fixed. Pair push handling with periodic reconciliation, or missed events stay missed.

• Best for developers designing webhook endpoints for redelivery with fast acks and queued work.
• Best for startups adding backoff, dead-letter review, and idempotent handlers by event ID.
• Best for freelancers monitoring delivery health and business outcomes separately for clients.

What we learned building this? #

BYOB follows the verify plus persist plus fast ack plus queue plus dedupe shape for webhooks documented in the Paddle webhooks overview and Stripe webhooks best practices linked in the post, and the retry and queue wiring is collected in the BYOB deployment pipeline guide and billing handling in the BYOB Paddle payments guide, both verified 200 live. Handler versions are logged with each event so a bad deploy maps to an exact replay set instead of time based guessing.