Building a Webhook Delivery Service: Signing, Retries, Dead Letters

What a webhook delivery service is and why inline calls fail

A webhook delivery service is a dedicated component that accepts events from business services, persists them, and owns every outbound HTTP attempt to subscriber endpoints. The order service records "order.paid" and returns. It does not call the customer's URL itself.

Why inline calls fail

When a business service POSTs to a webhook endpoint inside its own request path, three problems appear:

• Latency coupling. Your p99 now includes a stranger's server. A customer endpoint that takes eight seconds to time out holds a request thread for eight seconds.
• Lost events. If the process crashes after committing the database transaction but before the HTTP call succeeds, the event is gone. Nothing records that it was owed.
• Duplicated retry logic. Each team writes its own backoff, timeout and logging, each with different bugs and none with a shared view of delivery history.

Choosing an approach

The dedicated service adds real operational surface. That cost makes sense once you push to hundreds of endpoints you don't control. It is hard to justify for a handful of internal subscribers.

Who should use a webhook delivery service

The service is built for teams whose product sends events to systems they do not operate:

• API and platform teams whose customers register their own URLs to receive events such as payments, order changes, or account updates. These teams need signing, retries, and an audit trail they can show customers.
• Organizations with several producing teams. If billing, orders, and identity all emit webhooks, one shared service gives them one signing scheme, one retry policy, and one attempt log, instead of three partial implementations.
• Teams with support or compliance obligations that must answer "was this event delivered, when, and what did the receiver return?" The attempt log answers that directly.

It is a poor fit for a single team with a few internal consumers. Those consumers can usually read from a broker or call an API directly, and the extra component brings little benefit.

When to use it and when to avoid it

When it should be used

Build or adopt a dedicated delivery service when most of these hold:

• Subscribers are external, so you cannot hand them broker credentials or require a particular client library.
• Losing an event has a real consequence, such as a missed payment notification or an out-of-sync order state, so delivery must be durable and retried.
• Endpoints vary widely in reliability and speed, and a slow one must not hold up the others or your own request path.
• Customers expect to verify where requests come from (signatures), inspect delivery history, and replay failures themselves.
• More than one service emits events, so centralizing retry and signing logic removes repeated work.

When it should be avoided

Skip it, or postpone it, when:

• All consumers are internal services that can subscribe to a broker or event stream. Pull-based consumption gives them replay and backpressure without HTTP push.
• Volume is low and an occasional lost notification is acceptable, for example a chat alert on a deploy. An inline call with a short timeout and a log line is enough.
• Consumers need strict ordering or transactional guarantees across events. Webhooks are at-least-once and usually unordered (see below). A shared log or a pull API that consumers page through matches that need better.
• Nobody will own it on call. A delivery service with no owner collects a growing dead letter queue and silent failures, which is worse than having no service.
• Receivers cannot accept inbound HTTPS, for example clients behind strict firewalls. Polling or a long-lived outbound connection from the client is the realistic option.

How it works at a glance

The flow is the same no matter how it is implemented:

1. A business service commits a state change and an event record in one database transaction (the outbox).
2. A relay publishes committed events to a durable queue.
3. Workers take each event, find every subscribed endpoint, and create one delivery record per event-endpoint pair.
4. For each delivery, a worker signs the exact payload bytes with the endpoint's secret, makes the HTTPS call with short timeouts and SSRF checks, and logs the attempt.
5. The response is classified as success, retryable, or non-retryable. Retryable failures are rescheduled with jittered exponential backoff. Deliveries that use up their retry window go to a dead letter queue.
6. Operators and customers inspect the attempt log and replay dead-lettered deliveries once the endpoint is fixed.

The sections below cover each stage in detail.

Alternatives to a dedicated delivery service

The "Choosing an approach" table lists the main options. In more detail:

• Inline HTTP calls from the producing service. This is the simplest option and needs no new infrastructure. It suits low-stakes, low-volume notifications. You accept latency coupling and lost events on crashes. Adding a background thread with retries in memory reduces the latency problem but not the loss problem.
• Broker or event stream that consumers pull from. Consumers read at their own pace and replay from an offset, and ordering within a partition comes for free. This is usually the better choice for internal consumers. It works poorly for external customers, who would need network access, credentials, and client libraries for your broker.
• Polling or an events API. You expose an endpoint such as "list events since cursor X", and consumers poll it. No inbound HTTP is needed on their side, and consumers control pacing and replay. The costs are polling load on your API and delivery latency equal to the polling interval. Many providers offer this alongside webhooks so consumers can backfill missed events.
• Persistent connections (WebSockets or server-sent events). These suit real-time delivery to clients that keep a connection open. You still need a durable store and a resume-from-cursor mechanism for reconnects, so they complement rather than replace durable delivery.
• Managed webhook delivery. A third-party service handles queuing, retries, signing, and the customer-facing portal. You trade operational work for a vendor dependency, per-message cost, and event payloads passing through another party.
• Cloud provider event routing. If both you and your consumers run on the same cloud, its event routing and cross-account delivery features can replace HTTP push. This only works when consumers are on that platform.

A common combination is webhooks as the push notification plus an events API for backfill and reconciliation. Each one covers the other's weak spot: push gives low latency, and the API gives guaranteed recoverability.

How the delivery pipeline works: outbox, queue, workers and attempt logs

The pipeline has four stages, and each one exists to remove a specific failure mode.

Transactional outbox. When an order moves to paid, the service writes the state change and an outbox row in the same database transaction. The row holds the event ID, event type, aggregate ID and JSON payload. Both writes commit together or roll back together. You never get a paid order with no event, or an event for an order that was rolled back. Publishing straight to a broker from request code can't give you that guarantee, because the broker and the database don't share a commit.

Relay. A separate process reads unpublished outbox rows in insertion order. It publishes each one to a durable queue, then marks the row as sent. If it crashes between the publish and the mark, the event goes out twice. This is the first of several reasons delivery is at-least-once.

Delivery workers. Workers consume from the queue and fan each event out to every subscribed endpoint. Each event-endpoint pair gets its own delivery record with a state (pending, succeeded, failed, dead), an attempt count and a next_attempt_at timestamp. Without that split, one failing endpoint would block or re-send to the healthy ones.

Attempt log. Every HTTP call appends one row. A minimal schema has delivery_id, attempt_number, started_at, status_code (null on timeout or connection error), latency_ms, error_class and response_body truncated to a few kilobytes. The truncation keeps a misbehaving receiver from filling your storage. This table answers "did you send it?" when a customer asks, and it supplies the raw data for retry and dead-letter decisions.

Copy-- PostgreSQL. The state change and its event commit or roll back together.
BEGIN;

-- The app should check that exactly one row was updated before continuing,
-- and roll back otherwise, so a duplicate request doesn't emit a second event.
UPDATE orders
   SET status = 'paid', paid_at = now()
 WHERE id = 'ord_8f3a2c' AND status = 'pending';

-- The relay later reads unpublished rows in insertion order.
INSERT INTO outbox (event_id, event_type, aggregate_id, payload)
VALUES (
  gen_random_uuid(),  -- built in since PostgreSQL 13
  'order.paid',
  'ord_8f3a2c',
  jsonb_build_object('order_id', 'ord_8f3a2c', 'amount_cents', 4999, 'currency', 'USD')
);

COMMIT;

-- Minimal attempt log: one append-only row per HTTP call.
-- Assumes a deliveries table with one row per event-endpoint pair.
CREATE TABLE delivery_attempts (
  delivery_id    bigint      NOT NULL REFERENCES deliveries (id),
  attempt_number integer     NOT NULL,
  started_at     timestamptz NOT NULL,
  status_code    smallint,             -- NULL on timeout or connection error
  latency_ms     integer     NOT NULL,
  error_class    text,                 -- e.g. 'timeout', 'dns', 'tls', 'http_5xx'
  -- The worker truncates before insert. This check guards against a missed truncation.
  response_body  text CHECK (octet_length(response_body) <= 4096),
  PRIMARY KEY (delivery_id, attempt_number)
);

A PostgreSQL transaction that writes an order's state change and its outbox event together, followed by a minimal append-only attempt-log table that records status code, latency, error class and a size-capped response body.

Designing the HMAC webhook signature and rotating secrets

Each endpoint gets its own random secret, generated server-side with at least 32 bytes of entropy. A leak from one customer then exposes only that customer's deliveries. The signature is HMAC-SHA256 over a canonical string: {delivery_id}.{timestamp}.{raw_body}. The delivery ID and timestamp are joined with literal dots, and the body is the exact bytes you send. Three headers carry the values: Webhook-Id, Webhook-Timestamp (Unix seconds) and Webhook-Signature. The sender serializes the payload once and signs those bytes. It never signs a re-serialized object, because key order or whitespace could change between signing and sending.

On the receiver, verification follows a strict order:

1. Read the raw body before any JSON parsing. Frameworks that parse and re-encode the body will break the signature.
2. Recompute the HMAC and compare it to the received value in constant time, so timing differences don't leak how many bytes matched.
3. Reject the request if the timestamp falls outside a tolerance window of about five minutes.
4. Deduplicate on the delivery ID within that window.

The timestamp check limits replays, and the ID check catches replays that arrive inside the window.

For rotation, allow several active secrets per endpoint. During the overlap window, put one signature per secret in the header, for example v1=<sig1> v1=<sig2>. Receivers accept the request if any signature matches. Once the customer confirms the new secret is deployed, retire the old one.

Re-sign each retry with a fresh timestamp so that legitimate retries don't fail the tolerance check. The window still has to absorb clock drift on the receiver's side. Shared-secret HMAC is cheap and simple, but anyone who can verify a signature can also forge one. Asymmetric signatures such as Ed25519 solve that problem at the cost of key distribution and more complex receiver code.

Copyimport base64, hashlib, hmac, time

TOLERANCE = 300  # seconds; must also absorb receiver clock drift

def sign(secret: bytes, msg_id: str, ts: int, body: bytes) -> str:
    # Canonical string: {id}.{timestamp}.{raw body bytes}, never a re-serialized object
    signed = f"{msg_id}.{ts}.".encode() + body
    mac = hmac.new(secret, signed, hashlib.sha256).digest()
    return "v1=" + base64.b64encode(mac).decode()

def sender_headers(secrets: list[bytes], msg_id: str, body: bytes) -> dict:
    # Fresh timestamp on every attempt; one signature per active secret during rotation
    ts = int(time.time())
    sigs = " ".join(sign(s, msg_id, ts, body) for s in secrets)
    return {"Webhook-Id": msg_id, "Webhook-Timestamp": str(ts), "Webhook-Signature": sigs}

seen: dict[str, int] = {}  # msg_id -> ts; use a shared store with a TTL in production

def verify(secrets: list[bytes], headers: dict, raw_body: bytes) -> str:
    # raw_body is the exact request bytes, read before any JSON parsing
    msg_id = headers["Webhook-Id"]
    ts = int(headers["Webhook-Timestamp"])
    received = headers["Webhook-Signature"].split()
    expected = [sign(s, msg_id, ts, raw_body) for s in secrets]
    # Constant-time compare; accept if any received signature matches any active secret
    if not any(hmac.compare_digest(r.encode(), e.encode()) for r in received for e in expected):
        return "reject"
    if abs(time.time() - ts) > TOLERANCE:
        return "reject"  # stale or future-dated: possible replay
    if msg_id in seen:
        return "duplicate"  # replay or retry inside the window: ack with 2xx, skip processing
    seen[msg_id] = ts
    return "process"  # only now parse raw_body as JSON

secrets = [b"example-old-secret-32-bytes-xxxxx", b"example-new-secret-32-bytes-yyyyy"]
body = b'{"event":"invoice.paid","id":"inv_123"}'
h = sender_headers(secrets, "msg_2f9a7c", body)
print(verify(secrets[1:], h, body), verify(secrets[1:], h, body))  # process duplicate

A Python sender that signs {id}.{timestamp}.{body} with HMAC-SHA256 once per active secret, and a receiver that checks the raw bytes in constant time, then applies a five-minute timestamp window and deduplicates on the delivery ID.

Webhook retries: classifying responses and scheduling backoff

The worker's first decision after each attempt is classification. A 2xx is success, and the response body should be ignored. Timeouts, connection resets, DNS failures, 429 and 5xx are retryable, because the receiver may recover. Most other 4xx responses are non-retryable: a 400, 401 or 404 will fail the same way on every attempt, so retrying only adds load and delays the move to the dead letter queue. Some teams treat 408 and 409 as retryable. Whichever choice you make, keep the classification in one table in code so it is not scattered across call sites.

Schedule retries with exponential backoff and full jitter: delay = random(0, min(cap, base × multiplier^n)). The jitter keeps retries from thousands of deliveries from arriving at a recovering endpoint at the same moment. If the response includes Retry-After, use it, clamped to your cap so a hostile or misconfigured value cannot stall the queue. Also enforce a total retry window measured from the first attempt, not only a maximum attempt count.

Illustrative estimate, assuming base 30 s, multiplier 2, cap 1 h and a 24 h window:

Retry aggressiveness: Aggressive retries, with a small base and a low cap, deliver faster after brief blips but put pressure on endpoints that are already struggling. Conservative retries are gentler on receivers but add latency to every recovery.

Window length: A long window, such as 24–72 h, covers weekend outages but holds queue state longer. A short window sends failures to the dead letter queue sooner.

Copyimport random, time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

BASE_S, MULTIPLIER, CAP_S, WINDOW_S = 30, 2, 3600, 24 * 3600
RETRYABLE_4XX = {429}  # add 408/409 here if your policy treats them as transient

def classify(status):
    # status is None for timeouts, connection resets and DNS failures
    if status is None or status >= 500 or status in RETRYABLE_4XX:
        return "retry"
    return "success" if 200 <= status < 300 else "dead_letter"

def parse_retry_after(value):
    # Retry-After is either delta-seconds or an HTTP-date
    if not value:
        return None
    if value.strip().isdigit():
        return float(value.strip())
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None  # unparseable hint: fall back to our own backoff
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())

def next_delay(retry_number, retry_after=None):
    hinted = parse_retry_after(retry_after)
    if hinted is not None:
        return min(hinted, CAP_S)  # clamp so a hostile header cannot stall the queue
    ceiling = min(CAP_S, BASE_S * MULTIPLIER ** (retry_number - 1))
    return random.uniform(0, ceiling)  # full jitter spreads synchronized retries

def decide(first_attempt_at, retry_number, status, retry_after=None):
    outcome = classify(status)
    if outcome != "retry":
        return outcome, None
    run_at = time.time() + next_delay(retry_number, retry_after)
    if run_at - first_attempt_at > WINDOW_S:  # window measured from the first attempt
        return "dead_letter", None
    return "retry", run_at

A Python sketch that classifies delivery outcomes from a single table, computes capped exponential backoff with full jitter, honors a clamped Retry-After header, and sends deliveries to the dead letter queue once the 24-hour window from the first attempt is exceeded.

Delivery semantics: at-least-once, idempotency and ordering

A webhook delivery service can promise at-least-once delivery, not exactly-once. Suppose a worker sends a request, the receiver commits the work, and then the connection drops before the 2xx response arrives. The sender cannot tell this case apart from a request that never landed. The only safe option is to retry, so duplicates follow. No sender-side mechanism over HTTP to a third party can close that gap, because the commit and the acknowledgement happen in a system you don't control.

That makes deduplication the receiver's job, and your contract has to support it:

• Delivery ID. Each event gets one stable ID that stays the same on every retry and on manual replay. Send it in a header and include it in the signed payload. Receivers record it under a unique constraint and skip any ID they have already processed.
• Fast acknowledgement. Receivers should durably enqueue the event, return 2xx within a few seconds, and process it asynchronously. Slow handlers hit your timeout, which triggers a retry, which produces the duplicates they then have to filter.

Ordering is a separate trade-off:

Most services deliver in parallel and document that order is not guaranteed. Give receivers an occurrence timestamp or a per-resource version number so they can discard stale updates. Better still, they can treat the webhook as a notification and fetch the current state from your API.

Isolating endpoints and securing outbound calls

A shared worker pool is only as fast as its slowest destination. If one receiver starts taking 30 seconds to answer, workers pile up waiting on it, and healthy endpoints queue behind a problem they did not cause. Cap concurrent in-flight deliveries per endpoint so one destination can occupy only a bounded slice of workers. For the largest customers, use bulkheads: separate pools or queues, so their volume cannot crowd out everyone else.

Keep timeouts short and separate. A connect timeout of a few seconds catches dead hosts. A read timeout bounds slow responders. Receivers should acknowledge quickly and process asynchronously, so a long wait buys you nothing. Add a circuit breaker per endpoint. After a run of consecutive failures, stop sending and park new events. Probe occasionally, and resume when a probe succeeds. This stops you spending retry budget on a host that is clearly down.

SSRF defenses

Customers choose the URL, so the service will make requests to any address they type unless you stop it. Validating the hostname at registration is not enough. DNS can return a public address then and 10.0.0.5 or 169.254.169.254 later. Resolve the name at send time and check every resolved IP against private, loopback, link-local and other internal ranges. Then connect to the exact IP you checked, so the address cannot change between check and connect.

A few further limits close the remaining gaps:

• Require HTTPS.
• Disable redirects, or re-validate each hop.
• Cap the response body you read, because you only need the status code.

The example below performs the address check inside the connection step.

Copyconst dns = require('node:dns');
const https = require('node:https');
const net = require('node:net');

// Ranges a customer-supplied webhook URL must never reach.
const blocked = new net.BlockList();
for (const [net4, bits] of [['0.0.0.0', 8], ['10.0.0.0', 8], ['100.64.0.0', 10], ['127.0.0.0', 8],
  ['169.254.0.0', 16], ['172.16.0.0', 12], ['192.168.0.0', 16]]) blocked.addSubnet(net4, bits, 'ipv4');
for (const [net6, bits] of [['::1', 128], ['::ffff:0:0', 96], ['fc00::', 7], ['fe80::', 10]])
  blocked.addSubnet(net6, bits, 'ipv6'); // ::ffff:0:0/96 stops IPv4-mapped bypasses

// Runs at connect time. The socket connects to the address we return, so check and connect use the same IP.
function safeLookup(hostname, options, callback) {
  dns.lookup(hostname, { ...options, all: true }, (err, addrs) => {
    if (err) return callback(err);
    const bad = addrs.find((a) => blocked.check(a.address, a.family === 6 ? 'ipv6' : 'ipv4'));
    if (bad) return callback(new Error(`refusing ${hostname}: resolves to ${bad.address}`));
    if (options.all) return callback(null, addrs);
    callback(null, addrs[0].address, addrs[0].family);
  });
}

function deliver(endpointUrl, payload, signature) {
  const url = new URL(endpointUrl);
  if (url.protocol !== 'https:') return Promise.reject(new Error('HTTPS required'));
  return new Promise((resolve, reject) => {
    const req = https.request(url, {
      method: 'POST',
      lookup: safeLookup,
      timeout: 5000, // socket inactivity: bounds dead hosts and slow responders
      headers: { 'content-type': 'application/json', 'webhook-signature': signature },
    }, (res) => {
      res.destroy(); // only the status matters; read no body (Node does not follow redirects, so 3xx is a failure)
      resolve(res.statusCode);
    });
    req.on('timeout', () => req.destroy(new Error('delivery timed out')));
    req.on('error', reject);
    req.end(payload);
  });
}

A Node.js delivery function that requires HTTPS, checks every resolved IP against private and internal ranges inside a custom DNS lookup that runs at connect time, enforces a socket timeout, and discards the response body without following redirects.

Dead letter queue handling, replay and impact on cost and engineering time

When a delivery exhausts its retry budget, move it to the dead letter queue as a complete record. Store the exact payload bytes, the event ID, the signing key ID and every attempt with its timestamp, status code and latency. If you keep only an event ID, a replay has to rebuild a payload whose source data may have changed since.

Alert on how fast the DLQ grows per endpoint, not on its total depth. Otherwise one broken customer can hide a platform-wide failure.

Replays must be deliberate operations:

• Mark each replay with a header so receivers can tell it apart from the first delivery.
• Send replays through the same per-endpoint rate limits as normal traffic.
• Record who triggered the replay and why in an audit log.

Give customers the same inspect-and-replay view. They can then recover on their own after fixing their endpoint.

For endpoints that keep failing, pick one policy and document it:

Illustrative estimate (assumed numbers, not a benchmark). Engineering hours saved per year = T × (B + 12M) − (C + 12O).

Assumed inputs:

• T = 6 teams
• B = 40 h for each team to build its own retries and signing
• M = 4 h/month for each team to maintain it
• C = 160 h to build the central service
• O = 16 h/month to operate it

Result: 6 × 88 − 352 = 176 h saved in year one. This figure leaves out the debugging time saved by having attempt logs and the ad hoc fixes avoided across teams, so it understates the benefit.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to webhook delivery service, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.