What is outbox-based webhook delivery in a monolith, and why use it?

In outbox-based delivery, every webhook starts as a row in your own database. When an order is paid or a user is deleted, the code that changes the business tables also inserts a delivery record (event type, payload, target endpoint, attempt count, next attempt time) in the same transaction. A background worker inside the same application later reads pending rows, signs and sends them, records the outcome, and schedules retries. The table acts as both your webhook queue and the source of truth for what was promised to subscribers.

This gives you two guarantees that are hard to get otherwise. If the transaction commits, the delivery row exists, so a crash between "save order" and "send webhook" cannot lose the event. If the transaction rolls back, the row disappears with it, so subscribers never hear about a change that didn't happen. Sending HTTP calls inline, or publishing to a separate broker after commit, gives you neither guarantee without extra coordination. You also deploy nothing new: no broker, no separate delivery service, and no second datastore to back up.

The pattern fits teams running one application and one primary database, with webhook volume small enough that polling a table doesn't strain it.

The delivery semantics are at-least-once. A receiver can process a request and then time out before its response reaches you. You can't distinguish that case from a failure, so you have to retry. Exactly-once delivery to an endpoint you don't control isn't achievable over HTTP. Instead, give every event a stable ID that receivers can use to deduplicate.

+----------------- Monolith application (one deployable) ------------------+
|                                                                          |
|  +----------------------+                  +--------------------------+  |
|  | Business code        |                  | Delivery worker          |  |
|  | e.g. order paid,     |                  | 1. poll due rows         |  |
|  | user deleted, ...    |                  | 2. HMAC-sign body        |  |
|  +----------------------+                  | 3. POST w/ event ID      |  |
|             |                              | 4. log status            |  |
|             |                              | 5. retry w/ backoff      |  |
|             |                              +--------------------------+  |
|             |                                     |             |        |
+-------------|-------------------------------------|-------------|--------+
              | INSERT outbox row                   ^             |
              | in the same txn         poll/update | signed POST |
              v                                     v             v
+------------------- Primary database -------------------+  +--------------+
| business tables: orders, users, ...                    |  | Subscriber   |
| webhook_deliveries (outbox): event_id, endpoint,       |  | endpoints    |
|   payload, attempts, next_attempt_at, status           |  | dedupe by ID |
+--------------------------------------------------------+  +--------------+

COMMIT: state change and outbox row persist together. ROLLBACK: neither.
Delivery is at-least-once; receivers dedupe on the stable event_id.

Business code writes the state change and the outbox row in one transaction, and an in-process worker polls that table to sign, send, log and retry each delivery.

Who should use it?

This design suits a specific kind of team and system:

  • Teams running a single deployable application against one primary relational database. The pattern depends on the business write and the outbox insert sharing a transaction. If your state is already split across services and databases, you lose that property.
  • Products that send webhooks to customers or partners where a missed or phantom event causes real support work or data drift, such as billing, order, subscription or account lifecycle events.
  • Small to mid-sized teams that don't want to operate a broker. If nobody on the team is set up to run, monitor and upgrade a message cluster, keeping delivery inside the existing app and database avoids that on-call burden.
  • Systems with moderate webhook volume. Polling a table and writing attempt rows adds load to the primary. If that load is small relative to user traffic, the pattern is a good fit. If it is not, see the section on when to avoid it.

It is not a good fit if you need very high delivery throughput, many independent consumers of the same events, or delivery workers that scale and deploy independently of the web tier.

How does it work?

At a high level, each webhook moves through five steps:

  1. Record. Business code writes its state change and, in the same transaction, inserts an event row and one pending delivery row per subscribed endpoint.
  2. Claim. A delivery worker inside the application polls for rows that are due, takes a short time-bounded lease on a batch, and commits the lease before doing any network I/O.
  3. Sign and send. For each claimed row, the worker signs the exact body bytes with the endpoint's HMAC secret, adds the event ID, timestamp and signature headers, and POSTs to the endpoint with strict timeouts and SSRF protections.
  4. Log and classify. The worker writes an attempt row with the request, response and error class, then classifies the outcome: delivered, permanently failed, endpoint disabled, or retryable.
  5. Reschedule or finish. Retryable failures get a new next_attempt_at from capped exponential backoff with jitter, until the retry window runs out. Lease expiry lets another worker pick up rows abandoned by a crashed worker.

The following sections cover each step in detail: the tables and claim query, retry classification and scheduling, signing and rotation, logging, and worker isolation.

How does the transactional outbox and database webhook queue work?

Four tables carry the design. webhook_events stores the immutable payload, event type and stable ID. webhook_endpoints holds each subscriber's URL, subscribed event types, signing secret reference and an enabled flag. webhook_deliveries has one row per event/endpoint pair, with status, attempt_count, next_attempt_at and lease fields. webhook_attempts is append-only, with one row per HTTP try.

When an order is placed, the same transaction that writes the order also inserts the event and one pending delivery row for each matching endpoint. If the transaction rolls back, no webhook exists. If it commits, the delivery is durable. This closes the gap you get when you publish after commit and the process crashes in between.

Workers poll for due rows. One option is SELECT ... FOR UPDATE SKIP LOCKED with next_attempt_at <= now() and a small LIMIT. Concurrent workers then skip each other's rows instead of blocking on them. Doing the HTTP call while holding that row lock keeps a transaction open for the whole request, though, so a lease is usually safer. A worker sets locked_by and locked_until in a short transaction, commits, and only then makes the call. A row whose locked_until has passed counts as due again, which is how work held by crashed workers gets reclaimed. Keep the lease longer than your HTTP timeout so a slow but live worker doesn't lose its row to a second worker.

All of this costs the primary database. Each event multiplies writes by its endpoint count, every attempt adds rows, and polling adds index scans and contention on the hot rows. Index (status, next_attempt_at) and keep the polling interval reasonable rather than aggressive.

-- PostgreSQL. 1) Inside the business transaction: order, event and deliveries commit or roll back together.
BEGIN;
INSERT INTO orders (id, customer_id, total_cents)
VALUES ('5f0c1e2a-0000-4000-8000-000000000001', 42, 1999);

INSERT INTO webhook_events (id, event_type, payload)
VALUES ('9b7d3c10-0000-4000-8000-000000000002', 'order.placed',
        '{"order_id": "5f0c1e2a-0000-4000-8000-000000000001"}');

-- One pending delivery per enabled endpoint subscribed to this event type.
INSERT INTO webhook_deliveries (event_id, endpoint_id, status, attempt_count, next_attempt_at)
SELECT '9b7d3c10-0000-4000-8000-000000000002', e.id, 'pending', 0, now()
FROM webhook_endpoints e
WHERE e.enabled AND 'order.placed' = ANY (e.event_types);
COMMIT;

-- 2) Worker claim: a short transaction that only takes a lease, then commits.
--    The HTTP call happens after commit, so no row lock is held during the request.
UPDATE webhook_deliveries d
SET locked_by    = 'worker-7',
    locked_until = now() + interval '90 seconds'  -- longer than the HTTP timeout
WHERE d.id IN (
  SELECT id
  FROM webhook_deliveries
  WHERE status = 'pending'
    AND next_attempt_at <= now()
    -- Unleased rows, or leases left behind by crashed workers.
    AND (locked_until IS NULL OR locked_until < now())
  ORDER BY next_attempt_at
  LIMIT 10
  FOR UPDATE SKIP LOCKED  -- concurrent workers skip rows another claim is touching
)
RETURNING d.id, d.event_id, d.endpoint_id, d.attempt_count;

-- Supports the polling predicate above.
CREATE INDEX webhook_deliveries_due_idx
  ON webhook_deliveries (status, next_attempt_at);

Inserting an event and its pending deliveries in the same transaction as the order, then claiming due or expired-lease deliveries with a short FOR UPDATE SKIP LOCKED lease update in PostgreSQL.

How should webhook delivery retries be scheduled and classified?

Once a row is due, the worker has to decide whether the failure is one that time can fix. Classify the outcome before scheduling anything:

Outcome Action Reason
2xx Mark delivered The receiver accepted it. Ignore the body.
410 Gone Stop and disable the endpoint The receiver has retired the URL.
Other 4xx (400, 401, 403, 404, 422) Stop and mark failed Resending the same bytes gets the same answer.
408, 429 Retry The receiver is busy, not rejecting the event.
5xx, timeout, connection error Retry The failure is likely transient.

For retries, compute min(base × 2^attempt, cap) and then apply full jitter by picking a random delay between zero and that value. Jitter matters because many deliveries fail together during an outage. Without it, they all return at the same moment and hit the endpoint as it recovers. If the receiver sends Retry-After, honor it, but clamp it to your floor and cap so the receiver cannot stall a row indefinitely. Enforce the total window on elapsed time since the event was created, not on attempt count.

Illustrative estimate: assume a 30-second base, a 6-hour cap and a 24-hour window. Before jitter, the gaps run 30s, 1m, 2m, 4m … 256m, then 6h, 6h. That totals about 13 attempts before the event is abandoned at 24 hours.

Long windows have a cost. An event delivered 20 hours late may describe state that no longer exists. Tell consumers to treat payloads as notifications and refetch the current state. Strict per-endpoint ordering has a cost too: one failing event blocks every event queued behind it. Offer ordering only where consumers truly need it.

import random
from datetime import timedelta
from email.utils import parsedate_to_datetime

BASE, CAP, FLOOR = 30, 6 * 3600, 30   # seconds
WINDOW = timedelta(hours=24)           # measured from event creation

def classify(status, timed_out=False):
    if timed_out or status is None:    # timeout or connection error
        return "retry"
    if 200 <= status < 300:
        return "delivered"             # ignore the response body
    if status == 410:
        return "disable_endpoint"      # receiver retired the URL
    if status in (408, 429) or status >= 500:
        return "retry"                 # busy or transient
    return "failed"                    # same bytes, same answer

def retry_after_seconds(header, now):
    if not header:
        return None
    if header.strip().isdigit():       # delta-seconds form
        return int(header)
    try:                               # HTTP-date form
        return (parsedate_to_datetime(header) - now).total_seconds()
    except (TypeError, ValueError):
        return None                    # unparseable: fall back to backoff

def next_attempt_at(attempt, created_at, now, retry_after=None):
    # Full jitter: uniform between 0 and the capped exponential delay.
    delay = random.uniform(0, min(BASE * 2 ** attempt, CAP))
    hinted = retry_after_seconds(retry_after, now)
    if hinted is not None:
        # Honor the receiver, but clamp so it cannot stall the row forever.
        delay = min(max(hinted, FLOOR), CAP)
    due = now + timedelta(seconds=delay)
    if due - created_at > WINDOW:
        return None                    # abandon: retry window exhausted
    return due

A worker-side sketch that classifies a delivery outcome and computes the next attempt time using capped exponential backoff with full jitter, a clamped Retry-After, and a 24-hour window measured from event creation (now and created_at are timezone-aware UTC datetimes).

How do you sign webhooks with an HMAC signature and rotate secrets?

Generate a random secret per endpoint when the subscription is created, and store it encrypted rather than hashed, because the worker needs the plaintext to sign. To sign a delivery, compute HMAC-SHA256 over the Unix timestamp, a separator such as ., and the exact body bytes you are about to send. Signing the raw bytes matters. If the receiver parses the JSON and re-serializes it, key order or whitespace can change, and the signature will no longer match.

Send the result in headers. Choose your own names; for example, Webhook-Signature and Webhook-Timestamp. Include the stable event ID and the attempt number as well. The event ID lets receivers deduplicate retries. The attempt number helps both sides debug, and it should not be part of the signed content's identity.

Receiver verification

The receiver should take these steps in order:

  1. Read the raw body before any framework parses it.
  2. Recompute the HMAC over the timestamp and the raw body.
  3. Compare it to the received signature with a constant-time function, so the time a comparison takes reveals nothing about how many bytes matched.
  4. Reject the request if the timestamp is more than a few minutes away from local time. Because the timestamp is signed, an attacker cannot refresh it, which limits replay to that window. Event-ID deduplication closes the rest of the window.

Rotation

During rotation, keep two secrets active and put one signature per secret in the header, such as v1=<sig1>,v1=<sig2>. Receivers accept the request if any signature verifies. That lets them switch to the new secret on their own schedule. Then retire the old secret.

Limits

This scheme assumes the sender's and receiver's clocks are reasonably synchronized. It also cannot help after a secret leaks: anyone holding the secret can forge valid requests until you rotate it.

import hashlib
import hmac
import time

TOLERANCE_SECONDS = 300  # reject timestamps more than 5 minutes from local time

def sign(secret: bytes, timestamp: int, body: bytes) -> str:
    # Sign the timestamp, a '.' separator and the exact body bytes sent on the wire.
    msg = str(timestamp).encode() + b"." + body
    return hmac.new(secret, msg, hashlib.sha256).hexdigest()

def build_signature_header(active_secrets: list[bytes], timestamp: int, body: bytes) -> str:
    # During rotation, emit one v1 signature per active secret: "v1=<sig1>,v1=<sig2>".
    return ",".join("v1=" + sign(s, timestamp, body) for s in active_secrets)

def verify(raw_body: bytes, sig_header: str, ts_header: str, secrets: list[bytes]) -> bool:
    # raw_body must be the unparsed request bytes, read before any JSON parsing.
    try:
        timestamp = int(ts_header)
    except ValueError:
        return False
    # The timestamp is signed, so an attacker cannot refresh it; this bounds replay.
    if abs(time.time() - timestamp) > TOLERANCE_SECONDS:
        return False
    received = [p[3:] for p in sig_header.split(",") if p.startswith("v1=")]
    for secret in secrets:  # receiver may hold old and new secret during rotation
        expected = sign(secret, timestamp, raw_body)
        for candidate in received:
            # Constant-time comparison: timing reveals nothing about matching bytes.
            if hmac.compare_digest(expected, candidate):
                return True
    return False

old, new = b"whsec_placeholder_old", b"whsec_placeholder_new"
body = b'{"id":"evt_123","type":"invoice.paid"}'
ts = int(time.time())
header = build_signature_header([old, new], ts, body)
print(verify(body, header, str(ts), [new]))  # True: receiver already switched to new secret

Python sketch of HMAC-SHA256 signing over timestamp plus raw body, emitting one signature per active secret for rotation, and receiver verification with a timestamp tolerance and constant-time comparison.

What should delivery logs record, and how long should you keep them?

Log every attempt, not just every event. A webhook that succeeded on its fourth try has three failures worth seeing, and those failures explain why the customer's endpoint was slow last Tuesday.

For each attempt, record:

  • Request headers as sent, including the signature and timestamp headers.
  • Request body, truncated to a fixed cap.
  • Response status, or null if no response arrived.
  • Response body, truncated, because error pages are often large HTML.
  • Latency, from connection start to the last byte or the timeout.
  • Error class, such as timeout, DNS failure, TLS error, connection refused or non-2xx. Use the same classification the retry scheduler uses so the log explains each scheduling decision.

Show these logs to customers in their dashboard, next to a manual Redeliver button. The button should enqueue a fresh attempt through the normal outbox path, with a new timestamp and signature. Many "did you send it?" tickets close without engineering involvement once customers can see the 500 their own server returned.

Redact data before you write it, not when you display it. Payloads can carry personal data or tokens, so apply the same field-level rules you use for application logs. Never log signing secrets.

Data Illustrative retention Rationale
Full attempt (headers, bodies) 7–30 days Needed to debug recent failures. Bodies take up most of the storage.
Metadata (event ID, status, latency, error class) 90+ days Rows are small. Useful for trend analysis and disputes.

Keeping two tiers gives you diagnosability without unbounded growth. Partition the log tables by time so expiry means dropping old partitions instead of running large DELETE operations against a busy table.

How do you protect workers from bad endpoints and SSRF?

Track endpoint health and disable deliberately

Keep a per-endpoint counter of consecutive failures and the timestamp of the first failure in the current streak. When an endpoint has failed continuously for a sustained period, such as several days rather than a fixed number of attempts, disable it and notify the customer by email and in the dashboard. A time-based threshold avoids punishing a busy endpoint for one bad hour. The trade-off is real. If the notification is missed, the integration stops silently. That is still better than retrying into a dead URL forever, but document the behaviour and make re-enabling a single action.

Isolate slow endpoints

A webhook worker blocked on a slow endpoint is a worker not doing anything else. Run webhook delivery in its own worker pool, separate from email, billing and other background jobs. Within that pool, cap concurrent deliveries per endpoint or tenant. Without these limits, one tenant's hanging server, or your own bad deploy that makes every request time out, can consume every worker and back up unrelated queues.

Make outbound requests safe

Customer-supplied URLs are an SSRF vector, so apply these defaults:

  • Strict timeouts: Set short connect and read timeouts, bounded by a total deadline.
  • No redirects by default: A redirect can point at an internal host.
  • Response size limits: Read a few kilobytes for the log, then stop.
  • Address blocking at connect time: Block private, loopback and link-local ranges, including the cloud metadata address. Check the resolved IP when the connection is made, not when the URL is saved. Otherwise DNS rebinding can get around the check.

How can it save engineering time?

Most of the time saved comes from problems you don't have to debug or build around:

  • No lost or phantom events to reconcile. Because the outbox row commits with the business change, you don't write reconciliation jobs to find orders that were saved but never announced, or webhooks sent for transactions that later rolled back. Those bugs are expensive to investigate because they are intermittent and leave little trace.
  • No new infrastructure to stand up. There is no broker to provision, secure, monitor, upgrade or back up, and no second deploy pipeline. Delivery code ships with the application, uses the same database migrations, and is covered by the same tests and observability you already have.
  • Faster local development and testing. A developer can run the whole flow, from business write to signed POST, with just the app and its database. Integration tests can assert on outbox rows inside a transaction instead of mocking a broker client.
  • Fewer support escalations. Per-attempt logs and a self-service Redeliver button let customers answer "did you send it?" and "why did it fail?" themselves, instead of opening tickets that an engineer has to trace through application logs.
  • Predictable retry behaviour. Centralizing classification, backoff and endpoint disabling in one worker means individual features don't each implement their own ad hoc retry loops, and on-call engineers have one place to look when deliveries back up.
  • A cheap migration path. If you later outgrow the pattern, producers don't change. Only the relay changes, which limits the rework to the delivery side.

Illustrative estimate: assume a team handles 10 webhook support tickets a month, each taking an engineer 30 minutes to trace through logs, and assume customer-visible attempt logs let half of them be resolved without engineering. That frees about 2.5 engineer-hours a month on tickets alone, before counting time not spent operating separate infrastructure. Substitute your own ticket volume and handling time; the point is that the savings scale with how often people currently have to ask an engineer what happened to a delivery.

When should you avoid in-monolith webhooks, and what are the alternatives?

The outbox pattern stops being a good fit when the webhook queue competes with your product for the primary database. Each delivery attempt is an insert, several updates, and a log row, all on the same primary that serves user traffic. At high volume this shows up as table and index bloat, longer vacuum or maintenance cycles, and workers claiming rows with row locks while request handlers write to the same tables. Consider moving off the pattern if delivery workers need to scale separately from the web tier, or if a webhook bug should not require redeploying the whole application.

Approach Delivery guarantee Build cost Run cost Fits when
In-monolith outbox At-least-once, atomic with business write Low Low, shares existing DB Moderate volume, small team
Message broker At-least-once if fed from an outbox Medium Cluster to operate and monitor High throughput, many consumers
Separate delivery service At-least-once, own storage High New service, deploys, on-call Webhooks are a product in their own right
Fire-and-forget from handler None, lost on crash or timeout Very low Low Never for customer-facing events

Fire-and-forget looks cheap. It is not, because the failures it causes are silent and customers report them before you notice.

Early on, the outbox's main advantage is one codebase and one transaction boundary. You avoid dual writes, extra infrastructure, and a second deploy pipeline. You also keep a clean exit. The outbox table stays as the source of truth, and its relay switches from calling endpoints to publishing to a broker. Delivery workers then move to consumers without any change to how events are produced.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to webhook delivery retries, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.