What is in-monolith webhook delivery, and why decouple it from the request path?
In-monolith webhook delivery means the application that owns the data also sends the webhooks, without a separate messaging service. It follows the transactional outbox pattern. When a request changes business state, the same database transaction inserts an event row describing that change. Background workers later read those rows, expand them into one delivery per subscribed endpoint, and send them over HTTP.
The alternative is calling the receiver from inside the request handler, which fails in two ways:
- Lost events. The transaction commits, then the process crashes or is redeployed before the HTTP call finishes. The change is permanent and nothing remembers that a webhook was owed.
- Phantom events. The webhook is sent first, then the transaction rolls back. The receiver has been told about something that never happened.
The handler also slows down. A receiver that takes ten seconds to respond holds the user's request and a database connection for those ten seconds.
request
|
v
+----------------- one transaction ------------------+
| UPDATE orders ... INSERT INTO events ... |
+----------------------------------------------------+
|
v (fan out per subscription)
deliveries table
|
v (claim due rows)
worker pool ----HTTP POST----> receiver
|
v
delivery_attempts (log of each try)
The pattern provides an at-least-once guarantee. Every committed change produces a delivery that will keep being retried. A crash between sending the request and recording the result means a receiver can see the same event twice, so receivers must deduplicate by event ID.
This approach suits teams with a single relational database and modest to moderate event volume who want reliable delivery without operating a broker.
client request
|
v
+--------------------- one DB transaction ----------------------+
| UPDATE orders SET ... INSERT INTO events (...) |
| (business row) (event row, same commit) |
+---------------------------------------------------------------+
| commit: both rows exist, or neither does
v
+------------------+ fan out: one row per +--------------------+
| events (outbox) |----subscribed endpoint-->| deliveries |
+------------------+ | status, attempts, |
| next_attempt_at |
+--------------------+
| ^
claim due rows | | reschedule
v | on failure
+--------------------+
| worker pool |
| w1 w2 w3 ... |
+--------------------+
| |
HTTP POST (signed) | | record
v v every try
+-----------+ +-------------------+
| receiver | | delivery_attempts |
| (dedup by | | status code, |
| event id) | | latency, error |
+-----------+ +-------------------+
The business change and its event row commit in one transaction, then the event fans out into one delivery per subscription, which workers claim, POST to the receiver, log in delivery_attempts, and reschedule on failure, so every committed change is delivered at least once.
How do you design the webhook queue tables and claim work safely?
Use two tables. webhook_events holds one row per business event: ID, type, payload and creation time. webhook_deliveries holds one row per event per subscribed endpoint, with these columns:
statusattempt_countnext_attempt_atlast_errorlocked_byandlocked_untilfor leases
Splitting the tables matters because each endpoint fails and retries independently. A dead subscriber should not hold back a healthy one.
Write the event and its delivery rows in the same transaction as the business change. If the order commits, its webhooks exist. If it rolls back, they never existed. This is the outbox pattern, and it removes the dual-write gap that comes from publishing after commit.
Workers need to find due rows cheaply. In PostgreSQL, create a partial index on next_attempt_at that covers only pending rows, so delivered history doesn't bloat it. Where partial indexes aren't available, use a composite index on (status, next_attempt_at).
To claim work, select pending rows where next_attempt_at is at or before now, ordered by that column, with a small limit. Use FOR UPDATE SKIP LOCKED where your database supports it. Concurrent workers then skip each other's rows instead of blocking. Inside that transaction, write a lease (locked_by, locked_until) and commit before making the HTTP call. Holding a row lock across a network request ties up connections.
The lease also covers crashes. A claim query that treats expired leases as available lets a dead worker's jobs reappear automatically. Set the lease longer than your HTTP timeout so a slow request isn't delivered twice concurrently.
This design has an ongoing cost. Every worker polls, and every claim writes rows, which adds query load, lock churn and table bloat. Tune the poll interval and batch size against your latency needs.
CREATE TABLE webhook_events (
id bigserial PRIMARY KEY,
event_type text NOT NULL,
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now());
CREATE TABLE webhook_deliveries ( -- one row per event per endpoint
id bigserial PRIMARY KEY,
event_id bigint NOT NULL REFERENCES webhook_events(id),
endpoint_id bigint NOT NULL,
status text NOT NULL DEFAULT 'pending', -- pending | delivered | dead
attempt_count int NOT NULL DEFAULT 0,
next_attempt_at timestamptz NOT NULL DEFAULT now(),
last_error text,
locked_by text, locked_until timestamptz); -- lease fields
-- Partial index: only claimable rows, so delivered history doesn't bloat it.
CREATE INDEX webhook_deliveries_due ON webhook_deliveries (next_attempt_at) WHERE status = 'pending';
-- Outbox: event + deliveries commit or roll back with the business change.
BEGIN;
UPDATE orders SET status = 'paid' WHERE id = 42;
WITH ev AS (
INSERT INTO webhook_events (event_type, payload)
VALUES ('order.paid', '{"order_id": 42}') RETURNING id)
INSERT INTO webhook_deliveries (event_id, endpoint_id)
SELECT ev.id, s.endpoint_id FROM ev
JOIN webhook_subscriptions s ON s.event_type = 'order.paid'; -- assumed subscriptions table
COMMIT;
-- Claim (autocommits before any HTTP call). SKIP LOCKED avoids blocking on
-- rows another worker is claiming; expired leases make crashed work reappear.
UPDATE webhook_deliveries d
SET locked_by = 'worker-a', locked_until = now() + interval '2 minutes' -- > HTTP timeout
WHERE d.id IN (
SELECT id FROM webhook_deliveries
WHERE status = 'pending' AND next_attempt_at <= now()
AND (locked_until IS NULL OR locked_until < now())
ORDER BY next_attempt_at LIMIT 10
FOR UPDATE SKIP LOCKED)
RETURNING d.id, d.event_id, d.endpoint_id, d.attempt_count;
PostgreSQL schema for events and per-endpoint deliveries with a partial due-row index, an outbox insert in the business transaction, and a SKIP LOCKED claim that writes an expiring lease.
How should webhook delivery retries handle failures, backoff and Retry-After?
Every attempt ends in one of three outcomes.
| Outcome | Examples | Action |
|---|---|---|
| Success | Any 2xx | Mark delivered |
| Retryable | Timeout, connection refused, TLS error, 5xx, 429 | Reschedule |
| Terminal | 400, 401, 403, 404, 410, other 4xx | Mark failed, alert |
Terminal 4xx responses mean the receiver rejected the request itself, so sending it again returns the same answer. If a 429 or 503 response carries Retry-After, use that value as the next delay and clamp it to your cap. Otherwise, compute random(0, min(cap, base × 2^attempt)). This is capped exponential backoff with full jitter. The jitter spreads retries out so that endpoints that failed together do not all come back at the same moment. Stop after a fixed number of attempts or a fixed time window, whichever comes first.
Illustrative estimate: assume a 30 s base, a 1 h cap and 10 attempts. Before jitter, the nine waits add up to about 11,000 s, or roughly 3 hours. Full jitter brings the expected span down to about 1.5 hours. A longer window loses fewer events, but a receiver recovering from an outage then gets a burst of stale events. A shorter window means more events need a manual replay.
Delivery is at-least-once. If a receiver commits the event and the connection drops before your worker reads the 2xx, you will send that event again, so HTTP cannot guarantee exactly-once delivery. Put a stable event ID in both the payload and a header so receivers can deduplicate. Retries also make strict per-endpoint ordering impractical, because one stuck event would block every event behind it. Include timestamps or sequence numbers instead, and let receivers discard updates older than what they already have.
import random
import time
from email.utils import parsedate_to_datetime
BASE_S, CAP_S = 30, 3600 # 30 s base delay, 1 h cap per wait
MAX_ATTEMPTS = 10 # stop after 10 attempts...
MAX_WINDOW_S = 24 * 3600 # ...or 24 h since the first attempt
def classify(status):
# status is None for timeouts, refused connections and TLS errors
if status is None or status == 429 or status >= 500:
return "retry"
if 200 <= status < 300:
return "success"
return "terminal" # receiver rejected the request; resending won't help
def parse_retry_after(value):
# Retry-After is either delta-seconds or an HTTP-date
if not value:
return None
value = value.strip()
if value.isdigit():
return int(value)
try:
when = parsedate_to_datetime(value)
except (TypeError, ValueError):
return None # unparseable hint: fall back to backoff
return max(0.0, when.timestamp() - time.time())
def next_delay(attempt, retry_after=None):
hinted = parse_retry_after(retry_after)
if hinted is not None:
return min(hinted, CAP_S) # honor the receiver, but never past our cap
# Full jitter: uniform over [0, capped exponential]
return random.uniform(0, min(CAP_S, BASE_S * 2 ** attempt))
def should_give_up(attempt, first_attempt_at):
# attempt is 0-based; whichever limit is hit first marks the delivery failed
return (attempt + 1 >= MAX_ATTEMPTS
or time.time() - first_attempt_at >= MAX_WINDOW_S)
A minimal Python retry policy that classifies outcomes, honors Retry-After in both seconds and HTTP-date form, applies capped exponential backoff with full jitter, and stops at an attempt or time limit.
How does HMAC signing of outgoing webhooks work?
Ordering tells receivers what to apply. Signing tells them who sent it. When an endpoint is registered, generate a random secret for it (32 bytes from a CSPRNG is plenty), store it encrypted, and show it to the customer once. A per-endpoint secret limits the damage of a leak to a single receiver.
At send time, build the signed string as the Unix timestamp, a period, and the exact raw body bytes. Compute HMAC-SHA256 over that string and send two headers: the timestamp, and the signature written as v1=<hex digest>. The v1= prefix lets you add a new algorithm or canonical format later without breaking receivers that only understand the old one. The worker must serialize the payload once, sign those bytes, and send the same buffer. If it signs one serialization and the HTTP client sends another, the signature will not match.
Publish these verification rules for receivers:
- Verify the raw body before parsing JSON. This is the most common integration failure. A framework that parses and re-serializes the body changes key order or whitespace, and the signature breaks.
- Compare digests in constant time, using the comparison function your language provides for this purpose. A plain string equality check can leak how much of the signature matched.
- Reject stale timestamps outside a tolerance window, such as five minutes. Including the timestamp in the signed string stops an attacker from replaying a captured request with a fresh timestamp.
Rotation
To rotate a secret, keep the old and new secrets active at the same time and send one signature per secret in the header, for example v1=abc..., v1=def.... Receivers accept the request if any signature verifies. Once the customer confirms they have deployed the new secret, retire the old one.
import hashlib, hmac, json, secrets, time
# At endpoint registration: 32 random bytes from a CSPRNG, stored encrypted
new_endpoint_secret = secrets.token_bytes(32)
def sign(key: bytes, ts: str, body: bytes) -> str:
# Signed string = timestamp + "." + exact raw body bytes
mac = hmac.new(key, ts.encode() + b"." + body, hashlib.sha256)
return "v1=" + mac.hexdigest()
# --- Sender: serialize once, sign those bytes, send that same buffer ---
def build_request(payload: dict, keys: list[bytes]):
body = json.dumps(payload, separators=(",", ":")).encode()
ts = str(int(time.time()))
# During rotation keys = [new, old]: one signature per active secret
sig = ", ".join(sign(k, ts, body) for k in keys)
headers = {"Content-Type": "application/json",
"Webhook-Timestamp": ts, "Webhook-Signature": sig}
return headers, body # give these exact bytes to the HTTP client
# --- Receiver guidance: verify raw bytes BEFORE parsing JSON ---
TOLERANCE_SECONDS = 300
def verify(key: bytes, headers: dict, raw_body: bytes) -> dict:
ts = headers["Webhook-Timestamp"]
if abs(time.time() - int(ts)) > TOLERANCE_SECONDS:
raise ValueError("stale timestamp") # limits replay
expected = sign(key, ts, raw_body)
candidates = [c.strip() for c in headers["Webhook-Signature"].split(",")]
# Accept if any v1 signature matches; constant-time comparison
if not any(c.startswith("v1=") and hmac.compare_digest(c, expected)
for c in candidates):
raise ValueError("bad signature")
return json.loads(raw_body) # parse only after verification succeeds
A minimal Python sender that signs the exact bytes it sends with one v1 signature per active secret, and a receiver that checks the timestamp window and compares signatures in constant time against the raw body before parsing JSON.
What should delivery logs record, and how do operators redeliver?
Write one log row per attempt, not per event. A row should record:
- Attempt number and the delivery ID it belongs to, so a sequence of failures reads as one story.
- Request headers as sent, with the signature header and any credentials replaced by a fixed marker such as
[redacted]. Customers need to see that a signature was present, not its value. - Response status, plus the first few kilobytes of the response body. Truncate it, because a misconfigured endpoint can return an entire HTML error page.
- Latency, measured from connection start to the last byte read.
- Error class: DNS failure, connection refused, TLS error, timeout, or non-2xx status. A short, fixed set of classes is easier to filter and alert on than raw exception text.
Store the payload once per event rather than once per attempt. Run sensitive fields through the same redaction rules you use in application logs before the payload is written. Logs grow with attempts multiplied by endpoints, so set a retention window and delete old rows on a schedule. A short window, such as a few weeks, usually covers how long a customer takes to notice a broken integration. Partitioning the table by time makes these deletes cheap.
The redeliver action, available to operators and customers, creates a new delivery row that carries the original event ID. Receivers that deduplicate on event ID stay safe, and the history of the original delivery stays intact.
When customers can see "timeout after 10s" or "401 from your server" themselves, they fix their own endpoint. Without that visibility, the same problem reaches your engineers as a vague support ticket.
How do you protect the monolith from slow or hostile endpoints?
Every outbound call is untrusted I/O, so bound it on every axis. Set a short connect timeout (a few seconds) and a separate total read timeout. A read timeout alone won't stop an endpoint that trickles one byte per second. Cap the response body you read, say 64 KB, and discard the rest. You only need the status code and a snippet for the delivery log.
Isolate capacity
Run delivery in a dedicated worker pool with its own database connection pool, sized well below what core request handling needs. During a backlog, this is the main risk to the rest of the system. If thousands of deliveries time out together, shared workers sit blocked on sockets. Retries rewrite queue rows, and claim queries compete for locks and connections with checkout and login. A separate pool turns that into a slower webhook queue rather than an outage.
Add a per-endpoint concurrency limit, often one to four in-flight requests, so one slow customer cannot occupy every worker. When an endpoint fails continuously for a set window, such as several days, disable it automatically and notify the owner. Retrying a dead URL forever only spends capacity.
Block SSRF
Customers choose the URL, so treat it as an attack on your network. Before connecting, resolve the hostname and reject any address that is:
- private
- loopback
- link-local, including cloud metadata addresses
- inside your internal ranges
Connect to the IP you validated so DNS cannot change between the check and the request. Disable redirects, or re-validate every hop.
When should you avoid this approach, and what are the alternatives?
The database queue stops being the right tool in three situations:
- High sustained volume. Workers polling and taking row locks compete with checkout, login and every other query for connections, I/O and vacuum or purge capacity.
- Many independent producers. Once several services emit events, each one would rebuild its own tables, workers and delivery logs.
- Strict isolation requirements. If no third-party traffic may originate from hosts that hold customer data, delivery has to run elsewhere.
| Approach | Transactional guarantee | Operational cost | Fits when |
|---|---|---|---|
| In-request HTTP call | None: event can be lost or sent for a rolled-back write | None | Prototypes only |
| Outbox + in-process workers | Yes | Low: tables, workers, logs, small UI | One app, moderate volume |
| Outbox relayed to a message broker | Yes, up to the relay | Medium: broker to run and monitor | Volume or fan-out outgrows polling |
| Separate webhook delivery service | Yes, if fed from an outbox | High: new service, deploys, on-call | Many producers, isolation needed |
| Asymmetric signatures instead of HMAC | Not affected | Key rotation and publishing a public key | Receivers should verify without holding a secret |
The outbox with in-process workers is modest upfront work: a schema, a worker loop, log retention and a redelivery screen. A broker or a dedicated service adds infrastructure you must operate indefinitely.
You can take a staged path. Start with the outbox now. Later, if volume demands it, replace the in-process workers with a relay that publishes outbox rows to a broker. Producers keep writing events in the same transaction as the business change, so the guarantee holds and application code does not need to change. The migration is limited to the delivery side.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to webhook delivery retries, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.