What is reliable webhook delivery, and why is it a contract between sender and receiver?
A webhook is an HTTP request your system sends when something happens, such as an invoice being paid or a build finishing. Reliable delivery means every event eventually reaches the receiver at least once. It does not mean exactly once, because exactly-once delivery is not achievable over HTTP. Suppose the sender posts an event, the receiver processes it, and the connection drops before the 2xx response arrives. The sender cannot tell this apart from a request that never landed. Its only safe choice is to send the event again. The receiver will sometimes see the same event twice, and nothing in the protocol prevents it.
This is why webhook delivery is a contract. The sender has two obligations. It must keep retrying until it gets an acknowledgment, and it must attach a stable, unique event ID that stays the same across every attempt. The receiver must be idempotent. It records which IDs it has processed and treats a repeat as a no-op that still returns success.
Signing, retries and replay protection depend on each other:
- Retries create duplicates.
- Deduplication needs event IDs to catch those duplicates.
- Replay protection needs a signed timestamp. The signature proves the timestamp wasn't altered, so the receiver can reject an old, captured request instead of trusting it.
If one piece is missing, the others stop working.
This guide is for two groups. The first is teams sending webhooks to third parties or internal services. The second is teams consuming them, who must hold up the receiver's side of the contract.
How does HMAC signature verification work for webhooks?
The sender and receiver agree on a shared secret. For each delivery, the sender takes the current Unix timestamp, joins it to the exact request body with a period (timestamp.body), and computes HMAC-SHA256 over that string with the secret. It sends the timestamp in one header and the signature in another, for example Webhook-Timestamp and Webhook-Signature. The timestamp goes inside the signed string so an attacker cannot change it without breaking the signature.
Prefix the signature value with a scheme version, such as v1= followed by the hex digest. The version tells the receiver which algorithm and string format produced the value. That lets you move to a new algorithm later, or send two signatures during a transition, without breaking existing consumers.
The receiver repeats the computation and compares the result. It has to do this on the raw request bytes before any JSON parsing. Re-serializing a parsed object almost never reproduces the original bytes: key order, whitespace, Unicode escaping and number formatting can all change. Use a constant-time comparison function. A plain string equality check can return early on the first mismatched byte, and that timing difference can leak information about the correct signature.
Framework middleware that parses, decompresses or normalizes the body before your handler runs is one of the most common causes of verification failures. It is also hard to diagnose, because the logged payload looks correct. Configure the webhook route to keep the raw body available.
HMAC proves the message came from someone holding the secret and was not modified in transit. It does not encrypt anything, so the payload is still readable on the wire. HTTPS is still required.
import hashlib
import hmac
import json
import time
SECRET = b"whsec_example_placeholder" # shared secret; load from config in practice
def sign(secret: bytes, timestamp: str, body: bytes) -> str:
# Sign "timestamp.body" over the exact bytes that go on the wire
signed = timestamp.encode() + b"." + body
return hmac.new(secret, signed, hashlib.sha256).hexdigest()
# Sender: serialize once, sign those bytes, and send exactly those bytes
body = json.dumps({"type": "invoice.paid", "id": "evt_123"}).encode()
ts = str(int(time.time()))
headers = {
"Content-Type": "application/json",
"Webhook-Timestamp": ts,
# Version prefix says which algorithm and string format produced the value
"Webhook-Signature": "v1=" + sign(SECRET, ts, body),
}
# Receiver: raw_body must be the unparsed request bytes, not re-serialized JSON
def verify(secret: bytes, headers: dict, raw_body: bytes) -> dict:
ts = headers["Webhook-Timestamp"]
expected = sign(secret, ts, raw_body)
# During a rotation the header may carry several space-separated signatures
for part in headers["Webhook-Signature"].split():
scheme, _, value = part.partition("=")
# Ignore unknown schemes; compare in constant time to avoid timing leaks
if scheme == "v1" and hmac.compare_digest(value, expected):
# Parse JSON only after the raw bytes have been verified
return json.loads(raw_body)
raise ValueError("invalid webhook signature")
event = verify(SECRET, headers, body)
print(event["type"])
A sender signs 'timestamp.body' with HMAC-SHA256 and a v1= prefix, and a receiver recomputes the signature over the raw bytes, compares it in constant time, and only then parses the JSON.
How do you add replay protection and rotate signing secrets?
A valid signature also says nothing about when a message was sent. An attacker who captures one signed request can resend it unchanged, and it will verify every time. Replay protection needs two layers.
Layer one: a signed timestamp. The sender includes a timestamp in the signed string, not just in a header, so it cannot be altered without breaking the signature. The receiver rejects any request whose timestamp falls outside a tolerance window around its own clock.
Layer two: event ID deduplication. Inside the window, a replay still passes the timestamp check. The receiver records each event ID in a store such as a key-value cache. It sets a TTL longer than the window, so an ID cannot expire while its timestamp is still acceptable.
The receiver's flow:
- Verify the signature.
- Parse the timestamp and reject the request if it differs from now by more than the window.
- Attempt an atomic insert-if-absent of the event ID with the TTL.
- If the key already exists, return success without reprocessing, so the sender stops retrying.
Choosing the window is a trade-off:
- Wider windows tolerate clock skew and requests that sat in slow outbound queues.
- Narrower windows shrink the dedupe store and limit how long a captured request stays usable.
A few minutes is a common starting point.
Rotating secrets. Issue a separate secret per endpoint, so one leaked secret exposes only one receiver. During a rotation, the sender signs with both the old and new secrets and sends both signatures. The receiver accepts the request if either signature validates. Once the receiver confirms it has switched to the new secret, the sender drops the old one.
import hashlib, hmac, time
import redis
TOLERANCE_S = 300 # timestamp window: 5 minutes either side of our clock
DEDUPE_TTL_S = 600 # must outlive the window, so an ID cannot expire while still replayable
# During rotation, hold both secrets for this endpoint; drop the old one once the sender stops using it.
SECRETS = [b"whsec_new_placeholder", b"whsec_old_placeholder"]
store = redis.Redis(host="localhost", port=6379)
def signature_ok(timestamp: str, body: bytes, signatures: list[str]) -> bool:
# The timestamp is part of the signed string, so it cannot be changed without breaking the MAC.
signed = timestamp.encode() + b"." + body
for secret in SECRETS:
expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
# Accept if any signature validates against any secret currently held.
if any(hmac.compare_digest(expected, sig) for sig in signatures):
return True
return False
def handle(event_id: str, timestamp: str, body: bytes, signatures: list[str]) -> int:
if not signature_ok(timestamp, body, signatures):
return 401
try:
sent_at = int(timestamp)
except ValueError:
return 400
# Layer one: reject anything signed too far from now (old replays or bad clocks).
if abs(time.time() - sent_at) > TOLERANCE_S:
return 400
# Layer two: atomic insert-if-absent; returns None if the ID was already recorded.
if not store.set(f"webhook:seen:{event_id}", 1, nx=True, ex=DEDUPE_TTL_S):
return 200 # duplicate: acknowledge so the sender stops retrying, but do not reprocess
process(body)
return 200
def process(body: bytes) -> None:
... # application logic goes here
A Python receiver sketch that accepts either signature during secret rotation, rejects timestamps outside a 5-minute window, and dedupes event IDs with an atomic Redis SET NX whose TTL outlives that window.
When should webhook retries happen, and how should backoff work?
Retry only when a later attempt could plausibly succeed. Network errors, timeouts, 5xx responses and 429 all point to a transient condition on the receiver's side. Most other 4xx responses, such as 400, 401 or 404, mean the request is malformed, unauthorized or misaddressed. Resending identical bytes will fail the same way, so mark those deliveries as permanent failures. When a 429 or 503 carries a Retry-After header, use that value instead of your computed delay, because the receiver knows its own recovery time better than you do.
Space attempts with exponential backoff and add jitter. Without jitter, thousands of deliveries that failed during the same outage retry in lockstep and can overload the receiver again as soon as it recovers. Bound the total retry window as well as the attempt count.
Illustrative estimate. Assumed inputs: 30-second base delay, doubling each attempt, 1-hour cap, eight retries, and full jitter, where each delay is drawn uniformly between zero and the ceiling.
| Retry | Delay ceiling | Cumulative ceiling |
|---|---|---|
| 1 | 30 s | 30 s |
| 2 | 1 min | 1.5 min |
| 3 | 2 min | 3.5 min |
| 4 | 4 min | 7.5 min |
| 5 | 8 min | 15.5 min |
| 6 | 16 min | 31.5 min |
| 7 | 32 min | 63.5 min |
| 8 | 60 min (capped) | 123.5 min |
An aggressive schedule like this one surfaces failures within about two hours, but it abandons events when a receiver is down longer than that. A conservative schedule that stretches over several days survives a weekend outage, but backlogs grow while it waits. Retrying forever avoids data loss at the cost of unbounded queues, while giving up early keeps queues small and drops data. Neither option removes the need for a dead-letter state. Exhausted events must go somewhere operators can inspect them and replay them deliberately.
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
BASE_DELAY = 30 # seconds; first retry ceiling
MAX_DELAY = 3600 # cap any single computed delay at 1 hour
MAX_RETRIES = 8
MAX_WINDOW = 3 * 3600 # never retry later than 3 hours after the first attempt
def is_retryable(status):
# status is None for network errors and timeouts (no HTTP response at all)
return status is None or status == 429 or 500 <= status <= 599
def retry_after_seconds(value):
# Retry-After is either delay-seconds or an HTTP-date
if not value:
return None
if value.strip().isdigit():
return int(value)
try:
when = parsedate_to_datetime(value)
except (TypeError, ValueError):
return None
if when.tzinfo is None:
when = when.replace(tzinfo=timezone.utc)
return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
def next_delay(retry, status, headers, elapsed):
"""Seconds to wait before retry number `retry` (1-based), or None to dead-letter.
`headers` should be a case-insensitive mapping; `elapsed` is seconds since first attempt."""
if not is_retryable(status) or retry > MAX_RETRIES:
return None # permanent failure or attempts exhausted -> dead-letter
hinted = retry_after_seconds(headers.get("Retry-After")) if status in (429, 503) else None
if hinted is not None:
delay = hinted # the receiver knows its recovery time; skip our own backoff
else:
ceiling = min(MAX_DELAY, BASE_DELAY * 2 ** (retry - 1))
delay = random.uniform(0, ceiling) # full jitter spreads out synchronized retries
if elapsed + delay > MAX_WINDOW:
return None # the total window is bounded too, not just the attempt count
return delay
A retry policy that classifies responses, prefers Retry-After on 429/503, otherwise applies capped exponential backoff with full jitter, and sends the event to dead-letter once the attempt count or total time window is exhausted.
How should you architect a webhook delivery system?
The sender pipeline starts when application code writes the event to a durable outbox table in the same transaction as the business change. This prevents the case where the database commits but the event is lost, and the case where an event is sent for a change that rolled back. A dispatcher reads the outbox and fans each event out into one delivery record per subscribed endpoint. Each record has its own state: attempt count, next retry time and last error. Delivery workers send each request with a strict timeout, such as a few seconds, and a concurrency cap for each endpoint. Without these limits, one slow receiver can hold open every connection you have.
| Design | Head-of-line blocking | Operational complexity |
|---|---|---|
| Global queue | One failing endpoint delays everyone behind it | Low: one queue to monitor |
| Per-endpoint isolation | Contained to the failing endpoint | Higher: many queues, fairness scheduling |
Endpoint URLs are user-supplied, so treat every one as an SSRF risk. Resolve the hostname and reject loopback, link-local (including cloud metadata addresses), and private ranges. Pin the connection to the IP address you validated, so DNS rebinding cannot swap it afterward. Refuse redirects, or re-validate each redirect target.
Receivers should verify the signature, persist the raw event, and return a 2xx quickly. Processing then runs asynchronously from local storage. Slow inline work causes timeouts, and timeouts cause duplicate deliveries.
Senders cannot guarantee ordering, because retries and parallel workers reorder events. Include an event timestamp or a per-resource sequence number. Receivers then discard events older than the state they already hold.
+----------------------+ same DB txn +----------------------+
| App: business change |-------------->| Outbox table |
+----------------------+ +----------------------+
v
+----------------------+
| Dispatcher: fan-out |
| 1 record per endpoint|
+----------------------+
+--------------------------------+-----+
v v
+--------------------------+ +--------------------------+
| Endpoint A delivery rows | | Endpoint B delivery rows |
| attempts, next retry, | | attempts, next retry, |
| last error | | last error |
+--------------------------+ +--------------------------+
| Worker: few-sec timeout | | Worker: few-sec timeout |
| concurrency cap | | concurrency cap |
| SSRF: resolve, block | | SSRF: resolve, block |
| internal ranges, pin IP, | | internal ranges, pin IP, |
| no redirects | | no redirects |
+--------------------------+ +--------------------------+
| signed HTTPS POST |
v v
+-----------------------------------------------------------------+
| Receiver: verify signature -> persist raw event -> 2xx quickly |
+-----------------------------------------------------------------+
v
+-----------------------------------------------------------------+
| Async worker: skip events older than stored timestamp/seq |
+-----------------------------------------------------------------+
Transactional outbox fans out to isolated per-endpoint delivery state and SSRF-guarded workers; the receiver verifies, persists and acks fast, then processes async using timestamps or sequence numbers since order is not guaranteed.
What are the alternatives to HMAC-signed fat-payload webhooks?
Shared secrets vs. asymmetric signatures
With HMAC, sender and receiver hold the same secret, so anyone who has it can forge events. That includes a compromised receiver, a leaked config file, or a support engineer pasting it into a ticket. Each receiver needs its own secret delivered over a secure channel. Asymmetric signatures such as Ed25519 remove both problems. The sender keeps the private key, publishes the public key, and receivers can verify events but cannot forge them. The cost falls on receivers. They need a signature library rather than a standard HMAC call, and they must fetch, cache and refresh public keys during rotation.
Fat vs. thin payloads
| Concern | Fat payload | Thin payload (IDs only) |
|---|---|---|
| Ordering and staleness | Receiver may apply outdated state | Receiver fetches current state |
| Data exposure | Full record sent to the endpoint | Only IDs leave your API |
| API load | None added | One fetch per event, and spikes during retry storms |
| Availability coupling | Event is self-contained | Processing fails if the API is down |
Thin payloads reduce ordering bugs and leak less data. In exchange, the receiver depends on your API being up at the moment it handles the event.
Polling an events endpoint
A cursor-paginated events endpoint lets receivers pull at their own pace. Events arrive in a defined order, no inbound endpoint is needed, and catching up after downtime only means resuming from the last cursor.
Avoid webhooks when receivers cannot expose a public HTTPS endpoint, such as on-premises systems behind strict firewalls. Avoid them also when consumers need strict ordering. In those cases, offer polling as the primary delivery model, or offer it alongside webhooks.
How do operator tooling and design choices affect development cost and engineering time?
Most webhook support tickets come down to one question: "did you send it?" Four operator tools let receivers answer it themselves:
- Delivery log. Store each attempt with its timestamp, response status, latency and a truncated response body. Receivers can then see that their endpoint returned a 401, which usually means a signature mismatch from a stale secret, without opening a ticket.
- Manual redelivery. A "resend" button or API call lets receivers recover after fixing a bug. Without it, your engineers replay events by hand.
- Dead-letter state. Events that exhaust their retries move to a terminal, queryable state instead of disappearing. This makes recovery possible.
- Automatic disabling with notification. After sustained failure, pause the endpoint and email its owner. A permanently dead URL then stops consuming worker capacity.
The infrastructure has an upfront cost. An outbox, per-endpoint queues and a dedupe store take real build and maintenance effort. Weigh that against the recurring cost of incidents:
Illustrative estimate (assumed numbers, not a benchmark). Monthly support cost = tickets × hours per ticket × hourly cost. With 40 tickets, 1.5 hours each and $100 per hour, that is 40 × 1.5 × 100 = $6,000 per month. If a delivery log deflects half of those tickets, it saves $3,000 per month.
Build order:
- Small team, few endpoints: Build the outbox and delivery log first. Losing events is the failure you cannot repair later, and the log is the main tool for debugging.
- Hundreds of endpoints: Add per-endpoint isolation and auto-disable next. At this scale, a single slow receiver can starve delivery to all the others.
- Self-serve receivers: Expose redelivery to receivers once they can operate it without your help.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to webhook delivery best practices, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.