What is reliable webhook delivery, and why is it at-least-once by nature?
A webhook is an HTTP request your system sends to someone else's endpoint when an event happens: an invoice is paid, a build finishes, a user is deleted. Reliable delivery does not mean each event arrives exactly once. It means every event eventually arrives at least once, and the receiver can tell real deliveries apart from forged or stale ones.
The at-least-once property follows from how HTTP behaves under failure. Suppose the sender posts an event and the connection times out after 30 seconds. The receiver may have:
- never received the request,
- processed it fully and then lost the connection before responding, or
- crashed halfway through.
From the sender's side, all three look the same. The only safe choice is to send the event again. That means duplicates are a normal part of correct operation, not an edge case. Every receiver has to be idempotent: processing the same event twice must have the same effect as processing it once.
The channel is also adversarial. The endpoint is a public URL, so anyone who finds it can post to it, and anyone who captures a valid request can send it again later. A legitimate retry and a malicious replay are the same kind of event: a copy of a message you have already seen. Retries create benign duplicates and replay protection rejects hostile ones, so the two have to be designed together.
This matters to two groups. Platform teams that send events own the signing, retry schedule, and delivery guarantees. Backend teams that consume events own verification, deduplication, and safe acknowledgement.
How does HMAC signature signing and verification work?
The sender builds a canonical string from the Unix timestamp, a separator, and the exact raw body bytes it is about to transmit, for example 1730592000.{"id":"evt_123",...}. It computes HMAC-SHA256 over that string with the shared secret and sends the result in a header such as Webhook-Signature, with the timestamp in Webhook-Timestamp. The timestamp is part of the signed input, so an attacker cannot change it without invalidating the signature.
Flow: sender serializes body → signs timestamp.body → POST over HTTPS with headers → receiver reads raw bytes → recomputes HMAC → compares → checks timestamp → parses JSON
The receiver must verify in this order:
- Read the raw body as bytes, before any middleware touches it.
- Recompute the HMAC over the received timestamp header, the separator, and those bytes.
- Compare in constant time using your standard library's timing-safe comparison. Ordinary string equality returns as soon as one byte differs, and that timing difference can reveal the signature one byte at a time.
- Check the timestamp against a tolerance window, such as five minutes.
- Parse the payload only after the previous checks pass.
The most common failure is a framework that parses JSON automatically. When you re-serialize the parsed object, key order, whitespace, and number formatting can all change, so the HMAC no longer matches. Capture the raw body explicitly.
Signing proves who sent the message and that it was not modified. It does not encrypt the payload, and it does not prove the receiver processed the event. You still need HTTPS for confidentiality.
import hashlib, hmac, json, time
SECRET = b"whsec_example_placeholder_do_not_use" # shared secret, loaded from config in practice
TOLERANCE_SECONDS = 300 # five-minute window
# --- Sender: sign the exact bytes that will be transmitted ---
def sign(raw_body: bytes, timestamp: int) -> dict:
signed_input = str(timestamp).encode() + b"." + raw_body
sig = hmac.new(SECRET, signed_input, hashlib.sha256).hexdigest()
return {"Webhook-Timestamp": str(timestamp), "Webhook-Signature": sig}
# --- Receiver: verify in order, parse last ---
def verify_and_parse(raw_body: bytes, headers: dict):
ts_header = headers.get("Webhook-Timestamp", "")
sig_header = headers.get("Webhook-Signature", "")
# 1-2. Recompute over the received timestamp header and the untouched raw bytes
expected = hmac.new(SECRET, ts_header.encode() + b"." + raw_body,
hashlib.sha256).hexdigest()
# 3. Constant-time comparison; never use == on signatures
if not hmac.compare_digest(expected.encode(), sig_header.encode()):
raise ValueError("signature mismatch")
# 4. Reject stale or future-dated timestamps
try:
ts = int(ts_header)
except ValueError:
raise ValueError("bad timestamp")
if abs(int(time.time()) - ts) > TOLERANCE_SECONDS:
raise ValueError("timestamp outside tolerance")
# 5. Only now is it safe to parse the payload
return json.loads(raw_body)
if __name__ == "__main__":
body = b'{"id":"evt_123","type":"invoice.paid"}' # serialized once, sent as-is
headers = sign(body, int(time.time()))
print(verify_and_parse(body, headers))
# Re-serializing changes the bytes (whitespace here), so verification fails
reserialized = json.dumps(json.loads(body)).encode()
try:
verify_and_parse(reserialized, headers)
except ValueError as e:
print("rejected:", e)
Standard-library Python sender and receiver that HMAC-SHA256 sign timestamp.body, verify with hmac.compare_digest, enforce a timestamp window before parsing, and show that re-serialized JSON fails verification.
How do you prevent webhook replay attacks?
A valid signature only proves the message was authentic at some point. An attacker who captures one signed request (from a proxy log, a misconfigured debug endpoint or a compromised intermediary) can resend it unchanged, and the signature will still verify. Replay defense needs two layers.
Layer 1: timestamp tolerance. The sender includes a timestamp inside the signed content, and the receiver rejects any request whose timestamp falls outside a short window, typically a few minutes. Because the timestamp is covered by the HMAC, an attacker cannot change it without breaking the signature. This limits replays to a bounded window, but a very tight window causes problems of its own. Clock drift between hosts and time spent in the receiver's ingress queue can push legitimate requests outside the window. The sender must also re-sign every retry with a fresh timestamp. If it replays the original signature, retries scheduled more than a few minutes after the first attempt will always be rejected.
Layer 2: event ID deduplication. Inside the window, the receiver records each processed event ID and keeps it for at least the tolerance period. This one check handles both malicious replays and legitimate duplicate deliveries. The check must be an atomic insert-if-absent operation, as in the example below. A separate read followed by a write leaves a race in which two concurrent deliveries can both pass.
When the dedup store is unavailable, you have to choose a behavior. Failing closed means returning an error so the sender retries later, which delays delivery but preserves correctness. Failing open means processing without the check, which is acceptable only if your handlers are already idempotent.
import time
import redis
TOLERANCE_SECONDS = 300 # accept timestamps within +/- 5 minutes
DEDUP_TTL_SECONDS = 2 * TOLERANCE_SECONDS # keep IDs longer than the window
r = redis.Redis.from_url("redis://localhost:6379/0")
def accept_webhook(event_id: str, signed_timestamp: int, handle) -> int:
"""Call only after the HMAC over (timestamp + body) has verified."""
# Layer 1: reject stale or future-dated requests.
if abs(time.time() - signed_timestamp) > TOLERANCE_SECONDS:
return 400
# Layer 2: atomic insert-if-absent. SET with NX succeeds only for the
# first caller, so two concurrent deliveries cannot both pass.
key = f"webhook:seen:{event_id}"
try:
first_time = r.set(key, "1", nx=True, ex=DEDUP_TTL_SECONDS)
except redis.exceptions.RedisError:
# Fail closed: a 5xx makes the sender retry later.
return 503
if not first_time:
return 200 # duplicate or replay: acknowledge, do not reprocess
try:
handle()
except Exception:
# Release the claim so the sender's retry can be processed.
r.delete(key)
return 500
return 200
A receiver that rejects timestamps outside a 5-minute window, deduplicates event IDs with an atomic Redis SET NX EX, and fails closed when the dedup store is unreachable.
How should webhook retries be designed?
Retry only when a later attempt could plausibly succeed. Network errors, timeouts, 5xx and 429 are transient. Most other 4xx responses mean the request itself was rejected, so resending it unchanged will fail the same way.
| Response class | Retry? | Reason |
|---|---|---|
| Connection error / timeout | Yes | Receiver may be restarting or overloaded |
| 2xx | No | Delivered |
| 429 | Yes, honor Retry-After |
Receiver is asking you to slow down |
| 5xx | Yes | Server-side fault, often transient |
| Other 4xx (400, 401, 404, 410) | No, alert or disable | Bad payload, bad signature or gone endpoint |
Space attempts with exponential backoff and jitter. Without jitter, every delivery that failed during the same outage retries at the same instant and hits the receiver as it recovers. Use a short per-attempt timeout, around 10 seconds, so one slow receiver doesn't tie up delivery workers.
Illustrative estimate (assumed inputs, not a measurement): a 30 s initial delay, factor 2 and 10 total attempts give 9 waits: 30 + 60 + … + 7,680 s = 15,330 s, about 4.3 hours. Receivers should keep deduplication keys longer than that span. A 24-hour retention window covers it with margin for jitter and manual redelivery.
Limit concurrency per endpoint and add a circuit breaker. If one receiver starts failing, a burst of retries can keep it down, so the receiver's partial outage becomes overload you caused. When an endpoint's error rate crosses a threshold, open the circuit and stop sending. Then probe it with single requests and resume once it responds normally. This also keeps one failing endpoint from slowing deliveries to healthy ones.
import random
import time
from email.utils import parsedate_to_datetime
INITIAL_DELAY = 30 # seconds before the first retry
FACTOR = 2
MAX_ATTEMPTS = 10 # total attempts, including the first
PER_ATTEMPT_TIMEOUT = 10 # seconds; pass this to your HTTP client
def should_retry(status):
# status is None for connection errors and timeouts
if status is None or status == 429 or 500 <= status <= 599:
return True
return False # 2xx is delivered; other 4xx will fail the same way again
def retry_after_seconds(header):
# Retry-After is either delay-seconds or an HTTP-date
if not header:
return None
if header.isdigit():
return int(header)
try:
return max(0, parsedate_to_datetime(header).timestamp() - time.time())
except (TypeError, ValueError):
return None # unparseable hint: fall back to our own backoff
def next_delay(attempt, retry_after_header=None):
"""Seconds to wait after failed attempt `attempt` (1-based); None means give up."""
if attempt >= MAX_ATTEMPTS:
return None # caps total retry span: 9 waits, about 4.3 h before jitter
base = INITIAL_DELAY * FACTOR ** (attempt - 1)
# Jitter spreads retries so deliveries that failed together don't return together
delay = random.uniform(base / 2, base)
hinted = retry_after_seconds(retry_after_header)
if hinted is not None:
delay = max(delay, hinted) # never retry sooner than the receiver asked
return delay
A retry policy that retries only on connection errors, timeouts, 429 and 5xx. It uses exponential backoff with jitter, honors Retry-After and stops after a fixed number of attempts.
What should the receiver contract include for acknowledgement and ordering?
The sender's discipline only works if the receiver holds up its side. The core rule is to return a 2xx only after the event is durably stored, for example committed to a database table or written to a persistent queue. Then process it asynchronously. If you acknowledge first and the process crashes before the write, the sender considers the event delivered and will never retry it, so the event is lost. If you do the full business logic inline, a slow handler can exceed the sender's timeout. The sender then retries an event you are still processing, which produces duplicates and wasted load.
Receivers must also assume delivery order is not guaranteed. Retries, parallel workers and network delays can cause an updated event to arrive before the created event it follows. Use the fields on each event to decide what to keep:
- Event ID detects exact duplicates.
- Created-at timestamp gives a rough ordering. Clock skew makes it unreliable for close events.
- Resource version is a monotonically increasing number per object. If an incoming event's version is less than or equal to the stored version, discard it as stale.
Payload shape affects these trade-offs:
| Aspect | Fat payload (full object) | Thin payload (ID + type only) |
|---|---|---|
| Latency | No extra round trip | Receiver must call the API to fetch state |
| Data exposure | Sensitive fields travel to every endpoint and log | Data is fetched under the receiver's own credentials |
| Staleness | Snapshot may be outdated by the time it is processed | Fetch returns current state, so ordering matters less |
How do you rotate webhook secrets, and what are the alternatives to shared HMAC keys?
Give each endpoint its own secret, generated by the sender and shown to the receiver once. A leaked secret then exposes one integration instead of every customer. Rotation should never need a coordinated cutover. The sender keeps a list of active secrets per endpoint, signs each delivery with all of them, and puts every signature in one header, each tagged with a scheme or key identifier. The receiver loops over its own configured secrets and accepts the request if any one of them matches. To rotate:
- Add the new secret on the sender.
- Let the receiver deploy it alongside the old one.
- Keep both active for an agreed overlap window.
- Revoke the old secret.
At no point does a valid delivery fail verification.
| Property | Shared symmetric HMAC | Asymmetric signatures |
|---|---|---|
| Forgery risk | Anyone holding the secret, including the receiver, can mint valid signatures | Only the private-key holder (the sender) can sign |
| Blast radius of a leak | One endpoint, if secrets are per-endpoint | Public key leak is harmless; private key leak affects every receiver |
| Public verifiability | No; third parties would need the secret | Yes; auditors or proxies can verify with the public key |
| Key management and verification cost | Simple and cheap to compute; one secret per endpoint to store and rotate | Publish public keys with key IDs; verification costs more CPU, usually small next to network latency |
Push delivery is optional. An event-fetch API lets receivers poll with a cursor. Gaps and ordering are then explicit, and nothing breaks while the receiver is down. Avoid webhooks entirely when you need strict global ordering or exactly-once side effects. For those cases, give consumers a durable log they read at their own pace, or a synchronous API protected by idempotency keys.
What does production-grade webhook delivery cost, and how does it save engineering time?
Each of these pieces adds state, storage and code. Each also removes a class of support request that would otherwise need an engineer to read raw logs.
| Practice | What you build and run | What it saves |
|---|---|---|
| Dead-letter queue | Storage for events whose retries are exhausted, a retention policy, replay tooling | Events are no longer lost silently after the final attempt |
| Auto-disable with notification | A failure counter per endpoint, a threshold, and an alert to the endpoint owner | Retry traffic to dead URLs stops, and the owner learns about it before their customers do |
| Per-endpoint delivery logs | Attempt records with status code, latency and timestamp, plus retention | Consumers can answer "did you send it?" without filing a ticket |
| Manual redelivery | An API or UI that re-enqueues an event by ID | Consumers recover from their own outages |
| HTTPS-only endpoints | URL validation when an endpoint is registered | Payloads and signatures are never sent in plaintext |
Illustrative estimate (assumed numbers, not a benchmark)
- Log storage = events/day × attempts/event × bytes/record × retention days.
- With 100,000 × 1.2 × 2 KB × 30 days, that is 7.2 GB. If you keep only metadata and drop response bodies, the cost shrinks further.
- Time saved = tickets avoided/month × minutes/ticket.
- With 20 × 30 minutes, that is 10 engineer-hours a month.
When to adopt each practice
- Any size: Require HTTPS and keep a dead-letter queue. Both are cheap, and losing events is not acceptable.
- A few external consumers: Add delivery logs and auto-disable with notification.
- Many consumers, or self-serve onboarding: Add manual redelivery and expose the logs to consumers. At this scale, support time costs more than the storage.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to webhook delivery best practices, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.