What delivery guarantee do webhook providers actually give?

If your service receives webhook events that trigger side effects, this guide applies to you. Typical side effects include charging a card, provisioning an account, or syncing state into your database. Teams pick webhooks over polling for two reasons. Events arrive seconds after they happen instead of at the next poll interval. And you stop spending API quota on requests that mostly return "nothing changed."

The convenience comes with a weaker contract than many consumers assume. Design for these properties:

  • At-least-once delivery. The provider retries when your endpoint returns a non-2xx status or fails to respond within its timeout. The same event can arrive two, three or more times.
  • No exactly-once. No provider can promise this over HTTP.
  • Usually no ordering. Retries, parallel senders and network variance mean an updated event can land before the created event it depends on.
  • Possible forgery. Your endpoint is a public URL. Anyone who finds it can POST a convincing payload unless you verify signatures.

Exactly-once is impossible because the provider learns about your processing only through your HTTP response. Suppose your handler commits the charge to the database, and then the connection drops before the 200 reaches the provider. From the provider's side, that looks the same as a request that never arrived. It has two options: retry and risk a duplicate, or give up and risk a lost event. Reasonable providers choose the duplicate. That means deduplication is your job: record each event ID and skip events you have already processed.

Who should use these practices?

These practices are for engineers who own the receiving end of a webhook integration and whose handlers change state that matters:

  • Backend engineers integrating third-party event sources, such as payment processors, identity providers, source-control hosts or SaaS tools, where a duplicate or forged event changes money, access or customer data.
  • Platform or infrastructure teams building shared webhook ingress for several product teams. Verification, inbox storage, deduplication and dead-lettering belong in one shared component, so each integration doesn't reimplement them with its own bugs.
  • Teams that send webhooks to their own customers. The consumer-side requirements below show what a provider has to supply: a stable event ID, a signed timestamp, a resource version, overlapping secrets during rotation and a documented retry window.

Match the effort to the stakes. A handler that posts a chat notification can tolerate an occasional duplicate, so it may not need an inbox table, version guards or a reconciler. It should still verify signatures, because a forged request to a public URL costs nothing to send. A handler that moves money, grants access or writes records you are accountable for needs every layer in this guide.

How does webhook delivery work end to end?

Delivery passes through two systems, and each step maps to a section of this guide.

On the provider side:

  1. Something changes, and the provider records an event with a unique ID, a type and a payload that describes the resource.
  2. The provider serializes the payload and signs it with your shared secret, usually together with a timestamp.
  3. It POSTs the body and signature headers to your registered URL and waits for a response, up to its timeout.
  4. A 2xx response marks the event as delivered. Anything else, including a timeout, schedules a retry with backoff. When the retry window ends, the provider drops the event, surfaces it for manual redelivery or disables the endpoint.

On your side:

  1. Verify. Check the signature against the raw bytes and reject stale timestamps. See the signing and rotation sections.
  2. Persist, then ack. Write the raw event to a durable inbox and return 2xx quickly. See the persist-then-ack section.
  3. Process idempotently. A worker claims the event ID in a dedup table and applies the side effect in the same transaction.
  4. Guard against stale events. Apply an update only if its resource version is newer than the stored one, or fetch the current state. See the ordering section.
  5. Handle failure. Bounded retries move failing events to a dead-letter store, and replay tooling reprocesses them.
  6. Reconcile. A scheduled job finds events that never arrived and sends them through the same pipeline.

Each layer covers a failure the previous one can't. Signatures stop forgery but not duplicates. Deduplication stops duplicates but not reordering. Version guards stop reordering but can't recover an event that was never delivered. Reconciliation recovers lost events.

How does HMAC signature verification work, and where does it go wrong?

Deduplication only works if the event is genuine, and that is what the signature establishes. The provider computes an HMAC, usually HMAC-SHA256, using a shared secret as the key. The input is the exact bytes of the request body, often prefixed with a timestamp sent in a separate header. Your endpoint recomputes the HMAC over the same input and compares the two values.

The comparison has to run against raw bytes. Read the body into a buffer before any JSON parsing, verify the signature, and only then deserialize. If you parse first and re-serialize to verify, key order, whitespace and number formatting can change, and valid requests will fail. Other common failures:

  • Encoding mismatch. Providers send the digest as hex or base64. Decode the provider's value, or encode yours, so that both sides use the same format before comparing.
  • Ordinary string equality. == returns as soon as it finds a mismatched byte, which leaks timing information. Use your language's constant-time comparison function.
  • Middleware. Body parsers, decompression layers and logging proxies can consume or rewrite the stream before your handler sees it. Configure raw-body access for this route specifically.

A valid signature does not stop replay, where an attacker resends a captured request. To guard against it, reject requests whose signed timestamp falls outside a tolerance window. A tight window narrows the replay opportunity but rejects legitimate requests from servers whose clocks have drifted. A loose window tolerates skew but widens the attack window. A few minutes is a common compromise. To close the gap inside the window, also record event IDs you have already seen and reject repeats.

Keep in mind what HMAC actually proves: the sender knew the secret. It does not prove the sender is the provider. If the secret leaks, anyone who has it can produce valid signatures.

import hashlib, hmac, json, time
from flask import Flask, request, abort

app = Flask(__name__)
SECRET = b"whsec_example_placeholder_not_real"  # load from a secret store in practice
TOLERANCE_SECONDS = 300  # replay window: tighter = safer, looser = more clock-skew tolerance
seen_ids = set()  # demo only: use a shared store with expiry (>= the window) in production

@app.post("/webhooks")
def receive_webhook():
    body = request.get_data()  # exact raw bytes; never verify re-serialized JSON
    ts = request.headers.get("X-Webhook-Timestamp", "")
    sig = request.headers.get("X-Webhook-Signature", "")

    try:
        ts_int = int(ts)
    except ValueError:
        abort(400)
    if abs(time.time() - ts_int) > TOLERANCE_SECONDS:
        abort(400)  # stale or future-dated: possible replay or bad clock

    # Sign timestamp + body so the timestamp can't be swapped independently.
    signed_payload = ts.encode() + b"." + body
    expected = hmac.new(SECRET, signed_payload, hashlib.sha256).hexdigest()  # hex, matching the provider's format

    # Constant-time comparison on bytes; '==' would leak timing information.
    if not hmac.compare_digest(expected.encode(), sig.strip().lower().encode()):
        abort(401)

    event = json.loads(body)  # parse only after the signature checks out
    event_id = event.get("id")
    if not event_id:
        abort(400)
    if event_id in seen_ids:
        return "", 200  # replay or retry inside the window: acknowledge, don't reprocess
    seen_ids.add(event_id)

    # ... handle the event ...
    return "", 200

A Flask webhook handler that reads the raw body, rejects timestamps outside a tolerance window, verifies a hex HMAC-SHA256 over timestamp plus body with a constant-time comparison, and only then parses JSON and rejects already-seen event IDs.

How do you rotate webhook secrets without downtime?

Because a leaked secret lets anyone sign valid requests, rotation needs to be a routine operation you can run at any time. It should not be an emergency procedure you are rehearsing for the first time.

The overlap pattern

Most providers let you generate a new signing secret while the old one stays active. Some then sign each delivery with both secrets. Others switch to the new secret at a point you choose. Either way, the consumer has to accept both secrets for a while:

  1. Add the new secret to your secrets store alongside the old one. Deploy so every instance loads both.
  2. Activate the new secret on the provider side.
  3. Verify each request against every active secret, using the same constant-time comparison as before. Accept the request if any secret matches. Log which secret matched by ID, never by value.
  4. When no requests have matched the old secret for longer than the provider's retry window, remove the old secret and redeploy.

Step 4 is why you track which secret matched. Without that signal, you cannot tell when the old secret is no longer in use.

Handling the secrets

Load secrets from a secrets store at startup or on refresh. Keep them out of source control, container images and committed config files. Do not include them in error messages or debug logs. A "signature mismatch" log line that prints the expected value is itself a leak.

The overlap window has a cost. During overlap, a leaked old secret still forges valid requests. If you are rotating because of a suspected leak, keep the overlap as short as possible. Accept that some in-flight retries signed with the old secret will fail, and plan to recover those events another way.

import hashlib
import hmac
import json
import logging
import os

log = logging.getLogger("webhooks")


def load_secrets() -> dict[str, bytes]:
    # Injected at startup by the secrets store integration, e.g.
    # WEBHOOK_SECRETS='{"wh_2026_05": "<old>", "wh_2026_11": "<new>"}'
    # Never committed to source control, config files or container images.
    raw = json.loads(os.environ["WEBHOOK_SECRETS"])
    return {secret_id: value.encode() for secret_id, value in raw.items()}


ACTIVE_SECRETS = load_secrets()  # old + new during overlap; new only after


def verify(body: bytes, signature_hex: str) -> str | None:
    """Return the ID of the secret that matched, or None."""
    received = signature_hex.encode()
    matched = None
    for secret_id, secret in ACTIVE_SECRETS.items():
        expected = hmac.new(secret, body, hashlib.sha256).hexdigest().encode()
        # Constant-time compare against every active secret; no early exit.
        if hmac.compare_digest(expected, received):
            matched = secret_id
    if matched:
        # Log the ID only: this is the signal that tells you when the
        # old secret has stopped matching and can be retired.
        log.info("webhook signature verified secret_id=%s", matched)
    else:
        # Never log expected/received signatures or secret values.
        log.warning("webhook signature mismatch")
    return matched

A Python HMAC-SHA256 verifier that loads every active secret from the secrets store, checks the signature against each one with a constant-time comparison, and logs only the ID of the secret that matched so you can tell when the old secret is safe to remove.

How should you implement a persist-then-ack webhook pipeline?

The HTTP handler should do three things. First, verify the signature against the raw request body. Second, durably write the raw payload, relevant headers, and the provider's event ID to a database row or queue message. Third, return 2xx. Everything else belongs to workers that read from that store.

This split exists because providers enforce a delivery timeout. If your handler calls downstream APIs or runs slow queries inline, a request can exceed that timeout even though your side effect completed. The provider records a failure and retries, so you process the same event twice. Fast acks keep the provider's view of delivery consistent with yours.

Workers still receive duplicates, so make processing idempotent. Insert the event ID into a dedup table with a unique constraint, in the same transaction as the side effect. If the insert conflicts, the event was already applied, and the worker can skip it. If the transaction rolls back, neither the dedup row nor the side effect persists. Side effects outside the database, such as emails or third-party calls, cannot join that transaction. For those, pass the event ID as an idempotency key or route them through an outbox, and design handlers so a rerun converges on the same state.

The dedup table grows without bound, so you will need to prune it. Keep rows longer than the provider's maximum retry window. If you prune earlier, late retries are processed again.

Acking before processing moves retry responsibility to you. Workers need bounded retries with backoff and a dead-letter store for events that keep failing. Alert on dead-letter depth and on the age of the oldest unprocessed event. Build replay tooling, because the provider will not redeliver an event it considers delivered.

-- PostgreSQL. Inbox written by the HTTP handler before it returns 2xx.
CREATE TABLE webhook_inbox (
  event_id     text PRIMARY KEY,           -- provider's event ID
  headers      jsonb NOT NULL,
  raw_body     bytea NOT NULL,             -- exact bytes that were signature-verified
  received_at  timestamptz NOT NULL DEFAULT now(),
  attempts     int NOT NULL DEFAULT 0,
  processed_at timestamptz,
  dead_at      timestamptz                 -- set when retries are exhausted
);

-- Dedup table: the unique key is what makes reprocessing a no-op.
CREATE TABLE processed_events (
  event_id     text PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);

-- Worker: claim the dedup row and apply the side effect in ONE transaction.
BEGIN;
INSERT INTO processed_events (event_id) VALUES ($1)
  ON CONFLICT (event_id) DO NOTHING
  RETURNING event_id;
-- No row returned => already applied: ROLLBACK, then mark the inbox row processed.
UPDATE orders SET status = 'paid' WHERE id = $2 AND status <> 'paid';
UPDATE webhook_inbox SET processed_at = now() WHERE event_id = $1;
COMMIT;  -- on error, rollback removes both the dedup row and the side effect

-- Pruning: retention must exceed the provider's maximum retry window.
-- 30 days is an assumed placeholder; check your provider's documented window.
DELETE FROM processed_events WHERE processed_at < now() - interval '30 days';

-- Alerting inputs: dead-letter depth and age of the oldest unprocessed event.
SELECT count(*) FILTER (WHERE dead_at IS NOT NULL) AS dead_letters,
       now() - min(received_at) FILTER (WHERE processed_at IS NULL AND dead_at IS NULL) AS oldest_pending_age
FROM webhook_inbox;

A PostgreSQL inbox and dedup table where the worker claims the event ID and applies the side effect in one transaction, plus pruning and alerting queries.

How do you handle out-of-order and stale webhook events?

Providers dispatch webhooks from many workers in parallel, and each delivery retries on its own schedule. If subscription.updated fails once and is retried thirty seconds later, a subsequent subscription.canceled for the same subscription may already have arrived. Replay tooling makes this worse, because a replayed event is old by definition. Arrival order does not tell you which change happened last.

Guard updates with a version or timestamp

If the payload carries a monotonic resource version or a last-modified timestamp from the provider, store it with your local copy and apply an event only if it is newer. Make the comparison part of the write itself. For example, UPDATE subscriptions SET status = $1, provider_version = $2 WHERE id = $3 AND provider_version < $2. If zero rows change, the event is stale, so record it as processed and drop it. A read-compare-write split across two statements reintroduces the race this check exists to prevent. Prefer a version counter to wall-clock timestamps. Timestamps can collide at the same precision, and you have no way of knowing whether they come from a single clock.

Treat the webhook as a notification and fetch current state

Use only the event's resource ID and fetch the current object from the provider's API. Ordering stops mattering, because every handler converges on the latest state. The costs are real:

  • Latency: each event adds a network round trip.
  • Rate limits: a burst of events becomes a burst of API calls. Coalesce events per resource ID in your queue to reduce them.
  • Availability: if the provider's API is degraded, processing stalls, even though the events were delivered.

When the payload has no reliable version field, the fetch approach is often simpler to get right.

import psycopg

def apply_subscription_event(conn: psycopg.Connection, event: dict) -> bool:
    """Apply an event only if it is newer than the stored copy.
    Returns True if local state changed, False if duplicate or stale."""
    sub = event["data"]  # placeholder payload shape: id, status, version
    with conn.transaction():  # commits on normal exit, rolls back on error
        with conn.cursor() as cur:
            # Dedupe by event ID. A redelivered event matches the existing row
            # and inserts nothing.
            cur.execute(
                "INSERT INTO processed_events (event_id) VALUES (%s) "
                "ON CONFLICT DO NOTHING",
                (event["id"],),
            )
            if cur.rowcount == 0:
                return False

            # The comparison and the write run as one statement, so two handlers
            # racing on the same subscription cannot both apply. The upsert also
            # handles an event that arrives before we have stored the row.
            cur.execute(
                """
                INSERT INTO subscriptions (id, status, provider_version)
                VALUES (%s, %s, %s)
                ON CONFLICT (id) DO UPDATE
                   SET status = EXCLUDED.status,
                       provider_version = EXCLUDED.provider_version
                 WHERE subscriptions.provider_version < EXCLUDED.provider_version
                """,
                (sub["id"], sub["status"], sub["version"]),
            )
            # Zero rows means the stored version is the same or newer, so the
            # event is stale. The processed_events row is still committed, which
            # acknowledges the event and keeps it from being reconsidered.
            return cur.rowcount == 1

A PostgreSQL handler using psycopg that deduplicates by event ID, then applies the change only when the event's provider version is newer than the stored one, with the comparison and the write in a single atomic statement.

Which HTTP status codes should a webhook endpoint return?

The status code is the only feedback channel the provider reads, so treat it as an instruction about retries rather than a report of your internal state. A provider generally treats 2xx as "delivered, stop", 5xx or a timeout as "try again later", and 4xx inconsistently. Some providers stop retrying on most 4xx codes. Others retry them like any other failure. Others count them toward disabling the endpoint.

Situation Response Typical provider behavior Consumer responsibility
Event persisted, will be processed 200 or 204 Marks delivered, stops retrying Process from your store, not the request
Event persisted, but a duplicate or an ignored type 200 or 204 Marks delivered Record why it was skipped
Signature invalid or payload malformed 400 or 401 Often stops, sometimes retries or flags endpoint Alert, because this usually means a secret or config fault
Storage unavailable, can't persist 503 or other 5xx Retries with backoff Recover before the retry window closes
Handler too slow Timeout Treated as failure and retried, possibly causing duplicates Ack faster and move work off the request path

Don't return 4xx for "I don't care about this event." The provider may read a stream of 4xx responses as a broken endpoint.

Retry schedules, the total retry window, and what happens at the end differ by provider. Common outcomes are that the event is dropped silently, that it is surfaced in a dashboard for manual redelivery, or that the endpoint is disabled after sustained failures. Read each provider's documented policy. That window is how long an outage on your side can last before data loss starts.

When should you avoid relying on webhooks, and what are the alternatives?

Once that window closes, only reconciliation can recover the event. A scheduled job lists recent events, or queries the current state of resources you care about, and feeds anything unseen into the same idempotent pipeline the webhook handler uses. This has three costs: a second ingestion path to maintain, API quota spent on every run, and a hard dependency on the provider exposing list or query endpoints.

Webhooks alone are a poor fit when you need strict completeness, as in ledgers or billing, or strict ordering. They are also a poor fit when the provider offers no event listing, because a dropped delivery then leaves no trace you can detect.

Approach Latency Completeness Complexity Operational cost
Polling Bounded by interval High, if listing is reliable Low Quota, mostly empty calls
Inline processing in handler Lowest Events lost when the handler fails Lowest Slow work causes timeouts and retry storms
Managed queue or event bus Low Durable, often replayable Medium Another system to run and pay for
Webhook plus reconciliation Low, with periodic backfill Highest Medium–high Scheduler and quota

On engineering time, the queue and the reconciler add components you have to operate. Signature checks and dedup keys add almost none, and they prevent the incidents that are most expensive to handle: double charges and investigations into forged events.

Monitor these signals:

  • Signature failures. A spike points to a rotation bug or to probing.
  • Dedup hits. A rising count usually means your acks are too slow and the provider is retrying.
  • Queue lag.
  • Processing errors.
  • Records the reconciler finds that webhooks missed. Any nonzero count means deliveries are being lost.

How can these practices save engineering time?

The time savings come from turning open-ended investigations and manual data repair into routine, mechanical operations. Each practice removes a specific kind of work:

  • Idempotent processing replaces cleanup. Without deduplication, every retry storm can leave duplicate charges, emails or records that someone has to find, reverse and explain to affected users. With a dedup table, a duplicate becomes a skipped row and a log line.
  • Persist-then-ack makes incidents recoverable by replay. The inbox keeps the exact bytes the provider sent. When a downstream dependency fails or a bug ships, you fix the code and re-run the affected events from the dead-letter store. You don't have to reconstruct what the provider sent or ask the provider to redeliver.
  • Stored raw bodies shorten debugging. Signature mismatches and parsing errors can be reproduced locally against the stored payload, so you don't have to wait for the provider to send a similar event.
  • Version guards remove a class of heisenbugs. Out-of-order bugs show up intermittently and are hard to reproduce. Rejecting stale writes at the database means you won't spend days chasing a subscription that sometimes ends up in the wrong state.
  • Overlap rotation avoids coordinated cutovers. Accepting multiple secrets means rotation is an ordinary deploy, with no maintenance window or synchronized change across teams. That matters most when you rotate after a suspected leak.
  • Reconciliation replaces manual audits. You don't compare your records against the provider's dashboard by hand. A scheduled job finds the gaps and feeds them through the normal pipeline.
  • One shared implementation replaces many partial ones. If a platform team builds verification, the inbox, deduplication and dead-lettering once, each new integration only has to write its event handler.

Illustrative estimate, with assumed inputs rather than measured results: assume a team without deduplication or an inbox has two incidents per quarter caused by duplicate or lost webhook events. Assume each one takes two engineers one day to investigate, repair data and communicate with affected users. That is about four engineer-days per quarter. Also assume the persist-then-ack pipeline, deduplication and reconciler take about five engineer-days to build and a fraction of a day per quarter to maintain. Under those assumptions, the work pays for itself within a few quarters, before counting the cost of double charges to customers. Replace the assumed inputs with your own incident history and build estimates before you rely on the conclusion.

The trade-off is still real. The queue, dead-letter store and reconciler are components you have to run and monitor. For low-stakes integrations, signature verification and a dedup key may be all the investment that pays back. Add the other layers as the cost of a wrong or missing event goes up.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to webhook delivery best practices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.