What idempotency keys are and why retries create duplicates

An idempotency key is a client-supplied identifier attached to a mutating request. The server uses it to recognise repeats of the same logical operation, so it can apply the change once and return the original result to every retry.

The problem starts with timeouts. When a client sends POST /payments and the connection drops or the deadline expires, it cannot tell which of these happened:

  • The request never reached the server.
  • The server is still processing it.
  • The server committed the charge and the response was lost.

Retrying is the only way to make progress, but in the last two cases a naive retry charges the customer twice. Load balancers, service meshes and client libraries often retry automatically, so duplicates appear even when your own code never retries.

HTTP semantics cover part of this:

Method Idempotent by HTTP semantics Why
GET Yes Reads state without changing it
PUT Yes Replaces a resource with the same full representation each time
DELETE Yes The resource is absent after the first call and stays absent
POST No Each call typically creates a new resource or triggers a new action
PATCH Not guaranteed An operation such as "increment by 1" compounds on every call

The client generates the key, usually a random UUID, once per logical operation. It sends the same key on every retry of that operation and never reuses it for a different one. If a key is reused for a different operation, the server returns the old result for a request it never ran.

The pattern applies to teams that expose non-idempotent endpoints to callers that retry, such as payment capture, order placement and resource provisioning. In these cases a duplicate side effect costs real money or leaves orphaned infrastructure.

How the request lifecycle works: claim, execute, record, replay

The server handles every request carrying an idempotency key in the same four steps. The aim is that the key, not the caller's timing, decides whether the side effect happens.

Claim. The server inserts a record holding the key, a status of in_progress, and a fingerprint of the request. It computes the fingerprint as a hash of the canonical body, meaning fields are sorted and whitespace is normalized, so that two semantically identical payloads produce the same hash. A unique constraint on the key column (usually scoped to the caller or tenant) enforces the claim. Application-level "check then insert" logic leaves a race window that the constraint does not.

Execute. The business change runs in the same database transaction as the claim. If it fails, the claim rolls back with it, and no record is left behind claiming work that never happened.

Record. Before commit, the server updates the row to completed and stores the exact status code and response body it is about to return.

Replay. A later request with the same key gets that stored response byte for byte. The client cannot tell a replay from the original, which is the point.

When the insert conflicts, the server branches on the existing record:

  • New key: the insert succeeds, so proceed to execute.
  • In progress: another attempt holds the key. Return a conflict response telling the client to retry later. Do not wait indefinitely.
  • Completed, fingerprint matches: replay the stored response.
  • Completed, fingerprint differs: the client reused a key for a different request. Reject it with a client error rather than silently returning an unrelated result.
  Client request with Idempotency-Key header and body
                   |
                   v
  +--------------------------------------------+
  | BEGIN transaction                          |
  | fp = hash(canonical body)                  |
  | INSERT (tenant, key, in_progress, fp)      |
  | UNIQUE (tenant, key) enforces the claim    |
  +--------------------------------------------+
            |                                 |
    insert ok (new key)             unique conflict, read row
            v                                 |
  +---------------------+     +---------------+---------------+
  | Execute business    |     |               |               |
  | change in same txn  | in_progress    completed,      completed,
  | Record status code, |                fp matches      fp differs
  | body -> completed   |     v               v               v
  | COMMIT              |+----------+  +-------------+  +-------------+
  +---------------------+| Conflict:|  | Replay      |  | Client error|
    |          |         | retry    |  | stored code |  | key reused  |
  success     failure    | later    |  | and body,   |  | for another |
    |       rollback,    +----------+  | verbatim    |  | request     |
    v       no row left                +-------------+  +-------------+
  Return response

Claim, execute, record and replay: a unique insert claims the key, and on conflict the server branches on the stored status and request fingerprint.

Implementing the key record and the atomic transaction

The key record

Scope the key by caller and endpoint, not globally. Two tenants generating the same UUID, or one client reusing a key across endpoints, should never collide. A workable idempotency_keys table has a composite primary key on (tenant_id, endpoint, idem_key), plus request_hash, status (in_progress or completed), response_code, response_body, created_at, updated_at, and expires_at. The unique constraint does the real work. Application-level "check then insert" logic races, but the database will not admit two rows with the same key.

One transaction for claim and effect

The handler opens a transaction, inserts the key row as in_progress, writes the business row (the order, the transfer), updates the key row with the response code and body, and commits. Because the claim and the effect commit together, there is no state where the order exists but the key does not, and no state where the key exists but the order does not.

If the insert hits the unique constraint, read the existing row:

  • in_progress: another request with this key is still running. Return 409 Conflict with a Retry-After header. Do not run the handler in parallel. In a single-transaction design, a concurrent insert may block on the uncommitted row until the first transaction finishes, so set a lock timeout to keep that wait bounded.
  • Hash mismatch: return a 4xx such as 422. Replaying the original response would tell the client that a request it never made succeeded.

Canonicalization pitfalls

Hash a normalized form of the request, not raw bytes. JSON field order differs between serializers. An omitted field and its explicit default should usually hash the same. Exclude headers such as tracing IDs, User-Agent, and timestamps. If you include them, honest retries look like key reuse.

-- PostgreSQL. Scope keys by caller and endpoint so equal UUIDs never collide.
CREATE TABLE idempotency_keys (
  tenant_id     text        NOT NULL,
  endpoint      text        NOT NULL,  -- e.g. 'POST /v1/orders'
  idem_key      text        NOT NULL,
  request_hash  text        NOT NULL,  -- hash of the canonicalized request
  status        text        NOT NULL CHECK (status IN ('in_progress', 'completed')),
  response_code integer,
  response_body jsonb,
  created_at    timestamptz NOT NULL DEFAULT now(),
  updated_at    timestamptz NOT NULL DEFAULT now(),
  expires_at    timestamptz NOT NULL DEFAULT now() + interval '24 hours',
  PRIMARY KEY (tenant_id, endpoint, idem_key)  -- the constraint does the dedup
);

-- Per request. The app binds $1..$5. Claim and effect commit together.
BEGIN;
-- A duplicate insert waits on the first transaction's uncommitted row.
-- Bound that wait. On lock_not_available, ROLLBACK and return 409 + Retry-After.
SET LOCAL lock_timeout = '2s';

INSERT INTO idempotency_keys (tenant_id, endpoint, idem_key, request_hash, status)
VALUES ($1, $2, $3, $4, 'in_progress')
ON CONFLICT (tenant_id, endpoint, idem_key) DO NOTHING
RETURNING idem_key;
-- Zero rows returned means the key is already taken. ROLLBACK and run the lookup below.

INSERT INTO orders (tenant_id, customer_id, amount_cents)
VALUES ($1, 'cus_123', 4999);

UPDATE idempotency_keys
   SET status = 'completed', response_code = 201, response_body = $5, updated_at = now()
 WHERE tenant_id = $1 AND endpoint = $2 AND idem_key = $3;
COMMIT;

-- Duplicate path. If request_hash <> $4, return 422 and do not replay.
-- If status = 'completed', replay response_code and response_body.
SELECT status, request_hash, response_code, response_body
  FROM idempotency_keys WHERE tenant_id = $1 AND endpoint = $2 AND idem_key = $3;

This PostgreSQL example scopes the key table by tenant, endpoint and key, then claims the key and writes the order in one transaction. A bounded lock wait turns a concurrent duplicate into a 409, and a lookup on conflict returns 422 for a hash mismatch or replays the stored response.

Failure windows: stuck keys, cached errors, external calls and expiry

Stuck in-progress keys

If a worker crashes after claiming a key but before recording a result, the key stays in_progress and every retry gets a conflict. You can attach a lease timeout so the key can be reclaimed after it expires. The risk is that a slow request that is still running gets executed a second time. Set the lease well above the 99th-percentile handler latency, and only shorten it if the operation is safe to run twice.

Which errors to cache

Outcome Example Action
Deterministic failure 400 validation, 422 insufficient funds Store and replay
Transient failure Timeout, 503, deadlock Release the key so the retry runs
Success 201 Created Store and replay

Replaying a validation error is correct because the same input will fail again. Caching a timeout is wrong because it permanently blocks a request that could still succeed.

External calls

Model the handler as a state machine with recovery points: created, charge_sent, charge_confirmed. A retry resumes from the last committed step instead of starting over. For downstream calls, pass a key derived from yours, such as the original key plus a step name, so the downstream system deduplicates too. If a third party has no idempotency support, you need reconciliation against its records or a compensating action such as a refund.

Expiry

Set the time-to-live longer than the longest client retry window, and state that window in the API contract. Delete expired records on a schedule. Longer retention increases storage and keeps stored request bodies around longer, so store hashes rather than raw payloads where you can.

-- PostgreSQL. $n are bind parameters supplied by the handler.
CREATE TABLE idempotency_keys (
  key           text PRIMARY KEY,
  request_hash  text NOT NULL,        -- hash of the body, not the raw payload
  status        text NOT NULL,        -- 'in_progress' | 'completed'
  response_code int,
  response_body jsonb,
  locked_until  timestamptz NOT NULL, -- lease; set well above p99 handler latency
  expires_at    timestamptz NOT NULL  -- TTL longer than the client retry window
);

-- Claim: insert a new key, or take over a crashed worker's expired lease.
-- No row returned means another worker holds the lease or a result exists.
INSERT INTO idempotency_keys (key, request_hash, status, locked_until, expires_at)
VALUES ($1, $2, 'in_progress', now() + interval '60 seconds', now() + interval '48 hours')
ON CONFLICT (key) DO UPDATE
  SET locked_until = now() + interval '60 seconds'
  WHERE idempotency_keys.status = 'in_progress'
    AND idempotency_keys.locked_until < now()
    AND idempotency_keys.request_hash = EXCLUDED.request_hash
RETURNING key;

-- Success or deterministic failure (201, 400, 422): store it for replay.
UPDATE idempotency_keys
SET status = 'completed', response_code = $2, response_body = $3
WHERE key = $1 AND status = 'in_progress';

-- Transient failure (timeout, 503, deadlock): release so the retry runs.
DELETE FROM idempotency_keys
WHERE key = $1 AND status = 'in_progress';

-- Scheduled cleanup of records past their documented retention window.
DELETE FROM idempotency_keys
WHERE expires_at < now();

A PostgreSQL idempotency table that uses a lease to reclaim keys stuck in progress, stores deterministic outcomes for replay, releases keys on transient errors, and removes expired records on a schedule.

Propagating idempotency across services and message consumers

An idempotency key protects only the service that checks it. When a handler calls other services, a retry of the outer request re-runs those calls too, so each downstream call needs its own key that is stable across attempts. Generating a fresh random key per outbound call defeats this: the retry looks like new work. Instead, derive child keys deterministically, for example by hashing the incoming key together with a step name such as reserve-inventory or charge-card. A retried parent then produces identical child keys, and each downstream service deduplicates on its own. Avoid forwarding the parent key unchanged. If one request makes two distinct calls to the same service, both calls would carry the same key, and the second would be treated as a replay of the first.

Message consumers need the same guarantee. Most brokers deliver at least once. A consumer that commits its side effect but crashes before acknowledging the message will receive it again. To handle this, give each message a producer-assigned ID. Do not use a broker offset or delivery tag, because those can change on redelivery. Keep a processed-messages table with a unique constraint on that ID. In a single database transaction, insert the ID and apply the side effect. If the insert hits a constraint violation, the message was already handled, so skip it and acknowledge. Acknowledge only after the commit succeeds.

This approach works only when the side effect lives in the same database as the table. Calls to external systems still need derived keys. Keep processed IDs for longer than the broker's maximum redelivery window.

import hashlib
import psycopg2

# Schema (run once):
#   CREATE TABLE processed_messages (
#     message_id   text PRIMARY KEY,         -- producer-assigned ID, not a broker offset
#     processed_at timestamptz NOT NULL
#   );

def child_key(parent_key: str, step: str) -> str:
    # Same parent key + step name gives the same child key on every retry,
    # and different steps get different keys even when they call the same service.
    return hashlib.sha256(f"{parent_key}:{step}".encode()).hexdigest()

def handle_message(conn, message_id: str, order_id: str, amount_cents: int) -> None:
    # psycopg2: `with conn` commits on normal exit and rolls back on exception.
    with conn:
        with conn.cursor() as cur:
            cur.execute(
                "INSERT INTO processed_messages (message_id, processed_at) "
                "VALUES (%s, now()) ON CONFLICT (message_id) DO NOTHING",
                (message_id,),
            )
            if cur.rowcount == 0:
                return  # An earlier delivery was already handled. Skip it; the caller still acks.
            # The side effect is in the same database and the same transaction as the dedup row.
            cur.execute(
                "UPDATE orders SET paid_cents = paid_cents + %s WHERE id = %s",
                (amount_cents, order_id),
            )

# Consumer loop (the broker client is a placeholder):
# for msg in consumer:
#     handle_message(conn, msg.headers["message-id"], msg.body["order_id"], msg.body["amount_cents"])
#     msg.ack()  # ack only after the commit succeeded; if we crash before this, redelivery is deduplicated
#
# An external call made while handling a request uses a derived key instead:
# payments.charge(..., idempotency_key=child_key(incoming_key, "charge-card"))

An at-least-once consumer writes a processed-message row and its side effect in a single PostgreSQL transaction, and derives deterministic child keys for downstream calls.

When to avoid idempotency keys and what to use instead

Not every endpoint needs a key table. A PUT that replaces a resource at a known URI, or a DELETE that leaves it absent, already produces the same end state on every retry. Adding keys there only adds storage and latency. Low-value operations, such as marking a notification as read, rarely justify the cost either. Keys also fail when a client generates a new key for each retry attempt instead of once per logical operation. The server then treats every attempt as a new request. Before relying on the pattern, check how your client libraries create keys.

Alternative How it deduplicates Main limitation
Natural-key unique constraint The database rejects a second row with the same business key, such as an order reference Only works when a stable business key exists
Conditional request or version check A write applies only if the resource version still matches The client must read before it writes, and conflicts surface as errors
Client-supplied ID with PUT The client chooses the resource ID, so a retry targets the same row Clients must generate collision-resistant IDs
Separate in-memory key-value store The key is checked in a fast cache before the write No shared transaction with the business data

The separate cache needs particular care. Checking a key there is faster, but the check and the write no longer commit atomically. A crash between them leaves the two out of step. Eviction or a failover to a replica that has not yet received the key can also drop it silently, which reopens the duplicate window. Whichever mechanism you choose, keep a natural-key uniqueness constraint in the database as a backstop. It is the last layer that still rejects a duplicate when every layer above it has failed.

Testing for exactly one side effect, and the impact on cost and engineering time

Fault-injection tests

Happy-path tests prove little here, because the bugs live in the gaps between steps. Three tests cover most of them.

  • Crash after commit. Kill the process after the transaction commits but before the response is written, then retry with the same key. The client should receive the stored response, and the ledger should show exactly one row.
  • Concurrent duplicates. Fire N identical requests in parallel under one key. Exactly one should execute. The rest should get the replay or an in-progress conflict.
  • Mismatched payload. Reuse a key with a different body. The service should reject it rather than silently replay the first result.

In every case, assert on the side effect itself: rows written, messages published, and calls recorded by a stub of the downstream provider. Status codes are not enough, because a correct-looking 200 can sit on top of a second charge.

What it costs, and what it saves

Every mutating request adds at least one insert and one update to the key table. That makes it a hot, write-heavy table that needs an index on the client-scoped key and a cleanup job for expired rows. Canonicalizing payloads for fingerprinting also takes real effort. You have to settle field order, defaults, and number formats.

Illustrative estimate (assumed numbers, not a benchmark):

hours saved per year = (incidents × hours per investigation and reconciliation) + (endpoints × hours of bespoke dedup logic)

Assume 12 duplicate-charge incidents at 6 hours each and 10 endpoints at 8 hours each. That gives 72 + 80 = 152 hours saved per year. Compare it with your own build and maintenance estimate.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to idempotency keys microservices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.