Idempotency Keys for APIs: Safe Retries Without Duplicate Writes

Why retries create duplicate requests and what an idempotency key is

A client that sends a mutating request and gets a timeout cannot tell what happened. Maybe the request never reached the server. Maybe the server committed the write and the response was lost on the way back. From the client's side these two cases look the same. The safe choice for the client is to retry, but if the server already succeeded, a retry charges the card twice, creates a second order, or provisions a second VM.

HTTP semantics handle part of this. GET, PUT and DELETE are defined as idempotent. Sending them once or several times leaves the server in the same state. PUT replaces a resource with the same representation, and a repeated DELETE leaves the resource just as gone, even if the second call returns a different status code. POST makes no such promise. Each POST may create something new, so retrying it blindly produces duplicate requests.

An idempotency key closes that gap. The client generates a unique value for one logical operation and sends it with every attempt of that operation. The server records the key with the result of the first attempt that completes. When a later attempt arrives with the same key, the server returns the stored result and does not execute the operation again. This makes a POST idempotent: retries stay safe because the server deduplicates them.

You need this if you expose mutating APIs, especially for payments, orders and resource provisioning. It also applies to any endpoint that something retries automatically, such as an SDK with built-in retries, an API gateway, a load balancer, or a queue consumer with at-least-once delivery.

The client-server contract: one key per logical operation

The client generates a random identifier, typically a UUIDv4, when the user or upstream system initiates an operation. It sends that value in a request header such as Idempotency-Key and reuses it on every retry of that operation. A new operation gets a new key, even if its payload happens to be identical. "Charge this cart" twice on purpose is two operations. A timeout followed by a resend is one.

The server scopes keys to the authenticated principal: the API account, tenant, or credential that made the call. Without that scope, one caller could collide with another's key, by accident or deliberately, and receive someone else's stored response.

Keys are kept for a limited time, and that retention window belongs in the published API contract. A client retrying after the window closes risks a duplicate write, so client authors need to know how long their retry budget can safely extend.

Two client mistakes break the guarantee without raising any errors:

• Minting a fresh key per attempt. Every retry looks like a new operation to the server, so duplicates go through.
• Reusing one key across operations. For example, a key stored per session. Later requests silently return the first request's result.

A minimal client does the following:

• Creates the key once, outside the retry loop.
• Retries only on connection errors, timeouts, and retryable statuses such as 429 and 503.
• Waits with exponential backoff plus random jitter, so clients that failed together do not retry together.
• Returns any other 4xx response to the caller unchanged.

Copyimport random
import time
import uuid

import requests

RETRYABLE_STATUSES = {429, 502, 503, 504}


def post_once(session, url, payload, max_attempts=5, base_delay=0.5, max_delay=8.0):
    # One key per logical operation: minted once, OUTSIDE the retry loop,
    # so every attempt below is recognized by the server as the same request.
    headers = {"Idempotency-Key": str(uuid.uuid4())}

    for attempt in range(max_attempts):
        try:
            resp = session.post(url, json=payload, headers=headers, timeout=(3, 10))
            if resp.status_code not in RETRYABLE_STATUSES:
                # Success, or a non-retryable error such as 400/409/422:
                # hand it back unchanged. Retrying would not help.
                return resp
        except (requests.ConnectionError, requests.Timeout):
            # The request may or may not have reached the server; the
            # same key makes it safe to send again.
            if attempt == max_attempts - 1:
                raise
        else:
            if attempt == max_attempts - 1:
                return resp  # out of attempts: surface the last 429/5xx

        # Exponential backoff with full jitter, so clients that failed
        # together spread out. Keep the total retry time well inside the
        # server's documented key retention window.
        time.sleep(random.uniform(0, min(max_delay, base_delay * 2 ** attempt)))


# A deliberate second charge is a new operation and gets a new key.
resp = post_once(requests.Session(), "",
                 {"cart_id": "cart_123", "amount": 4200, "currency": "usd"})

A Python client that generates one Idempotency-Key per logical operation and reuses it across retries. It retries only on connection errors, timeouts and 429/502/503/504, using exponential backoff with full jitter.

How it works: the server-side record and request lifecycle

The server keeps one row per key, in a table whose unique constraint is on (principal, idempotency key). Scoping the key to the authenticated caller prevents one tenant's key from colliding with, or reading, another tenant's stored response. Each row holds:

• Fingerprint: a hash of method, path and canonicalized body. Canonicalization means fixed key order and normalized whitespace, so a semantically identical retry hashes the same.
• Status: in_progress or completed.
• Response code and body: written when the work finishes.
• Created and expiry timestamps: these bound storage growth and define how long a retry stays safe.

Every request begins with an atomic insert-or-fetch. Only one concurrent request can win the insert. Every other request reads the existing row.

Copyrequest + key
     │
insert row (in_progress) ──won──► run work ─► store response, mark completed
     │ lost (row exists)
     ▼
fingerprint matches? ──no──► 422
     │ yes
status? ──in_progress──► 409 + Retry-After
     │ completed
     ▼
replay stored code + body

The replay returns the original status code as well as the body. If the server re-ran validation instead, a retry could see a different result than the first attempt did, and the client would have no way to reconcile the two.

Copy+----------+  POST + Idempotency-Key   +-------------------------------+
|  Client  | ------------------------> |          API server           |
|          | <------------------------ |  1. authenticate -> principal |
+----------+   result, replay, 422,    |  2. fingerprint = hash(       |
               or 409 + Retry-After    |     method, path,             |
                                       |     canonicalized body)       |
                                       +-------------------------------+
                                          |                      |
                                          |  3. atomic           | 4. won:
                                          |  insert-or-fetch     | run work
                                          v                      v
            +-----------------------------------------+  +----------------+
            | idempotency_keys                        |  | Business logic |
            | UNIQUE (principal, idempotency_key)     |  | + primary data |
            |-----------------------------------------|  +----------------+
            | fingerprint    hash of request          |          |
            | status         in_progress | completed  |          |
            | response_code  set on completion        | <--------+
            | response_body  set on completion        |  5. store response,
            | created_at     insert time              |     mark completed
            | expires_at     end of retry window      |
            +-----------------------------------------+

The API server fingerprints each keyed request and does an atomic insert-or-fetch on a table unique on (principal, idempotency key). The winner runs the work and stores its response. Later requests get a replay, 422, or 409.

Implementing it: atomic claims, fingerprints and transactional responses

The storage is a table keyed by (client_id, idempotency_key), with a primary key or unique constraint on that pair. Other columns hold a request fingerprint, a status (in_progress or completed), the response status code, the response body, and a creation timestamp. The constraint makes the claim atomic. Without it, two concurrent requests can both check for the key, both find nothing, and both execute.

To claim a key, insert the row as in_progress with INSERT ... ON CONFLICT DO NOTHING (PostgreSQL syntax; other databases have equivalents). If a row was inserted, this request owns the key. If not, read the existing row and branch on its contents:

• Fingerprint mismatch: reject with 422, because the client reused a key for a different operation.
• Status in_progress: return 409 so the client retries later.
• Status completed: replay the stored response.

Compute the fingerprint by hashing the method, the path and a canonical body. Parse the JSON, sort the keys and re-serialize without whitespace before hashing. Otherwise a client library that reorders fields on retry triggers a false mismatch.

Write the business change and the update of the key row to completed, with its response, in one database transaction. If they commit separately, a crash between the two leaves the side effect done but the record unfinished. If you treat that key as reclaimable, the retry duplicates the write. If you leave it stuck, the original result, such as a created resource ID, is lost.

Persist final outcomes only: successes and deterministic 4xx validation errors. Do not persist 5xx responses or timeouts. Roll back or delete the claim instead, so the client's retry can run the operation rather than replay a transient failure.

Copy-- PostgreSQL. The primary key on (client_id, idempotency_key) makes the claim atomic.
CREATE TABLE idempotency_keys (
  client_id           text        NOT NULL,
  idempotency_key     text        NOT NULL,
  request_fingerprint text        NOT NULL,  -- sha256 hex of method + path + canonical JSON body
  status              text        NOT NULL CHECK (status IN ('in_progress', 'completed')),
  response_code       integer,
  response_body       jsonb,
  created_at          timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (client_id, idempotency_key)
);

-- Step 1: claim the key in its own short transaction so concurrent requests see it.
-- One row returned: this request owns the key. Zero rows: SELECT the existing row and
-- branch on it (fingerprint mismatch -> 422, in_progress -> 409, completed -> replay).
INSERT INTO idempotency_keys (client_id, idempotency_key, request_fingerprint, status)
VALUES ('client_42', '7c9e6679-7425-40de-944b-e07fc1f90ae7', '3f1a9c...e8', 'in_progress')
ON CONFLICT (client_id, idempotency_key) DO NOTHING
RETURNING idempotency_key;

-- Step 2: the side effect and the stored final response commit together or not at all.
BEGIN;
INSERT INTO payments (id, client_id, amount_cents, currency)
VALUES ('pay_8f2d', 'client_42', 2500, 'EUR');

UPDATE idempotency_keys
SET status = 'completed',
    response_code = 201,
    response_body = '{"payment_id": "pay_8f2d"}'
WHERE client_id = 'client_42'
  AND idempotency_key = '7c9e6679-7425-40de-944b-e07fc1f90ae7'
  AND status = 'in_progress';
COMMIT;

-- Step 3 (failure path): on a 5xx or timeout, after rolling back step 2, release the
-- claim instead of storing the failure, so the client's retry re-executes the operation.
DELETE FROM idempotency_keys
WHERE client_id = 'client_42'
  AND idempotency_key = '7c9e6679-7425-40de-944b-e07fc1f90ae7'
  AND status = 'in_progress';

A PostgreSQL schema with a unique key on (client_id, idempotency_key), an atomic INSERT ... ON CONFLICT DO NOTHING claim, a single transaction that commits the business write together with the completed response, and claim release on transient failures.

External side effects, crashes and stale in-progress keys

The idempotency record only protects state inside your database transaction. A charge at a payment provider, an email, or a message on a queue lives outside it, and a retry after a crash can repeat them.

Propagate a derived key downstream

If a dependency accepts its own idempotency key, as many payment providers do, send it a key derived from yours. Hashing the client key together with the step name works well. Each retry then sends the same downstream key, so the provider deduplicates the call. Use a separate key for each distinct downstream call so that a refund and a charge never collide.

Drive other effects from committed state

Emails and message publishes usually offer no dedupe. Write an outbox row in the same transaction as the business change and the stored response. A separate relay reads committed rows, sends them, and marks them done. The send happens only if the write committed. The relay can still crash after sending and before marking the row, so consumers will sometimes see a message twice.

This is at-least-once delivery plus reconciliation, not exactly-once. Without idempotency at the receiver, duplicates can only be made rarer and detectable.

Recover stale in-progress records

Give each claim a lease: a locked_until timestamp that the worker extends while it runs. A record still marked in-progress after its lease has expired belongs to a dead worker. Classify it before acting:

• Resumable: the record holds checkpoints, such as a stored downstream key, so a new worker can continue from the last completed step.
• Safely retryable: no external call was made, or every external call was idempotent. A new worker can re-claim the record and run it again.
• Needs manual reconciliation: a non-idempotent external call may have happened. Flag the record and query the dependency for the actual outcome. Never guess.

Copy-- PostgreSQL sketch. 1) Request transaction: business write, stored response
-- and outbox row commit together or not at all.
BEGIN;
INSERT INTO orders (id, customer_id, total_cents) VALUES ('ord_123', 'cus_42', 4999);
INSERT INTO outbox (id, topic, payload, dedupe_key)
VALUES (gen_random_uuid(), 'order.created', '{"order_id":"ord_123"}',
        'idem_7f3a:order.created');  -- derived key lets consumers drop repeats
UPDATE idempotency_keys
   SET status = 'completed', response_code = 201,
       response_body = '{"order_id":"ord_123"}'
 WHERE key = 'idem_7f3a' AND locked_by = 'worker-a';
COMMIT;

-- 2) Relay: only committed rows are visible. SKIP LOCKED lets relays run in parallel.
BEGIN;
SELECT id, topic, payload, dedupe_key FROM outbox
 WHERE sent_at IS NULL ORDER BY created_at LIMIT 100
   FOR UPDATE SKIP LOCKED;
-- ...publish each row; a crash here means a resend later (at-least-once)...
UPDATE outbox SET sent_at = now() WHERE id = ANY('{...}'::uuid[]);
COMMIT;

-- 3) Lease heartbeat: the owning worker extends its claim while it runs.
UPDATE idempotency_keys
   SET locked_until = now() + interval '30 seconds'
 WHERE key = 'idem_7f3a' AND locked_by = 'worker-a';

-- 4) Recovery: take over a stale in-progress record whose lease expired.
-- recovery_point says how to classify it: a stored checkpoint means resume,
-- 'started' means retry, and 'external_call_unconfirmed' means reconcile manually.
UPDATE idempotency_keys
   SET locked_by = 'worker-b',
       locked_until = now() + interval '30 seconds'
 WHERE key = 'idem_7f3a'
   AND status = 'in_progress'
   AND locked_until < now()
RETURNING recovery_point, downstream_key;

A PostgreSQL sketch that commits an outbox row with the business write, relays it with FOR UPDATE SKIP LOCKED, extends a lease with a heartbeat and reclaims expired in-progress keys by their recovery point.

When to avoid it and the alternatives

Idempotency keys add a storage write, a lookup and a cleanup job to every request. Skip them when the operation already tolerates repetition:

• Naturally idempotent operations. A PUT to a client-chosen ID, or an update conditioned on the current version, leaves the same state no matter how many times it runs.
• Read-only endpoints. GET has no side effect to deduplicate.
• Low-stakes writes. For analytics pings or "last seen" timestamps, a duplicate costs less than the bookkeeping that would prevent it.
• Side effects you cannot deduplicate. If a downstream call has no key, no status query and no reversal, the server cannot tell whether that call already happened. A stored key here hides the gap rather than closing it.

Impact on cost and engineering time

Whichever storage option you pick, the pattern has a running cost. It is usually small, but it never goes away.

You can reduce fingerprint fragility by hashing a canonical form of the request after parsing and normalization, not the raw bytes.

Illustrative estimate (assumed numbers, not a benchmark)

Storage ≈ mutating requests per day × retention days × bytes per record

Assume 2,000,000 writes per day, 7 days of retention, and 2 KB per record. The record holds the key, the fingerprint, and the stored response. That gives 2,000,000 × 7 × 2 KB ≈ 28 GB before index overhead. The stored response body dominates the per-record size, so keep it small.

Treat the retention window as a business decision. It should cover the longest period over which a client might realistically retry, including queued jobs and offline mobile clients. Your storage budget is a secondary input. Product and support teams should agree on the window, because a key that expires too early turns a late retry back into a duplicate.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to idempotency keys api, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.