Idempotency Keys in Microservices: Safe Retries for POST Requests

What idempotency keys are and why POST retries need them

An idempotency key is a client-supplied identifier that tells the server, "this request is the same logical operation as any earlier request carrying this key." The server uses it to perform the side effect once and return the same outcome on every repeat. HTTP defines GET, PUT and DELETE as idempotent, but not POST. Sending "create a payment for $50" twice creates two payments unless the server has a way to tell the second request is a duplicate.

Retries are routine in a distributed system, and they come from several layers:

• Client timeouts: the caller gives up waiting and sends the request again.
• Load balancer and service-mesh retries: proxies replay requests after connection resets or upstream errors, often without the application knowing.
• Queue redelivery: at-least-once brokers redeliver a message when an acknowledgment is lost.
• User double-submits: someone clicks "Pay" twice or refreshes a form.

The dangerous case is a timeout that happens after the server has committed. The order exists and the card was charged, but the response never arrived. From the client's side, this looks the same as a request that never landed, so retrying is the reasonable thing to do. Without deduplication, that retry produces a second charge.

The common convention is an Idempotency-Key request header. The client generates one unique value, such as a random UUID, for each logical operation and sends that same value on every retry. The header is an emerging standard. Not every client, SDK or proxy generates it or preserves it.

This guide argues that a response cache placed in front of a handler gives false confidence. Correct idempotency has to be enforced where the side effect is committed.

Who should use idempotency keys, and when to skip them

The pattern pays for itself where a duplicate POST causes harm that is expensive or impossible to undo. Payments and refunds come first: a retried charge moves real money and generates support tickets. Order creation, account or tenant provisioning, sending email or SMS, and allocating scarce inventory belong in the same group. The common thread is a POST with external or costly side effects that sits behind something that retries automatically. That retrier might be a mobile client on a flaky network, an SDK with built-in backoff, a service mesh, a load balancer, or a queue consumer that redelivers after a timeout.

Skip or defer it in these cases:

• Operations that are already idempotent. A PUT that replaces the full resource, or an upsert keyed on a natural identifier such as an external order number, converges to the same state however many times it runs. A key adds storage and latency without adding safety.
• Read-only endpoints. There is nothing to deduplicate.
• Low-stakes writes. If a duplicate "like" or log entry is cheap to detect and clean up later, an asynchronous reconciliation job is often simpler.
• Clients that won't keep the key. If a client generates a fresh key on every attempt, the server sees each retry as a new request. Fix the client first, or derive the key server-side from stable request fields.

Be precise about the guarantee. An idempotency key gives at-most-once execution per key within the retention window. It does not give exactly-once semantics across the system. Requests that arrive after the record expires, or that use a different key, will run again.

How it works: the idempotency record and the three request states

The server keeps one record per key. Its fields are:

• Scope. This is the tenant, or the caller plus endpoint. Two clients that happen to generate the same key must never collide.
• Fingerprint. A hash of the request payload.
• Status. One of in-progress, completed, or failed.
• Stored response. The response code and body.
• Timestamps. Created and expiry times.

The fingerprint catches clients that reuse a key for a different request. The expiry timestamp drives cleanup.

Store a response only if replaying it later would be correct. Successful results qualify. So do deterministic client errors such as a validation failure, because the same input will fail the same way. Transient 5xx failures usually should not be stored. Mark the record failed or delete it so a retry with the same key can execute again. If you cache a timeout, the client stays locked out of an operation that never completed.

Fingerprinting has a trade-off. Hashing raw bytes treats semantically identical requests as different. Reordered JSON fields, whitespace, or a client-generated timestamp in the body will all change the hash and cause false rejections. Before hashing, normalize the payload: sort keys, strip insignificant whitespace, and exclude fields that change between attempts.

Implementing it atomically in the primary database

Keep the record in the same database as the business data. A minimal idempotency_keys table holds scope (tenant or API client ID), key, request_hash, status, response_code, response_body, locked_until, and created_at, with a UNIQUE (scope, key) constraint. Scoping keeps two tenants who choose the same key from colliding.

The obvious approach is to SELECT the key and do the work if no row exists, but that check-then-act sequence races. Two retries arriving milliseconds apart both see no row, and both charge the card. The unique constraint closes that gap. The handler opens a transaction, INSERTs the row as in_progress, performs the business write, updates the row to completed with the serialized response, and commits. A concurrent duplicate blocks on the insert. Once the winner commits, the duplicate gets a unique-violation error. The handler catches it, reads the committed row, and either replays the stored response or returns 409 if the row is still in progress.

In-progress leases

Leases matter when the work can't fit in one transaction. For example, the handler may commit the in_progress row before calling an external system. In that case, set locked_until to now plus a lease. If the worker crashes, the row stays stuck in progress. After the lease expires, a retry can claim it with a conditional UPDATE ... WHERE locked_until < now() and resume.

Choosing the lease length is a tradeoff:

• Too short: a slow but healthy worker loses its lease, and a second worker duplicates the side effect.
• Too long: after a real crash, clients keep receiving 409s until the lease expires.

Set the lease comfortably above the operation's worst observed duration.

Why not a separate cache?

A separate cache is easier to add, but it loses atomicity with the side effect. Suppose the business write commits and the process crashes before the cache write. The retry finds nothing and runs again. Reversing the order fails differently: if the cache is marked completed and the business write never commits, you get a phantom success.

Copy-- PostgreSQL. Idempotency records live next to the business data.
CREATE TABLE idempotency_keys (
  scope         text        NOT NULL,  -- tenant or API client ID
  key           text        NOT NULL,  -- client-supplied Idempotency-Key
  request_hash  text        NOT NULL,  -- detects key reuse with a different body
  status        text        NOT NULL CHECK (status IN ('in_progress', 'completed')),
  response_code int,
  response_body jsonb,
  locked_until  timestamptz,           -- lease; only used when work spans transactions
  created_at    timestamptz NOT NULL DEFAULT now(),
  CONSTRAINT idempotency_keys_scope_key UNIQUE (scope, key)
);

-- Happy path: the key and the side effect commit or roll back together.
BEGIN;
INSERT INTO idempotency_keys (scope, key, request_hash, status)
VALUES ('tenant_42', 'k-7f3a9c', 'sha256:3b1e', 'in_progress');
-- A concurrent duplicate blocks on this INSERT. If the winner commits, the duplicate
-- fails with unique_violation (SQLSTATE 23505). Its handler then reads the row and
-- replays the stored response, or returns 409 if the row is still in progress.
INSERT INTO charges (tenant_id, amount_cents, currency)
VALUES ('tenant_42', 4999, 'USD');
UPDATE idempotency_keys
   SET status = 'completed', response_code = 201,
       response_body = '{"charge_id": "ch_123"}'
 WHERE scope = 'tenant_42' AND key = 'k-7f3a9c';
COMMIT;

-- Lease recovery: used when the in_progress row was committed before an external call.
-- The row is reclaimed only if the previous holder's lease has expired.
-- If zero rows are returned, the lease is still held, so the handler returns 409.
UPDATE idempotency_keys
   SET locked_until = now() + interval '30 seconds'
 WHERE scope = 'tenant_42' AND key = 'k-7f3a9c'
   AND status = 'in_progress' AND locked_until < now()
RETURNING request_hash;

A PostgreSQL schema with a UNIQUE (scope, key) constraint, a single transaction that records the key and performs the business write atomically, and a conditional UPDATE that reclaims an expired in-progress lease.

Propagating keys across services and handling non-transactional side effects

A key that protects only the first hop leaves the rest of the call chain unprotected. Downstream services need keys that stay identical across retries, and there are two ways to get them. Pass-through forwards the client's Idempotency-Key header unchanged, which works when each service performs one operation per request. Derivation computes each downstream key as a hash of the parent key plus a step name, such as hash(parent + "charge"). This keeps keys unique per step while staying stable on every retry. Minting a fresh random key at each hop breaks the guarantee: a retried order request sends the payment service a key it has never seen, so the customer is charged twice.

Copyclient retry (key K)
  └─ gateway ─────────── forwards K, does not dedupe
       └─ order svc ──── dedupes on K, returns stored response
            ├─ payment ─ dedupes on H(K,"charge")
            └─ outbox ── event id H(K,"order-created")
                 └─ broker ─ consumer dedupes on event id

Some side effects cannot join the database transaction. For events, write them to an outbox table in the same transaction as the order, and let a relay publish them afterward. For third-party APIs that accept an idempotency key, send the derived key so the provider does the deduplication. Brokers usually deliver at least once, so consumers must record processed event IDs before acting, or atomically with acting.

Gateway and service-mesh retries multiply attempts at every layer. Retrying three times at each of three layers can produce up to 27 calls (illustrative estimate). These retries are safe only if the key is preserved. They can also hide a slow dependency until it turns into a retry storm.

Copyimport hashlib
import uuid

def derive_key(parent_key: str, step: str) -> str:
    # Deterministic: same parent + step always gives the same child key.
    # The separator keeps ("ab", "c") distinct from ("a", "bc").
    return hashlib.sha256(f"{parent_key}\x00{step}".encode()).hexdigest()

charges_seen: set[str] = set()  # stands in for the payment service's dedupe table

def payment_charge(idempotency_key: str, amount_cents: int) -> str:
    if idempotency_key in charges_seen:
        return "duplicate ignored"
    charges_seen.add(idempotency_key)
    return f"charged {amount_cents}"

def handle_order(client_key: str, mint_fresh: bool = False) -> dict:
    # Correct: derive one stable key per downstream step.
    # Broken: a random key per hop is new on every retry.
    charge_key = str(uuid.uuid4()) if mint_fresh else derive_key(client_key, "charge")
    event_id = derive_key(client_key, "order-created")  # becomes the outbox event id
    return {"payment": payment_charge(charge_key, 4999), "event_id": event_id[:12]}

client_key = "7f3c9a2e-5b1d-4e8a-9c6f-2d4b8e1a0f35"  # from the client's Idempotency-Key header

# Derived keys: the retry reaches payment with the same key and is deduped.
print(handle_order(client_key))  # payment: charged 4999
print(handle_order(client_key))  # payment: duplicate ignored

# Fresh key per hop: the retry looks new to payment, so the customer is charged twice.
charges_seen.clear()
print(handle_order(client_key, mint_fresh=True))  # payment: charged 4999
print(handle_order(client_key, mint_fresh=True))  # payment: charged 4999

Deriving downstream keys by hashing the parent key with a step name keeps them stable across retries, so the payment service deduplicates the retry. Minting a fresh UUID per hop lets the same retry charge the customer twice.

Retention, expiry, and testing for exactly one side effect

Set the TTL from the longest retry window any caller can produce, not the typical one. Add client backoff ceilings, message redelivery delays, and mobile clients that queue requests offline and resend hours later, then add margin. If the record expires inside that window, the key provides no protection for the retry it was meant to cover.

Document what expiry means: once the record is gone, the server has no memory of the key. A retry after expiry is treated as a new request and runs again. State that the guarantee is time-bounded in your API contract.

For cleanup, store expires_at, index it, and delete in small batches so the job does not hold long locks. On high-throughput tables, time-based partitions you drop whole are usually cheaper than row deletes. A replayed response reflects state at the original request. If the resource has changed since, the replay is stale, so clients that need current state should follow up with a GET.

Testing for exactly one side effect

Run these tests against the real database:

• Crash after commit: inject a fault that kills the process after commit but before the response is sent. Retry with the same key.
• Concurrent duplicates: fire concurrent duplicates with the same key.
• Mismatched payload: reuse a key with a different payload and expect a rejection.

In each case, assert on side effects, not status codes. Count ledger rows, outbox messages, and calls recorded by a fake downstream service. Each count should be exactly one.

Some client bugs are invisible to the server. A client that regenerates the key on every retry looks the same as one sending genuine new requests. The same is true of a client that creates the key after the first attempt or reuses one key across unrelated operations. Generate the key once, inside a shared client library, before the first attempt.

Copyimport uuid
from concurrent.futures import ThreadPoolExecutor
import pytest

# `api`, `db`, `fault` and `downstream` are project fixtures (assumed): a real database,
# a restartable service process, a fault injector, and a fake downstream that records calls.
BODY = {"account_id": "acct_123", "amount_cents": 500}

def post(api, key, body=BODY):
    return api.post("/payments", json=body, headers={"Idempotency-Key": key})

def assert_exactly_one(db, downstream, key):
    # Count side effects, not status codes.
    assert db.scalar("SELECT count(*) FROM ledger_entries WHERE idempotency_key = %s", key) == 1
    assert db.scalar("SELECT count(*) FROM outbox WHERE idempotency_key = %s", key) == 1
    assert downstream.calls_for(key) == 1

def test_crash_after_commit_then_retry(api, db, fault, downstream):
    key = str(uuid.uuid4())
    fault.kill_after_commit()  # process dies after COMMIT, before the response is written
    with pytest.raises(ConnectionError):
        post(api, key)
    api.restart()
    assert post(api, key).status_code == 201  # replayed from the stored response
    assert_exactly_one(db, downstream, key)

def test_concurrent_duplicates(api, db, downstream):
    key = str(uuid.uuid4())
    with ThreadPoolExecutor(max_workers=10) as pool:
        statuses = [r.status_code for r in pool.map(lambda _: post(api, key), range(10))]
    # Losers may see 409 (in progress) or a replay; only one request may do the work.
    assert set(statuses) <= {201, 409}
    assert_exactly_one(db, downstream, key)

def test_same_key_different_payload_is_rejected(api, db, downstream):
    key = str(uuid.uuid4())
    assert post(api, key).status_code == 201
    assert post(api, key, {**BODY, "amount_cents": 900}).status_code == 422
    assert_exactly_one(db, downstream, key)

A pytest sketch, assuming project-specific fixtures, that tests three cases against a real database: a crash after commit, concurrent duplicate requests, and a reused key with a different payload, and asserts exactly one ledger row, one outbox message, and one downstream call in each case.

Alternatives and the impact on cost and engineering time

Idempotency keys are not the only way to make a retried POST safe. Several cheaper options cover narrower cases.

Cost

Every mutating request adds a record write and a lookup. You also need a cleanup job, recovery logic for expired leases, and clients that generate keys correctly.

Illustrative estimate (assumed numbers, not a benchmark). Use this formula:

storage = requests per second × 86,400 × retention days × record size

Assume 200 requests per second, 1 day of retention, and 1 KB per record (stored response included). That gives 200 × 86,400 × 1 × 1 KB ≈ 17 GB of live records. The table therefore needs indexing and steady expiry.

Savings

The pattern pays off by removing work elsewhere:

• Fewer incidents. Duplicate-charge incidents and manual reconciliations largely disappear.
• Simpler retry policies. They no longer need per-endpoint exceptions.
• Safer infrastructure retries. Gateways and service meshes can retry timed-out POSTs without someone first proving the downstream handler is safe.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to idempotency keys microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.