Idempotency Keys in Microservices: Safe Retries for Payment APIs

What an idempotency key is and why payment and order APIs need one

An idempotency key is a unique value the client generates for one logical operation, such as "charge this card for this order," and sends with every attempt at that operation. The server promises that all requests carrying the same key produce one effect and return the same result. This makes it a contract scoped to a single request, not a dedupe cache. A cache only suppresses repeats it happens to notice within some window. A contract defines what the server must do when a key arrives a second time, including while the first attempt is still running.

Retries are unavoidable because a timeout is ambiguous. If a client sends a charge and the connection drops before the response arrives, the client cannot tell which of these happened:

• The request never reached the server.
• The server failed partway through.
• The server committed the charge and the response was lost.

From the client's side all three look identical.

HTTP semantics already cover some methods. GET, PUT and DELETE are defined as idempotent: repeating them leaves the server in the same state, even if the response differs, such as a second DELETE returning 404. POST carries no such guarantee. A naive retry of "POST /charges" or "POST /orders" is a new instruction, so the customer is billed twice or two orders reserve the same inventory.

Teams exposing mutating POST endpoints with money or inventory side effects should adopt the pattern. It is needed for any non-idempotent operation that a client, SDK, load balancer or API gateway may retry automatically.

How the contract works: client keys, caller scope and request fingerprints

The client generates a random value, typically a UUID v4, once per logical operation, such as "charge this cart" or "create this order". It sends the same value on every retry of that operation. Generate it before the first attempt and persist it alongside the pending operation, so that a client crash or restart still retries with the original key. Two misuses defeat the pattern. A fresh key per retry turns every timeout into a potential duplicate charge. One key reused across distinct operations, such as two separate purchases, causes the second purchase to be rejected or silently replayed.

On the server, the effective key is the caller's identity (account, tenant or API credential) combined with the client-supplied value. Without that scope, two tenants that happen to generate the same string, or one tenant guessing another's key, could read each other's stored responses.

Alongside the key, store a hash of the canonical request parameters. If a request arrives with a known key but a different hash, reject it with a clear 4xx error such as 409 or 422 that names the mismatch. Replaying the old result would tell the client that a $50 charge succeeded when it actually asked for $500, and the bug would surface later as a reconciliation problem.

Canonicalization needs care:

• Sort keys and normalize formats before hashing. Reordered JSON fields or 10.0 versus 10 must not cause false mismatches.
• Apply defaults first. An omitted optional field and the same field sent explicitly with its default value mean the same operation.
• Hash every field that affects the outcome. Fields include amount, currency, destination and line items. Hashing only the amount misses real differences in the request.

Copyimport hashlib
import json
from decimal import Decimal

# Defaults are applied before hashing, so an omitted field equals its explicit default.
DEFAULTS = {"capture": True, "description": ""}
# Every field that changes the outcome is hashed, not just the amount.
FIELDS = ("amount", "currency", "destination", "line_items", "capture", "description")

class IdempotencyKeyMismatch(Exception):
    """Map to a 409 or 422 that names the mismatch. Never replay the old result."""

def fingerprint(params: dict) -> str:
    merged = {**DEFAULTS, **params}
    body = {k: merged.get(k) for k in FIELDS}
    # Normalize formats so 10, "10.0" and "10.00" hash the same.
    body["amount"] = str(Decimal(str(body["amount"])).quantize(Decimal("0.01")))
    body["currency"] = body["currency"].upper()
    # sort_keys plus fixed separators make field order and whitespace irrelevant.
    canonical = json.dumps(body, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

def check_idempotency(store: dict, caller_id: str, idem_key: str, params: dict):
    # Scope to the caller so equal keys from different tenants never collide.
    scoped_key = (caller_id, idem_key)
    fp = fingerprint(params)
    record = store.get(scoped_key)
    if record is None:
        # A production store needs an atomic insert-if-absent here.
        store[scoped_key] = {"fingerprint": fp, "response": None}
        return None  # first attempt: the caller executes the operation
    if record["fingerprint"] != fp:
        raise IdempotencyKeyMismatch(f"key {idem_key!r} was already used with different parameters")
    return record["response"]  # a retry of the same operation: replay the stored result

a = {"amount": "50.0", "currency": "usd", "destination": "acct_placeholder", "line_items": []}
b = {"line_items": [], "destination": "acct_placeholder", "currency": "USD", "amount": 50, "capture": True}
assert fingerprint(a) == fingerprint(b)                      # same operation, different formatting
assert fingerprint(a) != fingerprint({**a, "amount": "500"})  # a real difference is caught

A server-side check that scopes the idempotency key to the caller, hashes a canonicalized set of every outcome-affecting field, and rejects a reused key whose payload differs instead of replaying the old result.

The request lifecycle: atomic claim, execute, store and replay

The order of these steps is what makes retries safe:

claim → execute → store → replay

1. Claim. Insert the key with status in_progress. Use a unique constraint on (caller, key), or a conditional write that succeeds only if the key is absent. Do this before any side effect.
2. Execute the business operation.
3. Store the status code and response body against the key, and mark it completed.
4. Replay the stored response for any later request with the same key and fingerprint.

A check-then-act read does not give this guarantee. In that pattern, the service selects the key, finds nothing, then proceeds. Two retries arriving milliseconds apart can both read "absent" and both charge the card. Only an atomic insert or conditional write lets the database pick a single winner.

A concurrent duplicate that loses the claim while the original is still in_progress should get a conflict response, typically 409, ideally with a Retry-After header. Clients should treat it as retryable and back off. Once the original finishes, the retry receives the stored result.

Failures need an explicit caching policy:

Copy Client: POST + Idempotency-Key
        |
        v
+--------------------------------------------+
| 1. CLAIM  INSERT (caller, key) atomically  |
| status=in_progress, UNIQUE or cond. write  |
+--------------------------------------------+
   |                 |                    |
 inserted     dup, in_progress      dup, completed
   v                 v                    v
+------------+  +---------------+  +------------------+
| 2. EXECUTE |  | 409 Conflict  |  | 4. REPLAY stored |
| side effect|  | + Retry-After |  | status + body    |
+------------+  | retryable,    |  | same fingerprint |
   |            | back off      |  +------------------+
   |            +---------------+
   v
+------------------------------------------------------+
| 3. STORE status code + body; set status per policy   |
+------------------------------------------------------+
   |               |                  |
 2xx or 4xx     5xx before        timeout after
 (validation)   side effect       downstream call sent
   v               v                  v
+------------+  +---------------+  +----------------------+
| completed  |  | DELETE key    |  | keep in_progress     |
| replay it  |  | retry re-runs |  | reconcile, then      |
+------------+  +---------------+  | store or release     |
                                   +----------------------+

An atomic claim picks one winner. Concurrent duplicates get a retryable 409, completed keys replay the stored response, and the failure policy decides whether a key is stored, released or held for reconciliation.

Implementing idempotency key storage in a relational database

A single table holds the whole lifecycle. Give it these columns: scope (tenant or API client ID), key (the client-supplied value), request_hash (a digest of the canonicalized request), state (in_progress, completed, or failed_retryable), response_status, response_body, lease_expires_at, and created_at. A unique index on (scope, key) turns the claim into a single atomic operation. Scoping the index this way stops two tenants who generate the same UUID-like key from colliding.

Claiming the key

To claim a key, insert a row with state = 'in_progress' and a short lease. If the insert succeeds, this request owns the key. If it fails on the unique constraint, read the existing row and decide based on what you find:

• Hash mismatch: reject the request, because the key is being reused for a different payload.
• Completed: replay the stored response.
• In progress with a live lease: return a conflict.
• Expired lease: take the row over with a conditional UPDATE ... WHERE lease_expires_at < now().

Portable code catches the unique-violation error, SQLSTATE 23505. PostgreSQL also offers ON CONFLICT DO NOTHING, and MySQL offers INSERT IGNORE. Both avoid using exceptions for control flow, but each is specific to its database.

One transaction for the record and the write

When the operation is local, update the idempotency row to completed in the same transaction as the business write, such as the order or ledger row. Both commit, or neither does. That rules out a committed charge with no stored response, which would otherwise cause a retry to charge the customer twice.

Cleanup

A scheduled job deletes rows where created_at is older than your retention window. That window must be at least as long as the period during which clients may retry. Delete in bounded batches so long-held locks and replication lag don't build up. Partitioning by created_at lets you drop whole partitions instead of deleting rows one by one.

Copy-- Schema. Double-quoted "key" is standard SQL; MySQL needs ANSI_QUOTES or backticks instead.
CREATE TABLE idempotency_keys (
  scope            VARCHAR(64)  NOT NULL,  -- tenant or API client ID
  "key"            VARCHAR(255) NOT NULL,  -- client-supplied value
  request_hash     CHAR(64)     NOT NULL,  -- e.g. SHA-256 hex of canonicalized request
  state            VARCHAR(20)  NOT NULL,  -- in_progress | completed | failed_retryable
  response_status  INTEGER,
  response_body    TEXT,                   -- TEXT is widely supported but not standard SQL
  lease_expires_at TIMESTAMP    NOT NULL,
  created_at       TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE UNIQUE INDEX ux_idem_scope_key ON idempotency_keys (scope, "key");

-- 1. Claim. Success means this request owns the key.
--    SQLSTATE 23505 (unique violation): SELECT the existing row and branch on hash/state/lease.
--    :name placeholders depend on your driver.
INSERT INTO idempotency_keys (scope, "key", request_hash, state, lease_expires_at)
VALUES (:scope, :key, :hash, 'in_progress', CURRENT_TIMESTAMP + INTERVAL '30' SECOND);

-- 2. Takeover of an expired lease (hash already confirmed equal).
--    1 row updated = we own it; 0 rows = live lease elsewhere or already completed.
UPDATE idempotency_keys
   SET state = 'in_progress', lease_expires_at = CURRENT_TIMESTAMP + INTERVAL '30' SECOND
 WHERE scope = :scope AND "key" = :key AND request_hash = :hash
   AND state <> 'completed' AND lease_expires_at < CURRENT_TIMESTAMP;

-- 3. Business write and key completion commit together, or not at all.
START TRANSACTION;  -- PostgreSQL also accepts BEGIN
INSERT INTO ledger_entries (account_id, amount_cents, idempotency_key)
VALUES (:account_id, 4999, :key);
UPDATE idempotency_keys
   SET state = 'completed', response_status = 201, response_body = :response_json
 WHERE scope = :scope AND "key" = :key AND state = 'in_progress'
   AND lease_expires_at > CURRENT_TIMESTAMP;  -- 0 rows = lease lost: ROLLBACK instead
COMMIT;

-- 4. Scheduled cleanup; 7 days is a placeholder for your retention window.
DELETE FROM idempotency_keys WHERE created_at < CURRENT_TIMESTAMP - INTERVAL '7' DAY;

A portable SQL sketch of the idempotency key table: a unique (scope, key) index, an atomic claim insert, conditional takeover of expired leases, completion of the key in the same transaction as the ledger write, and a retention-based delete job.

Downstream calls, crashes mid-operation and asynchronous consumers

Your idempotency layer only protects work that stays inside your service. When the handler calls a payment processor, a ledger service or an inventory service, that call must also be safe to repeat. Otherwise a retry that your table correctly deduplicates can still charge a card twice. Propagate the client's key, or a deterministic key derived from it such as the key plus a step name, on every outbound call. If a third party offers no idempotency support, record your intent before calling it. Then resolve any ambiguous outcome with a status query, a reconciliation job against their reports, or manual review.

Keys stuck in progress

If a worker crashes after claiming a key, the row stays in progress forever unless you give the claim a lease that expires. When a new request takes over an expired lease, it should check downstream state first: did the charge actually go through? Only after that check should it re-execute. For multi-step operations, persist recovery points or an explicit state machine so the work resumes after the last completed step.

Choosing the lease length is a trade-off. If the lease is shorter than your slowest legitimate request, a live request and a retry can run concurrently. If it is much longer, clients sit blocked after a real crash.

Asynchronous consumers

Message brokers with at-least-once delivery create the same problem. Use the inbox pattern: insert the message ID into a processed-messages table in the same transaction as the side effect. A redelivered message then fails the unique constraint and is skipped.

In sharded or multi-region deployments, route every request for a given key to one authoritative store. Coordinating claims for the same key across replicas is much harder.

Copy-- PostgreSQL. Claim an idempotency key with a lease, and take over only if the lease expired.
CREATE TABLE idempotency_keys (
  key              text PRIMARY KEY,
  status           text NOT NULL,          -- 'in_progress' | 'completed'
  recovery_point   text NOT NULL,          -- last completed step, e.g. 'started', 'charge_created'
  lease_expires_at timestamptz NOT NULL
);

-- Lease (2 minutes here) must exceed the slowest legitimate request.
INSERT INTO idempotency_keys (key, status, recovery_point, lease_expires_at)
VALUES ('order-7f3a9c', 'in_progress', 'started', now() + interval '2 minutes')
ON CONFLICT (key) DO UPDATE
  SET lease_expires_at = now() + interval '2 minutes'
  WHERE idempotency_keys.status = 'in_progress'
    AND idempotency_keys.lease_expires_at < now()   -- only steal an expired lease
RETURNING recovery_point;
-- No row returned: another worker holds a live lease, or the key is completed.
-- Row returned with recovery_point past 'started': query the processor for
-- key 'order-7f3a9c:charge' before re-sending the charge.

-- Inbox pattern for at-least-once consumers.
CREATE TABLE processed_messages (
  message_id text PRIMARY KEY
);

BEGIN;
-- A redelivered message violates the primary key and the whole transaction rolls back,
-- so the side effect below is never applied twice.
INSERT INTO processed_messages (message_id) VALUES ('msg-01HZX4K2');
UPDATE ledger_balances SET amount_cents = amount_cents + 2500
 WHERE account_id = 'acct_123';
COMMIT;

PostgreSQL showing a lease-based key claim that takes over only expired leases and returns the recovery point to resume from, plus an inbox table that records the message ID in the same transaction as the side effect.

Alternatives and when to avoid a full idempotency key layer

The storage choice matters as much as the key itself. The primary relational database is the strongest option because the key row and the business write commit in one transaction. A key-value store with TTL is faster, but it is a separate failure domain. It also cannot commit atomically with the order insert, and memory pressure can evict a key before its TTL expires. A hybrid uses the key-value store as a fast-path cache in front of the database, with the database remaining authoritative.

Often a narrower mechanism is enough. If an order already carries a client-generated ID, a PUT to that ID gives you idempotency without a separate key layer. Reconciliation is still worth running alongside any of these approaches, but it should not be the only defence when money moves.

Skip the full layer for:

• Naturally idempotent operations, such as setting a status to a fixed value.
• Read-only endpoints.
• Internal calls that are never retried.
• Low-stakes writes where a duplicate does no harm, such as a debounced analytics event.

Impact on cost, latency and engineering time

Every mutating request now pays for at least one extra write: the claim row, then an update to store the response. A replayed or conflicting request also costs a read. These usually go to the same database as the business transaction, so they add round trips and contend for the same connection pool. Measure the latency on your own system rather than assuming it is negligible. The key table is also write-heavy and grows constantly, so it needs a cleanup job that deletes expired rows in small batches. Large batches can cause lock contention and bloat.

Retention is a direct trade-off. A longer window lets clients retry safely for longer but stores more rows.

Illustrative estimate (assumed numbers, not a benchmark)

• Storage ≈ mutating requests per day × retention days × average row size
• Assumed: 2,000,000 requests/day, 7-day retention, 1 KB per row (key, caller scope, fingerprint, status, stored response)
• 2,000,000 × 7 × 1 KB ≈ 14 million rows, roughly 14 GB before index and storage-engine overhead

Halving retention roughly halves the table. It also halves the window in which a delayed retry is still deduplicated.

The engineering time comes back elsewhere. You get fewer double-charge incidents, less manual refund and reconciliation work, and client teams can retry on timeouts without writing their own deduplication logic.

Document three things for clients:

• The retention window. A retry after it expires is treated as a new request.
• Exponential backoff with jitter. This stops retries from arriving in synchronized bursts.
• Conflict responses are retryable. A conflict means the original request is still in progress, not that it failed.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to idempotency keys microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.