Microservice Idempotency Keys: Deduplicating Retried Requests

What is an idempotency key and why do retries create duplicate requests?

An idempotency key is a client-generated identifier that names one logical operation, such as "charge order 8812 once," rather than one HTTP attempt. Every retry of that operation carries the same key, so the server can recognize repeats and return the original outcome instead of doing the work again.

Retries are unavoidable because a timeout tells the client nothing about what happened. The request may never have arrived. It may have committed just before the response was lost. It may still be running. The client cannot tell these cases apart, so the safe default is to retry.

Retries also stack. A mobile client retries, the API gateway retries, the service mesh sidecar retries, and the SDK inside the calling service retries. A message queue with at-least-once delivery redelivers anything that isn't acknowledged in time. One user tap can fan out into several attempts at the payment service, and no layer knows what the others did.

Operations that change state on each call need a dedup mechanism:

• POST endpoints that create resources
• Money transfers and charges
• Email, SMS, and push sends

PUT with a full replacement body and DELETE by ID are usually idempotent already. Read-only endpoints are unaffected. Keys are worth adopting if you expose non-idempotent writes or run money or messaging flows.

This guide focuses on where keys fail in practice. Most double-charges don't come from a missing key. They come from gaps between the key check, the business write, and the outbound call.

Who should use idempotency keys?

Use idempotency keys if any of the following describes your system:

• You expose non-idempotent writes over a network. Any endpoint where repeating the call creates a second resource, moves money again, or appends another entry is exposed to duplicate retries. Examples include POST /payments and POST /orders.
• You run payment, billing, ordering or provisioning flows. A duplicate here costs money or real resources, and fixing it means refunds, manual reconciliation, or tearing down infrastructure.
• You send messages people see. Duplicate emails, SMS and push notifications are visible to customers and cannot be recalled.
• You publish a public or partner API. You do not control your clients' retry logic, so the server has to be safe against whatever they do.
• You have retries at several layers. If the client, gateway, service mesh and SDK can each retry, assume duplicates will reach your write path.
• You consume at-least-once queues. Redelivery is part of the contract, so consumers need the processed-message variant of the same idea.

You can usually skip keys in these cases:

• Read-only services.
• Writes that are already idempotent, such as absolute sets, full-resource PUTs and DELETE by ID.
• Operations where the client supplies a natural unique identifier that a database constraint already enforces. The alternatives section below covers these cases.

How does it work, end to end?

At a high level, the mechanism has six steps. The sections that follow cover each one in detail.

1. The client mints a key once per logical operation. It generates a random value, typically a UUID, before the first attempt and stores it with the pending work. It sends the key, commonly in an Idempotency-Key header, on the first attempt and every retry.
2. The server scopes and claims the key atomically. It stores the key under a compound identity such as (tenant_id, key), along with a fingerprint of the request body. A unique constraint ensures that only one concurrent attempt wins the claim.
3. The winner does the work and records the outcome in the same transaction. The business write and the stored response commit together, or neither commits.
4. Repeats get the recorded outcome. A retry that finds a completed record gets the original status code and body. A retry that finds an in-progress record gets a conflict or retry-later response. A retry with the same key but a different fingerprint gets a client error.
5. Downstream calls get derived keys. Each service computes child keys deterministically from the parent key and a fixed step name. A retried parent therefore produces the same child operations, and each downstream service dedupes them in its own store.
6. External side effects go through an outbox. Emails, webhooks and published events are recorded in the same transaction as the business change and delivered by a relay. Receivers dedupe on the derived key.

Records expire after a retention window that covers the longest realistic retry horizon.

How does an idempotency key work on the server?

The client generates the key once per logical operation, typically a random UUID, and sends it on every retry of that operation. The key must be created before the first attempt and persisted alongside the pending work. A key regenerated inside the retry loop makes every attempt look new, which defeats the mechanism.

The server never treats the raw key as globally unique. It stores it under a compound identity such as (tenant_id, caller_id, key). Scoping prevents two clients that happen to pick the same string from reading each other's responses. It also stops an attacker who guesses a key from replaying someone else's result.

Each scoped key moves through three states:

The server should also store a fingerprint: a hash of the request's meaningful parameters, such as amount, currency and destination. Exclude volatile fields like timestamps or trace headers. If a known key arrives with a different fingerprint, reject it with a client error instead of replaying. Silently returning the old response would hide a client bug, such as a key reused across two different payments, and the caller would believe the second operation succeeded.

How do you implement atomic claim and commit for an idempotent API?

Start with a table such as idempotency_keys(scope, key, fingerprint, status, response_code, response_body, created_at, locked_until), with a unique constraint on (scope, key). The scope is usually the tenant or API client, so two customers can never collide on the same key string.

Claim and commit in one transaction

Begin a transaction and run INSERT ... ON CONFLICT (scope, key) DO NOTHING with status in_progress. If a row was inserted, you own the key. Do the business write, update the key row to completed with the serialized response, then commit. A crash before commit rolls back both writes, and a commit persists both. You never end up with work done but unrecorded, or with a key recorded for work that never happened.

If the insert hit a conflict, read the existing row. If the fingerprints differ, return a client error. If the status is completed, replay the stored response unchanged. If it is in_progress, return a conflict and tell the client to retry later.

Why not a cache set-if-absent

A conditional set in a cache gives you a fast claim, but the cache and the database commit separately. If the process crashes after the cache write, the key looks claimed even though no work happened. If the entry is evicted, a retry can run the work a second time.

Lock timeouts

Store locked_until so a crashed claim can be taken over later. If the timeout is too short, a slow request that is still running gets duplicated. If it is too long, a request that crashed blocks legitimate retries. Set it above your p99 handler latency.

CopyCREATE TABLE idempotency_keys (
  scope text NOT NULL, key text NOT NULL, fingerprint text NOT NULL,
  status text NOT NULL CHECK (status IN ('in_progress', 'completed')),
  response_code int, response_body jsonb,
  created_at timestamptz NOT NULL DEFAULT now(),
  locked_until timestamptz NOT NULL,
  CONSTRAINT idempotency_scope_key UNIQUE (scope, key)
);

BEGIN;
-- Claim. A concurrent INSERT of the same key waits on the unique index until we
-- commit or roll back. The DO UPDATE branch only matters if an in_progress row
-- was committed earlier (e.g. a split flow around an external call) and its lock
-- expired: then a retry with the SAME fingerprint may take it over.
INSERT INTO idempotency_keys (scope, key, fingerprint, status, locked_until)
VALUES ($1, $2, $3, 'in_progress', now() + interval '30 seconds') -- > p99 latency
ON CONFLICT (scope, key) DO UPDATE SET locked_until = EXCLUDED.locked_until
  WHERE idempotency_keys.status = 'in_progress'
    AND idempotency_keys.locked_until < now()
    AND idempotency_keys.fingerprint = EXCLUDED.fingerprint
RETURNING status;

-- 1 row returned: we own the key. Business write and key completion share the tx.
INSERT INTO payments (tenant_id, amount_cents, currency) VALUES ($1, 2500, 'EUR');
UPDATE idempotency_keys
   SET status = 'completed', response_code = 201,
       response_body = '{"payment_id": "pay_123"}', locked_until = now()
 WHERE scope = $1 AND key = $2;
COMMIT;  -- crash before this line rolls back both writes

-- 0 rows returned: ROLLBACK, then inspect the existing row.
SELECT fingerprint, status, response_code, response_body
  FROM idempotency_keys WHERE scope = $1 AND key = $2;
-- pseudocode:
--   if row.fingerprint <> $3        -> 422, key reused with a different body
--   elif row.status = 'completed'   -> replay row.response_code / response_body as-is
--   else ('in_progress', lock live) -> 409 Conflict with Retry-After

PostgreSQL sketch of claiming an idempotency key and doing the business write in one transaction, with the replay, fingerprint-mismatch, in-progress and expired-lock takeover paths.

How do you propagate idempotency keys across downstream services?

A key that stops at the order service only protects the first hop. If the order service calls payments with a fresh random key on every attempt, a retried checkout still charges the card twice. The fix is to derive each downstream key deterministically from the parent key and a stable step identifier, for example hash(parent_key + ":" + step_name). The same parent key always yields the same child keys. A retry of the parent therefore replays the same child operations, and each service can recognise them.

Each service enforces its own dedup using the claim-and-commit pattern on its own store. The order service does not need to know whether payments already ran. It sends the same derived key, and payments either executes or returns its stored result. This keeps ownership local: no service reaches into another's idempotency table, and a partial failure midway through the flow resumes cleanly on retry.

The weak point is the step identifier. Treat step names as a stable contract, not as labels you can refactor freely. If you rename charge-card to capture-payment, or derive keys from a step's position in a list and then reorder the steps, an in-flight request retried after the deploy produces different child keys. The payment service sees what looks like a new request and executes it again. Use explicit, versioned constants for step names. Keep old names working until every key issued under them has expired.

Copyimport hashlib

STEP_CHARGE_CARD = "charge-card.v1"
STEP_SEND_CONFIRMATION = "send-confirmation.v1"
STEP_ORDER_CREATED_EVENT = "order-created-event.v1"


def child_key(parent_key: str, step: str) -> str:
    # Deterministic: same parent + same step always yields the same child key.
    # Derive from the step name, never from the step's position in a list,
    # so reordering steps cannot shift keys onto different operations.
    return hashlib.sha256(f"{parent_key}:{step}".encode("utf-8")).hexdigest()


def plan_downstream_calls(parent_key: str) -> dict:
    # The order service sends these keys on every attempt, including retries.
    # Each callee runs its own claim-and-commit dedup on its own store.
    return {
        "payment": {"Idempotency-Key": child_key(parent_key, STEP_CHARGE_CARD)},
        "notification": {"Idempotency-Key": child_key(parent_key, STEP_SEND_CONFIRMATION)},
        "outbox_row_id": child_key(parent_key, STEP_ORDER_CREATED_EVENT),
    }


if __name__ == "__main__":
    k = "3f6c1a2e-8b4d-4e7a-9c11-5d2f0b7e6a90"  # client-generated key K
    first = plan_downstream_calls(k)
    retry = plan_downstream_calls(k)
    # A retry of the parent replays the same child operations.
    assert first == retry
    for target, value in first.items():
        print(target, value)

Derives deterministic child idempotency keys from the parent key and fixed, versioned step identifiers, so a retried checkout sends identical keys to payments, notifications and the outbox.

Handling side effects outside your database: outbox and idempotent consumers

An idempotency key protects state you control. It cannot recall an email, a webhook, or a message already published to a broker. Suppose a handler sends a confirmation email, then crashes before committing its idempotency record. The retry finds no record, runs again, and the customer gets two emails. The key worked as designed. The side effect simply escaped before the commit that would have recorded it.

The transactional outbox closes that gap. Do not call the external system inline. Instead, insert an outbox row in the same database transaction as the business change. The row holds the intended action, its payload, and a key derived from the request's idempotency key, such as the parent key plus a step name. Either the order and its pending email both commit, or neither does. A separate relay reads unsent rows, delivers them, and marks them sent. A relay can crash after delivering a row but before marking it, so delivery is at least once. The derived key travels with each message so the receiver can discard repeats.

Consumers of at-least-once queues need the same discipline. Broker-level "exactly-once" guarantees usually cover delivery within the broker, not your database writes. Dedupe in the consumer instead:

1. Begin a transaction.
2. Insert the message or operation ID into a processed_messages table that has a unique constraint on that ID.
3. If the insert conflicts, the message is a duplicate. Acknowledge it and stop.
4. Otherwise, apply the business change, commit, and then acknowledge.

Because the dedupe record and the effect commit together, a crash at any point leaves both in place or neither.

Copyimport json
import psycopg2

conn = psycopg2.connect("dbname=orders user=app password=example host=localhost")

def handle(message):
    """Apply a queue message once, even if the broker redelivers it.

    `message` stands in for your queue client's message object:
    it carries a stable id (e.g. the outbox-derived key), a body, and ack().
    """
    payload = json.loads(message.body)

    # psycopg2: `with conn` commits on success and rolls back on exception,
    # so the dedupe record and the business change land together or not at all.
    with conn:
        with conn.cursor() as cur:
            cur.execute(
                "INSERT INTO processed_messages (message_id) VALUES (%s) "
                "ON CONFLICT (message_id) DO NOTHING",
                (message.id,),
            )
            if cur.rowcount == 1:
                # First time we've seen this ID: apply the effect in the same transaction.
                cur.execute(
                    "UPDATE orders SET status = 'paid' WHERE id = %s",
                    (payload["order_id"],),
                )
            # rowcount == 0 means a duplicate: commit nothing new, just ack below.

    # Ack only after commit. If we crash before this line the broker redelivers,
    # and the unique key turns the retry into a no-op.
    message.ack()

An idempotent queue consumer that inserts the message ID into a uniquely keyed processed_messages table in the same transaction as the business update, acknowledging only after commit so redeliveries are discarded.

When should you avoid idempotency keys, and what are the alternatives?

A key table adds a write and a lookup to every request, so skip it where the operation already tolerates repetition. Reads need nothing: repeating a GET changes no state. Absolute writes such as "set status to shipped" or a full-resource PUT land in the same final state no matter how often they run. When the client can name the resource itself, for example a client-generated ID in PUT /orders/{id} or a merchant order number under a unique constraint, the database rejects the second insert. That constraint does the deduplication and needs no expiry. Keys earn their cost on relative or creating operations with no natural identity, such as "charge $20" or "append an entry."

What do idempotency keys cost, and how do you test and operate them?

Every keyed request adds a write to the dedup store, couples its schema to request and response formats, and adds a round trip. The store also becomes a new dependency on the critical path. You have to decide in advance what happens when that store is down:

Retention. Keep records for at least the longest realistic retry horizon, including client backoff and queue redelivery. A scheduled job or a TTL should remove expired rows. Cache successes and deterministic 4xx errors, because retrying those will produce the same result. Do not cache 5xx or timeout errors, so the client can still retry. A replay returns the original outcome, which may now be stale. Your API docs should say so.

Illustrative estimate (assumed numbers, not a benchmark). Rows = keyed requests/s × retention seconds. Storage = rows × bytes per row. Assume 50 req/s, a 24 h retention window, and 1 KB per row including the stored response. That gives 50 × 86,400 = 4.32M rows, or about 4.3 GB before index overhead.

These costs pay for themselves in engineering time. You get fewer duplicate-charge incidents and less manual reconciliation, and retry policies can be more aggressive without risk. The next section breaks this down.

Test plan. Run each of these cases:

• Kill the process after the commit but before the response is sent, then retry.
• Fire concurrent duplicate requests.
• Replay a key after it has expired.
• Reuse a key with a mutated payload. Expect a conflict.

In every case, assert that exactly one side effect occurred and that the responses are consistent.

Copy-- PostgreSQL. One row per (client, key); stores enough to replay the original outcome.
CREATE TABLE idempotency_keys (
  client_id     text        NOT NULL,
  idem_key      text        NOT NULL,
  request_hash  bytea       NOT NULL,  -- detects key reuse with a mutated payload
  status        text        NOT NULL CHECK (status IN ('in_progress', 'completed')),
  response_code int,
  response_body jsonb,
  expires_at    timestamptz NOT NULL DEFAULT now() + interval '24 hours',  -- longest retry horizon
  PRIMARY KEY (client_id, idem_key)
);
CREATE INDEX idempotency_keys_expires_idx ON idempotency_keys (expires_at);

-- Claim the key. If an expired row has not been cleaned up yet, take it over.
-- Zero rows returned means a live duplicate: compare request_hash (409 on mismatch), then replay or wait.
INSERT INTO idempotency_keys (client_id, idem_key, request_hash, status)
VALUES ($1, $2, $3, 'in_progress')
ON CONFLICT (client_id, idem_key) DO UPDATE
  SET request_hash = EXCLUDED.request_hash, status = 'in_progress',
      response_code = NULL, response_body = NULL,
      expires_at = now() + interval '24 hours'
  WHERE idempotency_keys.expires_at < now()
RETURNING client_id;

-- Success or deterministic 4xx: record the outcome in the SAME transaction as the side effect.
UPDATE idempotency_keys
SET status = 'completed', response_code = $3, response_body = $4
WHERE client_id = $1 AND idem_key = $2;

-- 5xx or timeout: release the claim so the client's retry can run.
DELETE FROM idempotency_keys
WHERE client_id = $1 AND idem_key = $2 AND status = 'in_progress';

-- Scheduled cleanup in small batches to keep lock times short.
DELETE FROM idempotency_keys
WHERE ctid IN (SELECT ctid FROM idempotency_keys WHERE expires_at < now() LIMIT 10000);

A PostgreSQL key table that claims keys atomically, takes over expired rows, caches only successes and deterministic 4xx outcomes, releases keys on 5xx or timeout, and purges expired rows in batches.

How can idempotency keys save engineering time?

The time savings come from removing recurring work that duplicates otherwise create.

Fewer duplicate-side-effect incidents to investigate. Without dedup, a double charge or duplicate order means reconstructing what happened from logs across several services, deciding which record is the real one, and then refunding, cancelling or deleting the other. With claim-and-commit in place, the retry that would have caused the incident becomes a replay of a stored response. The investigation never starts.

Less reconciliation code and manual cleanup. Teams without idempotency often build after-the-fact jobs that scan for near-identical records within a time window and flag them for review. Those jobs rely on heuristics, need tuning, and still miss cases. A unique constraint on (scope, key) prevents the duplicate at write time, so there is nothing to reconcile.

Retry policies you can turn on without a per-endpoint safety review. When writes are not idempotent, every retry setting in the client, gateway, mesh or SDK is a risk that someone has to reason about endpoint by endpoint. Often the result is that retries get disabled for writes, and transient failures surface as user-facing errors and support tickets. Once write endpoints honour keys, retries can be configured consistently across layers, and transient network failures recover on their own.

One pattern instead of many ad hoc fixes. The same key table, derived-key function, outbox and processed_messages consumer apply to every service. A new endpoint or consumer adopts an existing, tested pattern. Engineers do not have to invent a fresh dedupe scheme under incident pressure. The test plan above also becomes a reusable suite rather than something rewritten per service.

Faster debugging and support answers. The client key, and the child keys derived from it, identify one logical operation across every hop. Given a customer's request, an engineer can look up the key in each service's table and see whether each step is completed, in progress or absent. That answers "was I charged twice?" or "where did this order stall?" directly, instead of correlating timestamps across logs.

Safer deploys and restarts. Because a crashed or interrupted request can be retried safely, rolling restarts and failovers that cut in-flight requests no longer need special handling or after-the-fact cleanup for those requests.

The trade-off is upfront work: the key table, retention jobs, step-name discipline, and the fail-open or fail-closed decision all have to be built and maintained. That investment pays off most clearly on the flows listed under "Who should use idempotency keys?", where each duplicate otherwise turns into manual work.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice idempotency keys, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.