Exponential Backoff with Jitter: Safe Retries Between Services

What retries are for and when they make things worse

A retry exists to absorb transient failures: faults that clear up on their own within milliseconds or seconds. Typical cases are a TCP connection reset because a pod was rescheduled mid-request, a load balancer briefly returning 503 while instances drain, a downstream service shedding load for a few seconds, or a replicated datastore rejecting writes while it elects a new leader. In each case the second attempt usually succeeds because the condition that caused the first failure no longer holds.

The retry policy for service-to-service calls belongs to the backend and platform engineers who run those calls. The application team knows which operations are safe to repeat and how long a caller can wait. The platform team controls the shared HTTP and RPC clients, sidecars and gateways where retries are often switched on by default. When neither group makes a deliberate decision, retries end up configured at several layers independently.

Retries help only if the failure goes away. Against a persistent failure, such as a database that is down, an expired credential, or a dependency that is saturated, every retry is one more request that fails. As an illustrative estimate, a client allowed three retries can send up to four times its normal request rate to a dependency that is already failing. That extra load arrives exactly when the dependency has the least spare capacity.

This article's thesis follows from that. Exponential backoff with jitter spreads retries out over time. It does not limit how many retries happen. Capping the total requires separate controls, which later sections cover.

How retry storms and the thundering herd happen

Amplification starts with a reasonable decision made independently at every hop. Suppose an edge gateway calls an API service, which calls an inventory service, which calls a database. If each of the three calling layers makes up to N attempts and does not know about the others' policies, one user request can become up to N^K calls at the deepest service, where K is the number of retrying layers. The multiplication only happens when the failure is deep in the chain, so it shows up at exactly the moment the deepest service is struggling.

Illustrative estimate: with 3 attempts per layer at 3 layers, a single gateway request can produce up to 3 × 3 × 3 = 27 calls to the database. If the database slows down under normal load, every in-flight request is multiplied in the same way. The extra load keeps it degraded, which triggers more retries. This is a retry storm: the retries become the outage.

Capped exponential backoff on its own does not prevent this. It changes when retries arrive, but it does not spread them out. Clients that failed together, such as all the callers whose requests hit the same timeout during a brief stall, start their backoff schedules at the same moment. With identical base delays and multipliers, they all wait 100 ms, then 200 ms, then 400 ms, and they retry in lockstep. The recovering service sees quiet gaps followed by synchronized spikes, a thundering herd that repeats on every backoff step. Randomizing those schedules is what breaks the synchronization.

Copy  1 user request
        |
        v
+----------------+
|  Edge gateway  |
+----------------+
        |  up to 3 attempts  ->  3 calls reach API
        v
+----------------+
|  API service   |
+----------------+
        |  up to 3 each      ->  9 calls reach Inventory
        v
+----------------+
|   Inventory    |
+----------------+
        |  up to 3 each      ->  27 calls reach Database
        v
+----------------+
|    Database    |   3^3 = 27 calls (illustrative estimate)
+----------------+

 Capped backoff without jitter: clients that failed together
 time ms   0         100       300                 700
 client A  X-------->R-------->R------------------>R
 client B  X-------->R-------->R------------------>R
 client C  X-------->R-------->R------------------>R
 load      ^         ^         ^                   ^  synchronized spikes

 X = shared failure   R = retry (waits 100, 200, 400 ms)

With three retrying layers making up to three attempts each, one request can fan out to 27 database calls (illustrative estimate), and clients that failed together retry in lockstep on identical backoff schedules.

How exponential backoff with jitter works: full, equal and decorrelated

breaks the lockstep. Each scheme starts from the same exponential ceiling, ceil = min(cap, base × 2^attempt), and differs in how it draws the actual wait from it.

Full jitter is a reasonable default for three reasons. It uses the whole window, so the recovering service sees the flattest arrival rate. It needs no state beyond the attempt count. And its behaviour is easy to reason about.

The cost falls on individual requests. Some retries fire almost immediately, while others wait nearly the full ceiling. That widens tail latency even when the median looks fine. Equal jitter narrows this spread by trading away some load smoothing.

The cap sets the overall trade-off. A long cap spreads retries over more time, which lowers pressure on a struggling dependency. It also means a client may sleep through a recovery that already happened. A short cap recovers faster but can concentrate retries back into bursts. In either case the cap has to fit inside the caller's deadline. If a single backoff can exceed the remaining time budget, that retry is wasted: its result arrives after the caller has given up.

Copyimport random
import time

BASE = 0.1   # seconds; first ceiling
CAP = 5.0    # seconds; upper bound on any single wait

def ceiling(attempt):
    # Shared exponential ceiling: min(cap, base * 2^attempt)
    return min(CAP, BASE * 2 ** attempt)

def full_jitter(attempt):
    # Anywhere in [0, ceil]: flattest load, widest per-request spread
    return random.uniform(0, ceiling(attempt))

def equal_jitter(attempt):
    # At least ceil/2: guaranteed backoff, clients clustered in upper half
    c = ceiling(attempt)
    return c / 2 + random.uniform(0, c / 2)

def decorrelated_jitter(prev):
    # Grows from the previous wait, not the attempt count, so it needs state
    return min(CAP, random.uniform(BASE, prev * 3))

def call_with_retries(fn, deadline_s=3.0, max_attempts=6):
    start = time.monotonic()
    for attempt in range(max_attempts):
        try:
            return fn()
        except ConnectionError:
            if attempt == max_attempts - 1:
                raise
            wait = full_jitter(attempt)
            remaining = deadline_s - (time.monotonic() - start)
            # Don't sleep past the caller's deadline; that retry would be wasted
            if wait >= remaining:
                raise
            time.sleep(wait)

Python implementations of full, equal and decorrelated jitter, with a retry loop that uses full jitter and gives up rather than sleeping past the caller's deadline.

Implementing a safe retry policy: budgets, one retry layer and deadlines

Bound retry volume with a budget

Backoff spreads retries out over time. It does not cap how many there are. A retry budget does. One form is a ratio: allow retries only while they stay below a fixed fraction of successful requests, such as 10%. The other is a per-client token bucket. Each success adds a fraction of a token, each retry spends one, and an empty bucket means the call fails immediately. During a wide outage, success stops refilling the bucket, so retry traffic falls instead of tripling load on a struggling dependency.

Retry in one layer only

If the gateway, service, and client library each make three attempts, one user request can turn into 27 calls to the backend. Pick a single layer close to the original caller to own retries, because it knows the deadline and the user impact. Every inner layer should fail fast and pass the error up.

Fit attempts inside the deadline

Send the remaining deadline downstream so that no service keeps working after the caller has stopped waiting. The constraint is:

attempts × per_attempt_timeout + Σ backoff ≤ overall_deadline

Illustrative estimate (assumed numbers, not a benchmark): deadline 2,000 ms, timeout 400 ms, 3 attempts, maximum backoffs 100 ms and 200 ms. That gives 3 × 400 + 300 = 1,500 ms, which fits. Raising the timeout to 700 ms gives 2,400 ms, which does not.

A circuit breaker sits above all of this. When it is open, no attempt is made at all.

The sketch below combines jittered backoff, a token-bucket budget, and a check on the remaining deadline before each attempt.

Copyimport random
import time

class RetryBudget:
    """Per-client token bucket: successes refill slowly, each retry spends a whole token."""
    def __init__(self, max_tokens=10.0, refill_per_success=0.1):
        self.tokens = max_tokens
        self.max_tokens = max_tokens
        self.refill = refill_per_success

    def on_success(self):
        self.tokens = min(self.max_tokens, self.tokens + self.refill)

    def try_spend(self):
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True
        return False  # empty bucket: fail fast instead of adding load

def call_with_retries(call, budget, deadline_s=2.0, attempt_timeout_s=0.4,
                      max_attempts=3, base_s=0.1, cap_s=0.2):
    # Only this outermost layer retries. call() must fail fast and must not retry itself.
    deadline = time.monotonic() + deadline_s
    for attempt in range(max_attempts):
        if attempt > 0:
            # Full jitter: a random delay up to the capped exponential backoff.
            delay = random.uniform(0, min(cap_s, base_s * 2 ** (attempt - 1)))
            # Give up if the backoff plus a full attempt would overrun the deadline.
            if time.monotonic() + delay + attempt_timeout_s > deadline:
                raise TimeoutError("not enough deadline left for another attempt")
            if not budget.try_spend():
                raise RuntimeError("retry budget exhausted")
            time.sleep(delay)
        remaining = deadline - time.monotonic()
        try:
            # Propagate the smaller of the per-attempt timeout and the remaining deadline.
            result = call(timeout=min(attempt_timeout_s, remaining))
            budget.on_success()
            return result
        except (TimeoutError, ConnectionError):  # retry only transient failures
            if attempt == max_attempts - 1:
                raise

A single retry layer that combines full-jitter exponential backoff, a per-client token-bucket retry budget, and a remaining-deadline check before each retry, using the section's illustrative values of a 2 s deadline, 400 ms attempts and backoff capped at 200 ms.

Deciding what to retry and making writes idempotent

Not every failure deserves a second attempt. A retry helps only when the next attempt has a real chance of succeeding and repeating the request is safe.

When a response carries Retry-After (either seconds or an HTTP date), treat it as a floor for your next delay rather than an input to your backoff formula. The server knows its own recovery time better than your jitter does.

Timeouts and idempotency keys

A timeout tells you only that you stopped waiting. The server may have committed the write, so a blind retry of a payment or order creation can apply it twice. The fix is an idempotency key. The client generates a unique key per logical operation and reuses it on every retry. The server records the key with the result and returns the stored result for repeats.

This costs the server something. It needs durable storage for each key and its response. It needs an expiry window longer than the client's total retry horizon. It also needs a rule for a duplicate that arrives while the original is still executing. The usual approach is to lock the key and return a conflict or "in progress" response, so the operation is never run twice concurrently.

Copy-- PostgreSQL: durable record of each idempotency key and the response it produced
CREATE TABLE idempotency_keys (
  key            text PRIMARY KEY,           -- client-generated, reused on every retry
  request_hash   text NOT NULL,              -- detects the same key sent with a different body
  status         text NOT NULL CHECK (status IN ('in_progress', 'completed')),
  response_code  int,
  response_body  jsonb,
  created_at     timestamptz NOT NULL DEFAULT now()
);

-- 1. Claim the key atomically. If a row comes back, this request owns the operation.
INSERT INTO idempotency_keys (key, request_hash, status)
VALUES ($1, $2, 'in_progress')
ON CONFLICT (key) DO NOTHING
RETURNING key;

-- 2. If no row came back, the key already exists. Inspect it:
--    hash differs  -> 422, the client reused a key for a different request
--    'completed'   -> replay the stored response, do not execute again
--    'in_progress' -> 409, the original is still running; never run it concurrently
SELECT request_hash, status, response_code, response_body
FROM idempotency_keys
WHERE key = $1;

-- 3. After the write commits, store the result so later retries get the same answer.
UPDATE idempotency_keys
SET status = 'completed', response_code = $2, response_body = $3
WHERE key = $1;

-- 3b. If the operation failed before changing anything, release the key so a retry can proceed.
DELETE FROM idempotency_keys WHERE key = $1 AND status = 'in_progress';

-- 4. Expiry job. The window must exceed the client's total retry horizon.
--    A shorter window lets a late retry run the write a second time.
DELETE FROM idempotency_keys
WHERE created_at < now() - interval '24 hours';

A PostgreSQL idempotency-key table and the server-side statements that claim a key, replay completed results, reject duplicates that arrive while the original is still in progress, and expire old keys.

When to avoid retries and what the alternatives are

Some failures should not be retried at all. A persistent error, such as a 400, an authorization failure, or a schema mismatch, returns the same result on every attempt, so retrying only adds latency and load. A non-idempotent write without an idempotency key can be applied twice. In a deep call chain, only one layer should retry, and that is usually the outermost one. If every hop retries three times, five hops can produce up to 3⁵ = 243 attempts at the bottom for a single user request (illustrative estimate). Under a tight deadline, a backoff delay can use up the remaining time, so the retry arrives after the caller has already given up.

Hedging targets tail latency, not errors. It is safe only when the extra requests are capped, for example to a small fraction of traffic, and when the operation is idempotent.

Client-side policy also has a limit: it only works if every client follows it. Old SDK versions, scripts, and buggy clients will ignore your backoff rules. Server-side load shedding and admission control are the only protections that apply to all of them.

Observability, testing, and the impact on cost and engineering time

What to measure

Record three things at every client that retries:

• Attempts per logical call, as a histogram rather than an average, so a small fraction of calls burning through every attempt stays visible.
• Retry ratio, defined as retries divided by first attempts. This is your live amplification factor minus one.
• First-attempt errors and final errors as separate series. If you only count final errors, retries hide a degrading dependency until the budget runs out and failures arrive all at once.

Alert on a sustained rise in retry ratio, even while final error rates look healthy. A climbing ratio means downstream capacity is already being spent on repeat work. That is the earliest signal you get before amplification turns a slow dependency into a saturated one.

Testing for amplification

Local and unit tests rarely expose retry storms, because a storm needs concurrency, shared downstream capacity, and several retrying layers at once. Use fault injection (added latency, error responses, dropped connections) in staging under realistic load. Then confirm that the retry ratio stays within budget and that load on the dependency stops climbing once it degrades.

Cost versus time saved

The upfront cost is real. Teams must agree on which layer retries, idempotency keys need storage, and product owners must accept that some requests will fail fast when the budget is exhausted. The return comes from two places. First, a partial outage that stays partial avoids a much longer, multi-team incident. Second, one shared client library with a sane default policy replaces dozens of hand-rolled retry loops that each need separate review and debugging.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to exponential backoff with jitter, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.