What is a retry storm in microservices, and why does it compound with call depth?

A retry storm is a property of the whole call graph, not a bug in any one client. Each service's retry policy can be reasonable on its own. The trouble is that retries multiply across hops. When a downstream dependency slows down, every layer above it times out and retries at about the same moment. Load on the struggling service then grows exactly when it can least absorb it, and the extra load keeps it from recovering.

The growth is multiplicative. If each of N layers makes one attempt plus R retries, the leaf can receive up to (R+1)^N calls for a single user request.

Illustrative estimate. Assume 3 synchronous layers, each making 1 attempt plus 2 retries, and a leaf that fails every call:

user → A ×3 → B ×3 each → C ×3 each = 27 calls to C

Depth (N) Attempts per hop (R+1) Worst-case leaf calls per user request
1 3 3
2 3 9
3 3 27
4 3 81

The real multiplier is often higher than any configuration file suggests:

  • Hedged requests. Sending a second copy of a request before the first has failed raises the effective attempt count per hop, even when no retry is configured.
  • Stacked retry layers. A service mesh sidecar and an application client library can each retry the same call. That multiplies attempts within a single hop.

This matters most to backend and platform engineers who run multi-hop synchronous call graphs. Asynchronous, queue-based designs fail differently.

                         +-----------+
                         | User/Edge |  1 attempt + 2 retries
                         +-----------+
                               |
         +---------------------+---------------------+
         v                     v                     v
     +-------+             +-------+             +-------+
     |   A   |             |   A   |             |   A   |   3 calls
     +-------+             +-------+             +-------+
         |                     |                     |
   +-----+-----+         +-----+-----+         +-----+-----+
   v     v     v         v     v     v         v     v     v
 +---+ +---+ +---+     +---+ +---+ +---+     +---+ +---+ +---+
 | B | | B | | B |     | B | | B | | B |     | B | | B | | B |   9 calls
 +---+ +---+ +---+     +---+ +---+ +---+     +---+ +---+ +---+
  |||   |||   |||       |||   |||   |||       |||   |||   |||
  vvv   vvv   vvv       vvv   vvv   vvv       vvv   vvv   vvv
 +-----------------------------------------------------------+
 |  C (leaf, failing every call): 9 x 3 = 27 calls           |
 +-----------------------------------------------------------+

 Hidden multiplier inside ONE hop (stacked retry layers):
 +--------------+    +--------------+    +------------------+
 | client lib   |--->| mesh sidecar |--->| downstream sees  |
 | 1 + 2 tries  |    | 1 + 2 each   |    | up to 3 x 3 = 9  |
 +--------------+    +--------------+    +------------------+
 Hedged requests add attempts per hop even with no retry configured.

Illustrative estimate with assumed inputs: three retrying layers, each making 1 attempt plus 2 retries against a leaf that always fails, deliver up to (2+1)^3 = 27 calls to the leaf for one user request, and stacked client-library and sidecar retries or hedging raise the per-hop multiplier further.

How does deadline propagation fix independently set timeouts?

Timeouts chosen hop by hop rarely agree. A typical failure looks like this. The gateway gives up at 2 s, but service B calls C with a 5 s timeout and two retries. C keeps working, and B keeps retrying, long after anyone is waiting for the result.

Illustrative estimate (assumed numbers, not a benchmark): orphaned work ≈ (attempts × inner timeout) − outer timeout = (3 × 5 s) − 2 s = 13 s of downstream work per abandoned request.

Deadline propagation replaces these independent timeouts with one shared budget that travels with the request:

  • On entry, each hop computes remaining = deadline − now. If remaining ≤ 0, it rejects the request immediately.
  • Before calling downstream, it sets the call's timeout to remaining − reserve, where the reserve is the time the hop needs to handle the response. It forwards that value with the call.
  • Retries draw from the same remaining budget. A retry that cannot finish in time is never sent.
Encoding Strength Weakness
Absolute deadline (timestamp) Queueing and serialization time are counted automatically Clock skew between hosts shrinks or stretches the budget
Relative budget (duration) Needs no clock agreement Time on the wire, in buffers and in queues goes unmeasured, so the budget drifts too generous at each hop

The mechanism only works if every library, proxy and queue consumer reads the deadline and forwards it. If one hop drops it, everything below that hop falls back to its default timeout.

Async paths are harder. A message can sit in a queue past its deadline, so consumers must check the deadline before doing any work. Fan-out adds its own pitfall. Parallel calls share the parent's deadline, while sequential calls must each subtract the time already spent.

import time

RESERVE_S = 0.050      # time this hop needs to handle the downstream response
MIN_ATTEMPT_S = 0.100  # an attempt shorter than this cannot realistically succeed
BUDGET_HEADER = "x-deadline-budget-ms"  # placeholder name; use your stack's convention

class DeadlineExceeded(Exception):
    pass

def handle(headers, call_downstream, max_attempts=3):
    # Relative budget from the caller. Wire, buffer and queue time before this
    # line goes unmeasured, so the real budget is already somewhat smaller.
    budget_s = int(headers[BUDGET_HEADER]) / 1000
    if budget_s <= 0:
        raise DeadlineExceeded("expired on arrival")  # reject before doing any work

    # Pin the budget to this host's monotonic clock: immune to wall-clock skew.
    deadline = time.monotonic() + budget_s

    for attempt in range(max_attempts):
        # Every attempt, including retries, draws from the same shrinking budget.
        remaining = deadline - time.monotonic()
        timeout = remaining - RESERVE_S
        if timeout < MIN_ATTEMPT_S:
            # Not enough time to finish another attempt: don't send it.
            raise DeadlineExceeded(f"no budget left after {attempt} attempts")

        try:
            # Downstream timeout and forwarded budget are the same value, so the
            # callee stops when we stop waiting.
            return call_downstream(
                timeout=timeout,
                headers={BUDGET_HEADER: str(int(timeout * 1000))},
            )
        except TimeoutError:
            continue  # loop recomputes remaining; this retry costs budget

    raise DeadlineExceeded("attempts exhausted")

A hop reads the incoming budget, rejects work that has already expired, derives each downstream timeout from what is left minus a response-handling reserve, forwards that value, and makes retries draw from the same budget.

How do retry budgets work compared with fixed per-request retry counts?

A fixed per-request retry count ("retry up to 3 times") limits how many times one request is retried. It does not limit how much extra traffic the client produces overall. When a dependency fails, every request uses its full allowance at once, and the extra load reaches the dependency exactly when it is least able to absorb it.

A retry budget sets the cap per client instead of per request. Over a sliding window, retries may not exceed a fixed fraction of original requests, for example 10%. A small minimum allowance, such as a few retries per second, lets low-traffic clients retry an occasional transient failure, which a pure ratio would forbid.

Illustrative estimate (assumed numbers, not a benchmark). Assume original traffic R = 1,000 req/s and a dependency that fails every call.

  • Per-request count N = 3: attempts = R × (1 + N) = 4,000/s, which is 4× load.
  • Budget ratio b = 0.1, minimum m = 10/s: attempts = R + max(b × R, m) = 1,100/s, which is 1.1× load.
Property Fixed per-request count Retry budget
Worst-case load during an outage R × (1 + N) R + max(bR, m)
State None Shared counter per client
Low-traffic behaviour Always retries Retries only within the minimum

The usual implementation is a token bucket. Each original request deposits b tokens, each retry withdraws one, the bucket has a capacity limit, and a minimum refill rate covers quiet clients. Check the bucket after you have classified the failure as retryable and before you sleep for backoff. If the bucket is empty, return the original error.

Values copied from library defaults or from another service can be too loose or too strict for your traffic. Validate them with load tests that inject downstream latency as well as errors, because slow responses that time out drive retries differently from fast failures.

import random, threading, time

class RetryBudget:
    """Shared per client. Allows about ratio*R + min_per_sec retries/s at most."""
    def __init__(self, ratio=0.1, min_per_sec=10.0, capacity=100.0):
        self.ratio, self.min_per_sec, self.capacity = ratio, min_per_sec, capacity
        self.tokens, self.last = capacity, time.monotonic()
        self.lock = threading.Lock()

    def _refill(self):  # caller holds lock; minimum allowance for quiet clients
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.min_per_sec)
        self.last = now

    def on_request(self):  # each original request deposits `ratio` tokens
        with self.lock:
            self._refill()
            self.tokens = min(self.capacity, self.tokens + self.ratio)

    def try_spend(self):  # each retry withdraws one token, or is refused
        with self.lock:
            self._refill()
            if self.tokens >= 1.0:
                self.tokens -= 1.0
                return True
            return False

def call_with_retries(send, budget, is_retryable, max_attempts=3):
    budget.on_request()  # count the original request once, not per attempt
    for attempt in range(max_attempts):
        try:
            return send()
        except Exception as err:
            if attempt == max_attempts - 1 or not is_retryable(err):
                raise
            if not budget.try_spend():  # check after classifying, before backoff
                raise  # budget empty: surface the original error
            time.sleep(random.uniform(0, 0.1 * 2 ** attempt))  # jittered backoff

A thread-safe token-bucket retry budget, where original requests deposit a fraction of a token and a minimum refill rate covers quiet clients, checked after a failure is classified as retryable and before the backoff sleep.

Where should retries happen, and which failures are safe to retry?

Choose one layer to own retries, usually the edge proxy or the service closest to the user, and make every inner layer fail fast. This has a cost. A transient blip four hops down now re-runs the entire call chain instead of one cheap local hop. The trade is still worth it, because a single retry point caps amplification at that layer's budget, while retries at several layers multiply.

Inner services must tell the retrying layer when to back off. Use distinct signals for overload and for transient faults. For example, return HTTP 503 with Retry-After or gRPC RESOURCE_EXHAUSTED for overload, and a different code or header for a one-off fault. Pass overload signals upstream unchanged instead of collapsing them into a generic 500. That way the edge stops retrying at the moment retries would do the most damage.

Failure Safe to retry? Condition
Connection refused before request was sent Yes Request never reached the server
Transient error on idempotent read or PUT/DELETE Yes Within budget and remaining deadline
Explicit overload or throttling signal No Wait for the server-specified delay, if any
Timeout on non-idempotent write Only with dedup Server stores the idempotency key and its result
Validation or auth error (4xx) No Retrying cannot change the outcome

A timeout on a non-idempotent write is ambiguous because the server may have committed the change before the response was lost. Retrying blindly can charge a card twice. An idempotency key only protects you if the server persists it alongside the outcome and returns the stored result when the key is repeated.

Finally, inventory every component that can retry: mesh sidecars, cloud SDKs, HTTP and gRPC client libraries, hand-written application loops and queue redelivery. Explicitly disable retries in every component except the one that owns them.

import random, time, uuid
import requests

IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"}

def edge_call(method, url, deadline, budget, server_dedups=False, **kw):
    """Retry loop for the ONE layer that owns retries. Inner services call once and fail fast."""
    headers = dict(kw.pop("headers", {}))
    safe = method in IDEMPOTENT or server_dedups
    if method not in IDEMPOTENT:
        # Same key on every attempt, so a server that stores key+result can dedupe.
        headers.setdefault("Idempotency-Key", str(uuid.uuid4()))
    attempt = 0
    while True:
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            raise TimeoutError("deadline exceeded")
        try:
            r = requests.request(method, url, headers=headers,
                                 timeout=(min(1.0, remaining), remaining), **kw)
        except requests.exceptions.ConnectTimeout:
            pass  # never connected, so the request was not processed: retryable
        except requests.exceptions.ReadTimeout:
            if not safe:
                raise  # server may have committed the write; do not repeat it blindly
        else:
            if r.status_code in (429, 503):
                return r  # overload: hand it back unchanged so callers stop retrying too
            if r.status_code < 500 or not safe:
                return r  # success, 4xx, or unsafe-to-repeat failure
        # Spend from the shared retry budget, e.g. a token bucket capped at ~10% of requests.
        if not budget.try_spend():
            raise RuntimeError("retry budget exhausted")
        attempt += 1
        # Capped exponential backoff with full jitter, never sleeping past the deadline.
        backoff = random.uniform(0, min(2.0, 0.1 * 2 ** attempt))
        time.sleep(min(backoff, max(0.0, deadline - time.monotonic())))

An edge-layer retry loop that retries only connect timeouts and transient 5xx on idempotent or deduplicated requests, reuses one idempotency key across attempts, and passes 429/503 overload responses back without retrying.

Why isn't exponential backoff enough without jitter?

When a dependency fails, it fails for every caller at once. A thousand clients that time out at the same instant and all follow the same schedule (wait 100 ms, then 200 ms, then 400 ms) will retry at the same instant too. Doubling the delay only spaces the waves further apart. Each wave still arrives as a synchronized spike, and it often lands just as the recovering service is warming caches or reopening connection pools, which knocks it over again.

Full jitter removes the synchronization. Instead of sleeping for the computed delay, each client sleeps for a uniformly random duration between zero and min(cap, base * 2^attempt). The upper bound still grows exponentially, but arrivals spread across the whole window rather than stacking at its edge. The recovering service sees a smooth trickle instead of a pulse. The cap stops the delay from growing without bound after a few attempts.

Backoff controls when retries happen, not how many. With jitter, a thousand clients that each retry three times still send three thousand extra requests. They just arrive spread out over time. Only the retry budget from section 3 caps that volume, so backoff has to run alongside a budget and cannot replace one.

The cap also costs latency. A 30-second maximum delay is pointless if the caller's propagated deadline expires in two seconds. The retry will be answered after the user has already given up. Before sleeping, check the time remaining on the deadline, and skip the retry if the delay plus a realistic attempt duration won't fit inside it.

What are the alternatives and complements: circuit breakers, load shedding and admission control?

Retry budgets cap how much extra load clients generate, but they assume the server can still say no cheaply. Most production systems need defenses on both sides of the call.

A circuit breaker and a budget overlap but target different failure shapes. A breaker trips when a dependency is clearly down and stops sending traffic at all, which saves the time and connections spent on doomed calls. A budget handles the murkier case of 5–20% errors, where a breaker stays closed but naive retries still multiply load.

On the server, a bounded queue matters more than it looks. An unbounded queue turns overload into latency, and latency turns into client timeouts and retries. A short queue fails fast instead. Drop any request whose propagated deadline has already passed before doing work on it, because nobody is waiting for that answer. When shedding, return an explicit overload status with a retry-after hint so well-behaved clients back off instead of guessing.

Hedged requests send a second copy after a latency threshold. They are retries by another name, so count them against the same budget.

Mechanism Protects against Runs on Does not cover
Retry budget Retry amplification during partial failure Client Total outage; first-attempt overload
Circuit breaker Wasted calls to a dead dependency Client Intermittent errors below trip threshold
Load shedding / admission control Accepting more work than capacity Server Clients that ignore rejections
Bounded queue Overload hidden as latency Server Expensive requests already admitted
Expired-deadline drop Work nobody will receive Server Requests sent without deadlines
Overload response + retry-after Synchronized client retries Server → client Clients that discard the hint
Budgeted hedging Tail latency Client Capacity shortfall; adds load

When should you adopt retry budgets, when should you avoid them, and what do they cost?

Retry budgets pay off where retries can multiply: deep synchronous call graphs, dependencies shared by many callers, and high-fan-out services where one user request becomes dozens of downstream calls. They add little where retries cannot compound.

Situation Retry budget value Reason
Deep synchronous graph (3+ hops) High Attempts multiply per layer
Shared database or auth service High Many clients' retries converge
High fan-out aggregator High One slow leaf triggers many retries
Shallow single-hop system Low A per-request cap already bounds load
Batch or async jobs with durable retry queues Low The queue already paces retries
Very low-traffic service Low The minimum allowance dominates

Illustrative estimate (assumed numbers, not a benchmark): effective allowance = max(min_retries_per_sec, ratio × request_rate). With an assumed ratio of 0.1 and a floor of 10 retries/s, a service at 20 req/s gets max(10, 2) = 10 retries/s. That is 50% of its traffic, so the budget constrains nothing.

The cost is real. Deadlines must be plumbed through every hop. You need an inventory of every retry layer. You also need instrumentation and tuning. Expect pushback from product teams when a tight budget fails more requests during a brief blip that unlimited retries would have absorbed. The savings come from fewer cascading incidents and faster diagnosis, because retry amplification becomes visible. You also stop hand-tuning timeouts service by service.

Measure the retry ratio per client and per dependency, attempts per logical request, remaining deadline at each hop, and requests shed because their deadline expired. Alert on a rising retry ratio. It moves before latency and error rates do, which gives you time to act before the storm forms.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to retry storm microservices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.