What is a retry storm and how does it become a metastable failure?

A retry storm happens when clients respond to failures by sending more requests, and those extra requests become the main source of load on a system that is already struggling. The retries are meant to hide transient faults. Instead they push a degraded service into sustained overload.

The sequence usually runs like this. A partial outage, such as a slow database replica, a garbage-collection pause, or a bad deploy on a few hosts, raises latency or error rates. Clients hit their timeouts and retry. Offered load is now well above organic traffic, so queues grow and latency rises further. More requests time out, and more retries follow. This feedback loop is what makes the failure metastable: the system stays overloaded even after the original trigger is fixed, because retry traffic alone is enough to keep it saturated. Restarting the faulty component doesn't help. Recovery usually requires shedding load or stopping the retries.

Deep call graphs make this much worse, because retries multiply across tiers. If each of N tiers makes up to R+1 attempts per request (one initial attempt plus R retries), the worst-case load on the leaf service grows roughly as (R+1)^N.

Illustrative estimate. Assume three tiers, each making 3 attempts (R = 2). If every call fails, one user request produces up to 3 × 3 × 3 = 27 requests at the leaf, which is a 27x multiplier on the component least able to absorb it.

Backend and platform engineers who own services several hops deep are the ones who pay for this, often without seeing which caller is generating the load.

          1 user request
                |
                v
        +----------------+
        |     Tier A     |
        +----------------+
                |  up to 3 attempts
                v  = 3 requests
        +----------------+
        |     Tier B     |
        +----------------+
                |  up to 3 attempts per call
                v  = 3 x 3 = 9 requests
        +----------------+
        |     Tier C     |
        +----------------+
                |  up to 3 attempts per call
                v  = 3 x 3 x 3 = 27 requests
        +----------------+
        |  Leaf service  |----> queues grow, latency rises
        |   (degraded)   |              |
        +----------------+              v
                ^             callers hit timeouts
                |                       |
                +---- more retries <----+

  Illustrative estimate: 3 tiers, 3 attempts each, every call fails.
  Worst-case leaf load ~ (R+1)^N = 3^3 = 27x one user request.

Illustrative estimate: when each of three tiers makes up to 3 attempts and every call fails, one user request becomes 27 requests at the leaf, and the resulting timeouts produce more retries that keep the leaf overloaded.

Who should use jittered backoff and retry budgets?

Any code that retries a network call should use capped, jittered backoff. Retry budgets are for a narrower group: the people whose retries can multiply or whose services absorb other people's retries.

  • Backend engineers on multi-tier services. If your service calls another service that calls a third, your retry policy affects load several hops away. You are the one who can stop the multiplication at its source.
  • Platform and infrastructure teams that own shared client libraries, SDKs or service mesh configuration. Retry behaviour set in one shared place applies to every caller. This is the most effective place to add budgets, deadline propagation and error classification, because individual teams get them without writing their own.
  • Owners of shared dependencies. Teams running databases, auth services, configuration stores or other components that many callers depend on are the first to be overwhelmed. They benefit from callers having budgets, and they control the server-side complements: load shedding and Retry-After hints.
  • SREs and on-call engineers. They need the retry metrics described later to tell a retry storm apart from organic load during an incident, and they decide whether budget fractions are tuned correctly after incident reviews.

Teams running a single-tier, low-traffic service, or a batch consumer that already paces itself, usually need only a capped attempt count with jitter. The section on when to avoid budgets covers this in more detail.

How does retry storm prevention work?

No single mechanism prevents retry storms. The techniques in this article work as layers, each covering a gap the others leave open. For one outgoing call, the flow is:

  1. Classify the failure. Validation errors (400, 422) are never retried. Overload rejections (429, 503) are retried only after an explicit server-provided delay, if at all. Only transient errors, such as connection resets or gRPC UNAVAILABLE, go on to the retry path.
  2. Check the deadline. The caller's remaining time travels with the request. If the next backoff delay plus a realistic attempt would exceed it, the call fails now instead of doing work nobody will wait for.
  3. Check the retry budget. The client asks its budget whether retries to this destination are still under the allowed fraction of recent requests. If not, the error goes straight back to the caller. This is what bounds total load during a broad outage.
  4. Wait a jittered, capped delay. The delay grows exponentially up to a maximum and is randomized, so clients that failed together don't retry together.
  5. Retry with the same idempotency key. For writes, the server uses the key to return the stored result instead of applying the operation twice.
  6. Stop at the attempt cap. A fixed maximum attempt count limits the work spent on any single request.

Two structural rules sit around this loop. First, only one layer in the call graph, usually the edge, runs it. Inner services call once and fail fast, so attempts do not multiply across tiers. Second, the server takes part: it sheds load cheaply before doing expensive work and tells clients when to come back. Retry metrics then show whether the layers are working, by tracking retry ratio, attempts per request and budget exhaustion per destination.

Each layer handles a different dimension. Jitter controls when retries arrive. Budgets and attempt caps control how many. Deadlines control whether the work is still useful. Single-layer retries control where amplification can happen. The sections below go through each in turn.

Why doesn't exponential backoff with jitter prevent retry storms on its own?

Plain exponential backoff fixes the rate of retries per client but not their phase. When a dependency fails, every caller sees the error at roughly the same moment, computes the same delay sequence (100 ms, 200 ms, 400 ms…), and fires again together. The struggling service gets synchronized waves: a thundering herd on every retry interval, with quiet gaps between that it cannot use.

Jitter breaks the synchronization by randomizing each delay. With base as the initial delay, cap as the maximum, and n as the attempt number, the three common variants are:

Variant Delay computation Spread Downside
Full jitter random(0, min(cap, base·2ⁿ)) Widest; arrivals close to uniform across the window Can retry almost immediately
Equal jitter t = min(cap, base·2ⁿ); t/2 + random(0, t/2) Half the window is fixed, so spread is narrower Keeps some clustering; guarantees a minimum wait
Decorrelated jitter min(cap, random(base, prev·3)) Wide, and each delay depends on the previous one Harder to reason about worst-case delay

Full jitter spreads load most evenly because the delay can fall anywhere in the window, including near zero. The same property gives the widest latency distribution. Some requests wait close to the full window and others retry almost at once, so p99 for retried requests can get worse even while aggregate server load is smoother.

Jitter does not cap anything. If each of three layers retries three times, the downstream service still receives up to 27x the original traffic. Jitter only spreads that traffic over time. It shapes when the extra load arrives but leaves the total amount unchanged, so it cannot prevent a retry storm by itself.

import random

BASE = 0.1  # initial delay in seconds
CAP = 10.0  # maximum delay in seconds

def no_jitter(n):
    # Same delay for every client: failures that start together retry together
    return min(CAP, BASE * 2 ** n)

def full_jitter(n):
    # Anywhere in [0, window]: widest spread, but can be close to zero
    return random.uniform(0, min(CAP, BASE * 2 ** n))

def equal_jitter(n):
    # Half of the window is fixed, so there is always a minimum wait
    t = min(CAP, BASE * 2 ** n)
    return t / 2 + random.uniform(0, t / 2)

def decorrelated_jitter(prev):
    # Depends on the previous delay, not on the attempt number
    return min(CAP, random.uniform(BASE, prev * 3))

def downstream_amplification(attempts_per_layer, layers):
    # Jitter changes WHEN retries land, not HOW MANY there are.
    # Each layer multiplies the calls made by the layer above it.
    return attempts_per_layer ** layers

if __name__ == "__main__":
    print([round(no_jitter(n), 3) for n in range(4)])    # identical for all clients
    print([round(full_jitter(n), 3) for n in range(4)])  # differs on every run
    prev = BASE
    for _ in range(4):
        prev = decorrelated_jitter(prev)
        print(round(prev, 3))
    # 3 layers that each make 3 attempts: up to 27 downstream calls per user request
    print(downstream_amplification(3, 3))

Python versions of no-jitter, full, equal and decorrelated jitter, plus a count showing that three layers with three attempts each still reach 27x load whatever the timing.

How does a retry budget bound retry amplification?

A retry budget caps retries as a share of recent traffic instead of as a count per request. Each client tracks requests and retries over a sliding window, say the last 10 seconds, and permits a retry only while retries stay under a fraction b of requests (for example 10%) plus a small minimum rate. The minimum keeps retries available when traffic is low. When the downstream fails broadly, the budget runs out and further failures return immediately. That breaks the feedback loop: more errors no longer produce proportionally more load.

Per-request retries multiply across layers as (R+1)^N. A budget bounds each layer at (1+b) times its incoming traffic, so worst-case load at depth N is roughly base × (1+b)^N, plus the minimum-rate terms.

Illustrative estimate (assumed numbers, not a benchmark): Assume 1,000 req/s at the edge, N = 3 layers, b = 0.1, and a minimum of 10 retries/s. The bottom service sees about 1,000 × 1.1³ ≈ 1,331 req/s. With R = 2 and no budget, it sees up to 27,000 req/s.

The accompanying code puts the budget check in front of jittered backoff. The backoff has a maximum delay and a maximum attempt count. Before each sleep, the code checks the caller's remaining deadline and skips any retry that would finish after it.

There are two costs:

  • Lost recoveries during real blips. During a genuine transient blip, some requests that one retry would have saved will fail instead.
  • Per-instance budgets don't bound fleet-wide load. The bound applies to each client instance. As you scale clients from 10 to 100, total retry volume grows with the fleet. Per-destination or global budgets fix this, but they require shared state or coordination between clients.
import random, time
from collections import deque

class TransientError(Exception): pass

class RetryBudget:
    """Retries allowed: ratio * recent requests + min_per_sec * window (not thread-safe)."""
    def __init__(self, ratio=0.1, min_per_sec=10, window_s=10.0):
        self.ratio, self.min_per_sec, self.window_s = ratio, min_per_sec, window_s
        self.requests, self.retries = deque(), deque()

    def _trim(self, now):
        for q in (self.requests, self.retries):
            while q and now - q[0] > self.window_s:
                q.popleft()

    def record_request(self):
        self.requests.append(time.monotonic())

    def try_spend(self):
        now = time.monotonic(); self._trim(now)
        allowed = self.ratio * len(self.requests) + self.min_per_sec * self.window_s
        if len(self.retries) >= allowed:
            return False  # budget exhausted: fail fast instead of adding load
        self.retries.append(now); return True

def call(fn, budget, deadline, max_attempts=3, base_s=0.05, cap_s=1.0):
    budget.record_request()  # count original requests only
    for attempt in range(max_attempts):
        try:
            return fn(timeout=deadline - time.monotonic())
        except TransientError:
            if attempt == max_attempts - 1:
                raise  # attempt cap reached
            delay = random.uniform(0, min(cap_s, base_s * 2 ** attempt))  # full jitter, capped
            if time.monotonic() + delay >= deadline:
                raise  # retry would outlive the caller's deadline
            if not budget.try_spend():
                raise  # over budget: don't amplify a broad outage
            time.sleep(delay)

A sliding-window retry budget (10% of recent requests plus a minimum rate) checked after a capped, fully jittered backoff and a deadline check, so retries stop when the budget runs out, the attempt cap is reached, or the caller's time would expire.

Where in the call graph should you retry, and what is safe to retry?

Pick one layer to own retries. This is usually the edge, or the caller nearest the user. Inner services should fail fast and return an overload status or an explicit do-not-retry signal, so that upstream callers don't add attempts of their own. Propagate the deadline on every hop. A service that receives a request with 80 ms remaining should not start a retry that needs 200 ms. That work is wasted even if it succeeds.

This has a cost. A blip three tiers down now costs a full end-to-end retry instead of one cheap local retry. Accept that cost, because the alternative compounds: three layers each making three attempts can turn one user request into 27 calls (3×3×3) against the deepest service. That is an illustrative estimate.

Retry only idempotent operations. For non-idempotent writes, the client sends an idempotency key so the server can return the stored result instead of applying the write twice. Keys have real costs:

  • Storage: you must keep both the keys and their responses.
  • Retention: the retention window must outlast your longest retry horizon.
  • Key store outages: you must decide whether writes fail or proceed unprotected when the key store is down.
  • Collisions: keys need enough entropy, such as random UUIDs, so unrelated requests never collide.
Error class Examples Retry? Why
Transient Connection reset, gRPC UNAVAILABLE Yes, with backoff Likely to succeed on another attempt
Overload rejection 429, 503 with Retry-After No, or only after the stated delay The server is asking for less load
Validation 400, 422 Never The same input fails the same way

Retries configured in client libraries, the service mesh and application code multiply silently. To audit them:

  1. List every hop on each critical request path.
  2. Record the retry count and timeout configured at each layer of each hop.
  3. Multiply the attempts along each path.
  4. Confirm in distributed traces that the observed attempt counts match your calculation.
import random, time, uuid

VALIDATION = {400, 422}   # same input fails the same way: never retry
OVERLOAD = {429, 503}     # server asks for less load

def call_at_edge(send, payload, budget_s, max_attempts=3, base_s=0.05, cap_s=1.0):
    """Edge-only retry loop. Inner services call send() once and fail fast."""
    deadline = time.monotonic() + budget_s
    key = str(uuid.uuid4())  # one random key reused on every attempt of this write
    resp = None
    for attempt in range(max_attempts):
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            break
        # Propagate the remaining deadline so downstream hops skip doomed work.
        # 'X-Deadline-Ms' is a placeholder name; use your stack's convention.
        resp = send(payload, timeout=remaining, headers={
            "Idempotency-Key": key,
            "X-Deadline-Ms": str(int(remaining * 1000)),
        })
        if resp.ok or resp.status in VALIDATION:
            return resp
        if resp.status in OVERLOAD:
            retry_after = resp.headers.get("Retry-After")
            if not (retry_after and retry_after.isdigit()):
                return resp  # no stated delay: surface the rejection, don't retry
            delay = float(retry_after)
        elif resp.transient:  # e.g. connection reset, gRPC UNAVAILABLE
            delay = random.uniform(0, min(cap_s, base_s * 2 ** attempt))  # full jitter
        else:
            return resp  # unknown class: treat as non-retryable
        if delay >= deadline - time.monotonic():
            return resp  # retry could not finish before the deadline
        time.sleep(delay)
    if resp is None:
        raise TimeoutError("deadline exhausted before first attempt")
    return resp

An edge-only retry loop that sends the remaining deadline downstream and reuses a single idempotency key across attempts. It retries transient errors with full-jitter backoff, retries an overload rejection only after an explicit Retry-After delay, and never retries validation errors.

What are the alternatives and complements to client-side retry budgets?

A client-side budget only caps how much extra load callers add. The server knows its own saturation better than any client, so the most effective complements put the server in the loop.

Load shedding lets an overloaded service reject excess work cheaply, before parsing bodies or acquiring locks. A fast rejection costs far less than a request that times out after consuming resources. Retry-after hints, such as the HTTP Retry-After header or equivalent metadata, tell clients when to come back. Clients should treat the hint as a floor under their own backoff and still charge the retry to the budget.

Circuit breakers stop calls to a dependency that is failing. When they are scoped badly, they cause outages. A breaker keyed on the whole dependency can trip because one bad host behind the load balancer pushes the aggregate error rate over threshold. That turns a partial failure into a full outage for that dependency. Breakers with low minimum-request thresholds or short windows also flap between open and closed. Prefer per-host outlier ejection, require a minimum request volume before tripping, and send limited probes in the half-open state.

Hedged requests send a second copy of a request after a delay to cut tail latency. They add load on purpose, so every hedge must draw from the same retry budget. Without that, hedging becomes an unbounded second retry path during a slowdown.

Approach Protects against Where it runs Main failure mode
Load shedding Server saturation Server Shedding too late, after costly work
Retry-after hints Synchronized client returns Server signals, client obeys Clients ignoring or overriding hints
Circuit breaker Calling a failing dependency Client or proxy Flapping; one bad host trips the whole dependency
Hedged requests Tail latency Client Unbudgeted extra load during slowdowns

When should you use retry budgets, and when should you avoid them?

Use budgets when retries can multiply

Retry budgets pay off when amplification compounds:

  • Multi-tier call chains. Retries stack per tier, so worst-case load is (1 + retries_per_tier) ^ tiers.
  • Shared dependencies. A database or auth service used by many callers takes retries from all of them at once.
  • High fan-out. One user request that spawns dozens of downstream calls turns a small error rate into many retries.
  • Large client fleets. Thousands of instances each retrying "reasonably" still add up to a thundering herd.

Illustrative estimate (assumed numbers, not a benchmark): assume 3 tiers and 3 retries per tier. The worst case is (1 + 3)^3 = 64× load at the bottom tier. With a 10% budget per tier, the bound is (1 + 0.1)^3 ≈ 1.33×.

Avoid them or keep them simple elsewhere

  • Single-tier, low-traffic services. A capped retry count with jittered backoff is enough when there is nothing to multiply.
  • Batch jobs that pace themselves. A rate-limited queue consumer already bounds its own load.
  • Paths that need a predictable schedule. Examples are payment reconciliation or webhook delivery with documented retry intervals. A budget that silently drops retries breaks that contract.

The cost trade-off

The upfront work is real. You need per-client budget accounting, an error classification that separates retryable from non-retryable failures, and idempotency stores for non-idempotent writes. That cost buys fewer incidents that sustain themselves after the trigger is gone. Recovery is shorter because load falls back toward baseline once the dependency heals. On-call engineers also spend less time shedding load by hand, scaling down callers, or flipping feature flags during an outage.

How can retry budgets and jittered backoff save engineering time?

The time savings come from removing work that retry storms otherwise create, both during incidents and in the weeks around them.

  • Incidents end when the trigger is fixed. In a metastable failure, fixing the slow replica or rolling back the bad deploy is not enough. Engineers then have to drain queues, throttle or restart callers, and bring traffic back in stages. With budgets in place, retry load stays bounded, so the system can usually recover once the original fault is gone. That removes the second, often longer, phase of the incident.
  • Less manual mitigation under pressure. Without bounded retries, the usual tools are blunt: scaling down callers, disabling features or blocking traffic at the edge. Each needs coordination across teams during an outage. Budgets and server-side load shedding do the equivalent automatically and per destination.
  • Faster diagnosis. Retry ratio, attempts per request and budget exhaustion events, labelled by destination, answer the first incident question directly: is this organic load or amplification, and which caller is producing it? Without them, engineers reconstruct the answer from traces and logs across several services.
  • One policy instead of many. Putting jitter, budgets, deadline propagation and error classification in a shared client library or mesh configuration means each service team doesn't design, tune and debug its own retry loop. A fix or tuning change made once applies everywhere.
  • Less cleanup after duplicate writes. Idempotency keys mean a retried write that already succeeded returns the stored result. Without them, retries during an incident can leave duplicate orders, charges or records that someone has to find and reconcile by hand afterwards.
  • Fewer surprises from hidden retry layers. The audit described above, multiplying attempts along each path and checking against traces, finds stacked retries in SDKs, proxies and application code before they cause an incident, not during the post-incident review.

The trade-off is the upfront work listed in the previous section, plus ongoing budget tuning after incident reviews. For teams running multi-tier systems with shared dependencies, that is usually a smaller and more predictable cost than repeated long incidents.

Which retry metrics should you monitor and alert on?

None of these effects shows up unless the retry layer emits its own signals. Three metrics cover most of what matters. Label each by destination service and, where possible, by endpoint, because a fleet-wide average hides the one dependency that is degrading.

  • Retry ratio: retries divided by total attempts sent to a destination. This is the most direct measure of amplification. A steady 1–2% may be normal background noise. A climb toward the budget fraction means callers are already spending their headroom.
  • Attempts per logical request: a histogram recorded once per caller-level operation. The mean tells you the load multiplier. The tail shows requests that hit every allowed attempt, which is the first sign of a storm forming.
  • Budget exhaustion events: a counter that increments whenever a retry is denied because the budget is spent. Occasional exhaustion during an incident is the budget doing its job. Sustained exhaustion during normal traffic means the budget is too tight or the dependency is chronically unhealthy.

Alert on the trend in the ratio, not on single errors. For example, page when sum by (destination) (rate(client_retries_total[5m])) / sum by (destination) (rate(client_attempts_total[5m])) stays above half the configured budget fraction for ten minutes. That fires while you still have room to act.

Compare retry ratios across tiers to find silent retriers. If a backend receives more attempts than its direct callers report sending, some layer between them, such as a proxy, SDK, or client library, is retrying without emitting metrics. Set budget fractions from observed baseline ratios plus a margin, and revisit them after each incident review.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to retry storm prevention, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.