What is a microservice retry storm and why does it persist?
A retry storm is a capacity feedback loop. A dependency slows down or starts failing some requests. Callers time out and retry. The retries add load to a service that is already short on capacity, so more requests time out, which triggers more retries. Each retry is reasonable for the client that sends it, but the combined effect is a self-inflicted overload.
These storms often outlast their cause, which is the metastable failure pattern. The original trigger might be a slow deploy, a garbage-collection pause, or a brief network fault. After it clears, the retry traffic alone can keep the dependency past capacity. The system stays degraded until something breaks the loop, such as shedding load, disabling retries, or scaling well beyond normal headroom.
Amplification multiplies across layers. Illustrative estimate, assumed inputs: three layers each make up to 3 attempts, so one user request can produce up to 3 × 3 × 3 = 27 attempts at the deepest service.
user ──> gateway (3 attempts)
└─> service A [3 calls, each makes up to 3 attempts]
└─> service B [9 calls, each makes up to 3 attempts]
└─> database [up to 27 attempts]
The rule follows from the multiplication: retry at one layer and have the others fail fast. That layer is usually the one closest to the failure, which knows the most about what went wrong, or the edge, which knows the most about the user's deadline. Choosing between them is a judgment call, and someone has to own it. If one team adds "just two retries" to a client library, it multiplies every other layer's policy without changing anyone else's configuration.
ILLUSTRATIVE ESTIMATE (assumed: 3 retrying layers, up to 3 attempts each)
┌───────────┐
│ user │
└─────┬─────┘
│ 1 request
▼
┌───────────┐
│ gateway │ up to 3 attempts
└─────┬─────┘
│ up to 3 calls ┌───────────────────────┐
▼ │ feedback loop: │
┌───────────┐ │ slow DB -> timeouts │
│ service A │ up to 3 attempts per call │ -> retries at every │
└─────┬─────┘ │ layer -> more DB load │
│ up to 3 x 3 = 9 calls └───────────┬───────────┘
▼ │
┌───────────┐ │
│ service B │ up to 3 attempts per call │
└─────┬─────┘ │
│ up to 9 x 3 = 27 attempts │
▼ │
┌───────────┐ │
│ database │◄────────────────────────────────────────┘
└───────────┘ slow or partially failing
Illustrative estimate with assumed inputs: when three layers each retry up to 3 times, one user request can become up to 27 attempts at the database, and the timeouts this causes feed more retries back into the already overloaded dependency.
Why don't exponential backoff and jitter prevent retry storms?
Backoff and jitter solve a real problem, but it is a narrower one than most teams assume.
Exponential backoff lowers the rate at which each client retries. Instead of hammering a failing dependency every few milliseconds, a client waits progressively longer between attempts, which spreads its own load over time. Jitter addresses a different failure mode. When many clients fail at the same moment and back off on the same schedule, their retries arrive in synchronized waves. Full jitter picks each delay uniformly between zero and the current backoff ceiling, and decorrelated jitter derives each delay from the previous one with randomness. Both scatter those waves into a smoother stream.
Neither one caps the total number of retries across the fleet. The full-jitter example accompanying this section makes this concrete. It sets the delay to a random value up to min(cap, base * 2^attempt) and loops until attempt reaches max_attempts. Change base, cap, or the jitter strategy, and only the delays move. Every failed request still produces max_attempts calls. Under sustained failure, a fleet sending 1,000 requests per second with three attempts each still asks the dependency for up to 3,000 calls per second (illustrative estimate). The extra load arrives later and more evenly, but it is not removed.
This is a capacity problem, not a timing problem. It also has a limit. If a dependency is overloaded by its baseline traffic alone, no retry policy will bring it back. Recovery requires load shedding, added capacity, or less demand.
import random
def call_with_full_jitter(op, max_attempts=3, base=0.1, cap=5.0, sleep=lambda s: None):
# In production, pass time.sleep. The stub keeps this demo instant.
for attempt in range(max_attempts):
try:
return op()
except ConnectionError:
if attempt == max_attempts - 1:
raise # Out of attempts: surface the failure.
# Full jitter: pick a delay uniformly between 0 and the capped ceiling.
# This changes WHEN the next call happens, never WHETHER it happens.
sleep(random.uniform(0, min(cap, base * 2 ** attempt)))
calls = 0
def failing_dependency():
# Simulates sustained failure: every call reaches the dependency and fails.
global calls
calls += 1
raise ConnectionError("dependency unavailable")
# Tune the delay curve aggressively in both directions.
for base, cap in [(0.01, 0.5), (1.0, 60.0)]:
calls = 0
for _ in range(1000): # 1,000 incoming requests
try:
call_with_full_jitter(failing_dependency, base=base, cap=cap)
except ConnectionError:
pass
# Both configurations print 3000: load on the dependency is unchanged.
print(f"base={base} cap={cap}: {calls} calls for 1000 requests")
Full-jitter backoff with two very different delay curves still sends exactly max_attempts calls per failed request, because tuning the backoff only moves the retries in time and does not reduce how many there are.
How does a retry budget work?
A retry budget makes retry eligibility depend on recent traffic instead of on the individual request. The client counts requests and retries over a sliding window, often around ten seconds, and allows a retry only while retries stay under a fixed fraction of requests, such as 10%. It also allows a small minimum rate, for example a few retries per second, so a client sending two requests a minute can still recover from one dropped connection. When the budget is spent, the client returns the error immediately.
This works because the limit is on load, not on attempts per request. Each client can add at most r × requests extra calls, so total load on the dependency stays at roughly (1 + r) times baseline. This holds whether one client is retrying or ten thousand are, because each one is capped against its own traffic.
Illustrative estimate (assumed numbers, not a benchmark). Assume 1,000 requests/s, a 10% budget, and a three-hop call chain.
- Worst-case load per hop is
1,000 × (1 + 0.1) = 1,100requests/s. - Across three hops, amplification is
1.1³ ≈ 1.33×. - With a fixed policy of three retries per request instead, the per-hop multiplier is 4, so three hops give
4³ = 64×.
Teams running multi-hop service graphs, shared dependencies, or a service mesh should adopt budgets. Those are the setups where amplification compounds across hops and lands on one shared dependency.
The cost shows up during brief, widespread failures. Some requests that one more retry would have rescued fail instead, so users see more errors and the system stays recoverable. The minimum floor also has a cost. For quiet callers, the ratio is noisy because a handful of retries can exceed it. Across many callers, the floor adds a baseline of retries: 500 low-traffic clients each allowed 1 retry/s can still send 500 retries/s to a struggling service.
How do you implement a retry budget with a token bucket?
A bucket per caller holds tokens up to a cap. Each original request deposits a fraction r of a token (0.1 allows roughly one retry per ten requests), each retry withdraws one whole token, and a floor refill adds f tokens per second so a quiet client can still retry occasionally. The cap stops a long healthy period from banking a burst of retries for the next outage.
Check-before-retry path: on a retryable error, confirm the deadline has time left and the bucket holds at least one token. If both are true, withdraw a token, then back off with jitter and retry.
Fail-fast path: if either check fails, return the original error immediately and increment a "budget exhausted" counter. Never queue the retry, because queued retries are how a storm builds.
| Placement | Consistency | Coordination cost | Failure mode |
|---|---|---|---|
| Per client instance | Local only | None | Absolute volume grows with instance count |
| Per destination (in-process) | Local, scoped per dependency | None | One failing dependency cannot drain the budget for others |
| Mesh sidecar | Uniform across languages | Proxy configuration | Proxy cannot see application deadlines or idempotency |
| Global shared | Fleet-wide | Network round trips, shared state | Budget store becomes a dependency that can itself fail |
Per-instance budgets bound retries as a ratio of each instance's traffic, but the absolute volume still scales with the size of the fleet.
Illustrative estimate (assumed numbers, not a benchmark): max retries/s ≈ N × (r × q + f). With N = 2,000 instances, q = 50 req/s per instance, r = 0.1 and f = 1, the fleet can send 2,000 × (5 + 1) = 12,000 retries/s.
Hedged requests must withdraw from the same bucket, and you should never hedge non-idempotent operations. The development cost is small: one shared library that every team uses, which replaces per-team tuning of backoff curves.
import random, threading, time
class RetryableError(Exception): pass
class RetryBudget:
def __init__(self, ratio=0.1, floor_per_s=1.0, cap=10.0):
self.ratio, self.floor, self.cap = ratio, floor_per_s, cap
self.tokens, self.last = cap, time.monotonic()
self.lock = threading.Lock()
def _refill(self): # floor refill lets quiet clients retry; cap prevents banking
now = time.monotonic()
self.tokens = min(self.cap, self.tokens + (now - self.last) * self.floor)
self.last = now
def deposit(self): # every original request adds a fraction of a token
with self.lock:
self._refill()
self.tokens = min(self.cap, self.tokens + self.ratio)
def try_withdraw(self): # every retry (or hedge) spends one whole token
with self.lock:
self._refill()
if self.tokens >= 1.0:
self.tokens -= 1.0
return True
return False
def call(send, budget, deadline, metrics, attempt=0):
budget.deposit()
while True:
try:
return send(timeout=deadline - time.monotonic())
except RetryableError:
attempt += 1
if deadline - time.monotonic() <= 0 or not budget.try_withdraw():
metrics["retry_budget_exhausted"] += 1 # fail fast, never queue
raise
remaining = deadline - time.monotonic()
time.sleep(random.uniform(0, min(remaining / 2, 0.05 * 2 ** attempt)))
A per-destination retry budget: requests deposit 0.1 token, retries spend one, a floor refill and cap bound the bucket, and retries fail fast when the deadline or budget runs out.
How should you set timeouts and propagate deadlines alongside retries?
Set each per-try timeout from the dependency's measured latency distribution, not from a round number like 1 or 5 seconds. A common starting point is a high percentile, such as p99, plus headroom. That value fails fast on true outliers without cutting off requests that are only slow.
Each try needs its own timeout, and the whole request also needs a deadline that travels with it. The edge service sets an absolute deadline. Every hop passes the remaining time downstream in request metadata and subtracts its own elapsed time. When the deadline expires, every service in the chain stops working and stops retrying. Without this, a service deep in the graph keeps retrying for a caller that gave up long ago, which adds load that helps no one.
The constraint is:
attempts × per_try_timeout + sum(backoff delays) ≤ remaining_deadline
Before each retry, compute the remaining time. Retry only if it is at least the next backoff delay plus one full per-try timeout.
Illustrative estimate (assumed numbers, not a benchmark):
| Input | Value |
|---|---|
| Dependency p99 | 180 ms |
| Headroom | 1.5× |
| Per-try timeout | 180 × 1.5 = 270 ms |
| Overall deadline | 800 ms |
| Backoff delays | 50 ms, then 100 ms |
- Three attempts need 3 × 270 + 150 = 960 ms. That does not fit in 800 ms.
- Two attempts need 2 × 270 + 50 = 590 ms. That fits, so cap retries at one.
The trade-off:
- Timeouts set too tight turn normal latency rises, such as GC pauses or cold caches, into spurious retries. That adds load exactly when the dependency is already slower.
- Timeouts set too loose hold threads and connections for slow calls, which can exhaust pools upstream.
Latency distributions shift as code, traffic, and infrastructure change. Recompute these values periodically from current percentiles so that last year's p99 isn't still setting this year's timeouts.
import time
PER_TRY_TIMEOUT = 0.270 # dependency p99 (180 ms) x 1.5 headroom
BACKOFFS = [0.050, 0.100] # delay before retry 1 and retry 2
class DeadlineExceeded(Exception):
pass
def deadline_from_header(remaining_ms: str) -> float:
# Convert the caller's relative budget into a local absolute deadline.
return time.monotonic() + int(remaining_ms) / 1000
def call_with_deadline(send, deadline: float):
"""send(timeout, headers) makes one attempt and raises TimeoutError on timeout."""
attempt = 0
while True:
remaining = deadline - time.monotonic()
if remaining <= 0:
raise DeadlineExceeded() # caller has given up; do no more work
# A single try must never outlive the overall deadline.
try_timeout = min(PER_TRY_TIMEOUT, remaining)
try:
# Propagate this attempt's budget so downstream stops when we do.
return send(timeout=try_timeout,
headers={"x-deadline-ms": str(int(try_timeout * 1000))})
except TimeoutError:
if attempt >= len(BACKOFFS):
raise # retry limit reached
delay = BACKOFFS[attempt]
remaining = deadline - time.monotonic()
# Retry only if the backoff plus one full per-try timeout still fits.
if remaining < delay + PER_TRY_TIMEOUT:
raise
time.sleep(delay)
attempt += 1
# With an 800 ms deadline: try 1 times out at ~270 ms, 530 ms remain (>= 50 + 270),
# so retry; try 2 ends at ~590 ms, 210 ms remain (< 100 + 270), so stop.
# deadline = deadline_from_header("800")
A retry loop that checks remaining time against the next backoff plus a full per-try timeout before retrying, caps each try at the remaining deadline, and passes the attempt's budget downstream in a request header.
When should you retry, and when should you avoid retrying?
Retry only when a second attempt is both safe and likely to succeed. Safety comes from idempotency. GET, PUT, and DELETE are idempotent by design. POST is safe to retry only when the client sends an idempotency key that the server deduplicates. Without that key, retrying a request that timed out can charge a customer twice, because a timeout does not tell you whether the server ran the operation.
A retry is likely to succeed only when the failure is transient: a refused connection, a reset, or a 503/UNAVAILABLE that means "try another instance or try shortly." Errors that describe the request itself will return the same result on every attempt.
| Error class | Example | Action |
|---|---|---|
| Connection failure | refused, reset before response | Retry if idempotent |
| Transient unavailability | HTTP 503, gRPC UNAVAILABLE |
Retry with backoff, within budget |
| Rate limited | HTTP 429 | Wait for Retry-After, else back off |
| Client error | 400, 401, 403, 404, 409 | Do not retry |
| Upstream deadline exceeded | DEADLINE_EXCEEDED from a callee |
Do not retry; the caller's time is gone |
| Server says stop | explicit do-not-retry metadata | Do not retry |
Treat Retry-After and pushback signals as hard requirements. The server knows its own load better than your client does.
Retries also happen outside your own code. SDK defaults, HTTP client libraries, sidecar proxies, load balancers, and service mesh policies can each retry on their own. If three layers each make three attempts, one request can turn into 27. Audit every hop and turn off retries everywhere except the single layer that enforces the budget. One configured budget limits only the retries that pass through it.
What are the alternatives and complements to retry budgets?
A budget covers only one failure mode: clients amplifying their own load. The other controls cover different risks, and a production system usually needs several of them together.
| Mechanism | What it limits | Enforced by | How it fails |
|---|---|---|---|
| Retry budget | Extra attempts as a share of normal traffic | Client | Cannot see load from other clients; too low a ratio drops recoverable requests |
| Circuit breaker | All calls to an endpoint that looks unhealthy | Client | Trips on partial failures and blocks requests that would have succeeded; slow to detect recovery |
| Server-side load shedding | Accepted work when the server is saturated | Server | Rejects some valid requests by design; bad priority rules shed the wrong traffic |
| Concurrency limit | In-flight requests per instance | Server or proxy | A static limit goes stale when capacity or latency changes |
| Fast, cheap rejection | Cost of saying "no" | Server | If rejection does real work, such as auth or parsing, overload still wins |
| Hedged requests | Tail latency, by sending a second copy | Client | Adds load; without a cap it behaves like a retry storm |
A circuit breaker and a budget are easy to confuse, but they behave differently. A breaker stops all calls to an endpoint that looks unhealthy, including first attempts. A budget caps only the extra retry load, so normal traffic keeps reaching a service that is degraded but still working.
Client-side controls are still not enough, because each client sees only its own traffic. The server is the only component that sees aggregate load from every caller, including misconfigured or unbudgeted ones. That makes load shedding and concurrency limits on the server a requirement. Clients reduce how much excess load reaches the server, and the server still has to protect itself from whatever arrives.
How do you monitor and test retry budgets before an incident?
Track four signals, each split by caller and by destination:
- Attempts per logical request. Record the attempt count as a histogram on the client, tagged by first try versus retry. If the mean rises above 1 + budget ratio, a policy is retrying outside the budget.
- Retry ratio per caller and per destination. Aggregate numbers hide the one caller that is hammering a single dependency.
- Budget exhaustion rate. This is how often a retry was denied because the bucket was empty. A small, nonzero rate during a dependency failure means the budget is working. A rate that stays nonzero in steady state means the budget is too tight or the dependency is unhealthy.
- Deepest-service load versus edge request rate. This ratio shows amplification directly, without reasoning about each hop separately.
To test the budget in staging, inject latency and errors into one dependency. Use a sidecar or proxy fault filter, or a test flag in the dependency itself. Then confirm that the request rate reaching it stays near 1 + budget ratio times the healthy rate.
Illustrative estimate (assumed numbers, not a benchmark): Assume an edge rate of 1,000 rps, a budget ratio r = 0.1, and n = 3 hops, each with its own budget.
- Worst-case deepest load = edge × (1 + r)^n = 1,000 × 1.1³ ≈ 1,331 rps.
- Without budgets, with 3 attempts per hop, the same formula gives 1,000 × 3³ = 27,000 rps.
If the measured load lands well above the budgeted bound, look for a retry layer you didn't account for, such as a library default or a mesh policy. Also check for timeouts that are longer than the caller's deadline.
The test catches stacked retry policies and misaligned timeouts in staging rather than during an outage. In a real incident, the amplification ratio is already on a dashboard, so you don't have to reconstruct it from logs.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to microservice retry storm, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.