Why do per-hop static timeouts cause cascading timeouts?
Consider a synchronous chain: the edge gateway calls service A, A calls B, and B calls C. The edge gives up after 2 seconds. A, B and C were each configured by different teams to wait 5 seconds on their downstream call. Each value looks reasonable in isolation. Together they are inconsistent.
When C slows down, the edge returns an error at 2s. Nothing tells A, B or C. A keeps waiting on B, and B keeps waiting on C, for up to 5s. For roughly 3 seconds, every hop holds a connection, a worker thread or goroutine, memory, and CPU for a response nobody will read.
Illustrative estimate (assumed numbers, not a benchmark). Assume 100 req/s at the edge and normal latency of 100 ms per hop.
- In-flight requests per hop = arrival rate × time held (Little's law).
- Normal: 100 × 0.1 = 10 in flight.
- Incident: 100 × 5 = 500 in flight, and (5 − 2) / 5 = 60% of that work is for callers that have already left.
- If clients retry twice (3 attempts total), arrival becomes 300 req/s and in-flight reaches 1,500. If each of the three layers also makes 3 attempts, C can receive up to 3³ = 27 calls per original request.
These requests pile onto C at the moment it is slowest. Its queues grow, latency rises further, and the backlog spreads upstream.
C's slow code started the incident, but it is not the root cause of the cascade. The cause is timeouts set as independent constants, with no shared idea of how much time the original caller has left. Any team running synchronous call chains more than one hop deep carries this risk.
Client: 100 req/s (300 req/s if clients retry twice)
|
v
+-----------+ +-----------+ +-----------+ +-----------+
| Edge |---->| Service A |---->| Service B |---->| Service C |
| timeout 2s| | timeout 5s| | timeout 5s| | (slow) |
+-----------+ +-----------+ +-----------+ +-----------+
^ ^ ^ |
+-------- queues grow, backlog spreads upstream ------+
time (s) 0 1 2 3 4 5
|---------|---------|---------|---------|---------|
Edge [===================]X edge returns error at 2s
A waits B [=================================================]
B waits C [=================================================]
C working [=================================================]
|<----- wasted work ~3s ----->|
conns, threads, memory, CPU held
for callers that already left
In flight per hop = rate x time held (Little's law)
normal: 100 x 0.1s = 10 incident: 100 x 5s = 500 (60% wasted)
with client retries: 300 x 5s = 1,500
Retry amplification, 3 attempts at each layer:
Edge x3 ---> A x3 ---> B x3 ---> C receives up to 27 calls/request
Illustrative estimate with assumed values: a 2s edge timeout over three 5s inner timeouts leaves A, B and C working about 3s after the edge gives up, and layered retries can multiply calls to the slow service C up to 27-fold.
Who should use deadline propagation?
Deadline propagation pays off when the failure mode above is realistic for your system. It is a good fit for:
- Synchronous call chains deeper than two hops. Each extra hop is another place where a static timeout can outlive its caller.
- Systems owned by several teams. When different teams choose timeouts independently, nobody holds the whole chain in their head, and inconsistencies like 2s-over-5s go unnoticed until an incident.
- User-facing paths with a latency objective. If a request is worthless after 1 second, downstream services should know that and stop at 1 second.
- Services that fan out or retry. Both multiply load, and both need a shared budget to stay bounded.
- Teams already on gRPC. Deadlines and cancellation are built in, so the main work is making sure every hop passes the incoming context along.
It is less useful for single-hop services, systems that are mostly asynchronous or queue-driven, and batch workloads where no caller is waiting. Those cases are covered later under when to avoid or adapt it.
What is the difference between a timeout, a deadline, and a request budget?
A timeout is a relative duration: "give up after 500 ms." A deadline is an absolute point in time: "give up at 12:00:00.800." A request budget is the time remaining before that deadline, computed as remaining = deadline − now at the moment it is checked.
A deadline carries through a call chain without adjustment because every hop measures against the same fixed point, so time spent anywhere is accounted for automatically. A forwarded duration does not. Each hop must subtract its own elapsed work and queueing before passing the duration on, or the downstream call can outlive the caller that is waiting for it.
Illustrative estimate (assumed numbers, not a benchmark): the caller allows D = 800 ms. Service A spends E = 300 ms working and Q = 50 ms queued before calling B. B's correct budget is D − E − Q = 450 ms. If A forwards the raw 800 ms instead, B can keep working for up to 350 ms after the caller has given up.
| Approach | Correctness | Coupling | Clock-skew sensitivity |
|---|---|---|---|
| Static per-hop timeouts | Breaks when the sum of hop timeouts exceeds the caller's timeout | Low at runtime, high when tuning | None |
| Forwarded duration | Correct only if every hop subtracts elapsed and queue time | Every hop must follow the protocol | None |
| Propagated deadline / remaining budget | Composes across hops | Every hop must follow the protocol | High if absolute timestamps are sent between hosts. Low if remaining time is sent and turned into a local deadline on receipt, at the cost of ignoring network transit time |
gRPC uses the third approach. It sends the remaining time on the wire, and each receiver converts it to a local deadline.
Propagation is not always required. Static timeouts can be enough when they are set top-down, so that each hop's timeout plus retries fits inside its caller's timeout, and when circuit breakers and bulkheads limit the damage from a slow dependency. This works for shallow chains of one or two hops that change rarely and are owned by one team. As chains get deeper or change more often, keeping hand-tuned timeouts consistent becomes the problem.
How does deadline propagation work?
The mechanism is the same whether the transport is gRPC or plain HTTP:
- Set the budget once, at the edge. The first service that receives the user's request decides how long the whole operation may take, usually from the latency objective.
- Convert to a local deadline on arrival. Each hop reads the remaining time from the request and computes
local_deadline = now + remainingusing its own monotonic clock. - Check before doing work. If the remaining time is below the minimum an operation needs, the hop fails immediately instead of starting work it cannot finish.
- Derive every downstream limit from the deadline. Outbound calls, database statements, pool waits and retries all get a timeout of
deadline − now, minus some headroom, instead of a fixed constant. - Forward the shrinking remainder. Each outbound request carries the smaller remaining budget, so the next hop repeats steps 2–5.
- Cancel when the deadline passes. Expiry or caller cancellation is signalled to in-flight downstream work so it stops and releases connections and threads.
The next two sections show this mechanism in gRPC, where the library does most of it, and over plain HTTP, where you build it in middleware.
How do gRPC deadlines propagate through a call chain?
gRPC removes the hand-tuning by treating the deadline as part of the request, so it travels with the call.
The client sets a deadline once, usually with context.WithTimeout(ctx, 800*time.Millisecond). When the call goes out, the library converts the remaining time into the grpc-timeout header. The value is a duration such as 750m for 750 milliseconds, not a wall-clock timestamp, so clock skew between hosts does not distort it. On the server, the library reads the header and attaches a deadline to the request's ctx. Any outgoing gRPC call made with that same ctx recalculates what is left and sends a smaller grpc-timeout to the next hop. The budget shrinks at every hop without anyone configuring it.
Cancellation follows the same path. If the deadline passes or the caller cancels, for example because a user closed the page, the client resets the stream. The server's ctx.Done() channel closes and ctx.Err() returns DeadlineExceeded or Canceled. Downstream calls tied to that context are cancelled too. Database calls such as db.QueryContext(ctx, ...) also stop if the driver honors context cancellation, which frees connections instead of finishing work nobody will read.
The chain only holds if every hop passes the incoming ctx along. The most common bug is a handler that calls context.Background(), often added to "fix" an early cancellation. That call discards both the deadline and the cancellation signal. From that point on, downstream work runs without any limit. It keeps running after the original caller has already given up and possibly retried, so the abandoned work and the retry compete for the same downstream capacity.
// Go. Client: set the budget once at the edge of the call chain.
ctx, cancel := context.WithTimeout(context.Background(), 800*time.Millisecond)
defer cancel() // release timer resources even if the call returns early
order, err := ordersClient.GetOrder(ctx, &pb.GetOrderRequest{Id: "ord-123"})
// Server: grpc-go has already decoded grpc-timeout into ctx's deadline.
func (s *server) GetOrder(ctx context.Context, req *pb.GetOrderRequest) (*pb.Order, error) {
// Downstream gRPC call: passing ctx sends the *remaining* time as a
// smaller grpc-timeout and forwards cancellation to the next hop.
stock, err := s.inventory.CheckStock(ctx, &pb.StockRequest{OrderId: req.Id})
if err != nil {
return nil, err // DeadlineExceeded or Canceled flows back up
}
// Database query: aborts when ctx is done, if the driver honors context.
var st string
err = s.db.QueryRowContext(ctx,
"SELECT status FROM orders WHERE id = $1", req.Id).Scan(&st)
if err != nil {
return nil, err
}
// BUG (do not do this): a fresh root context drops both the deadline
// and the cancellation signal. This call keeps running after the
// client has given up and possibly retried.
//
// s.inventory.CheckStock(context.Background(), &pb.StockRequest{...})
return &pb.Order{Id: req.Id, Status: st, InStock: stock.Available}, nil
}
A Go client sets an 800 ms deadline once, and the server passes the incoming ctx to a downstream gRPC call and a database query so the shrinking budget and cancellation reach both, while context.Background() would cut the chain.
How do you implement request budget propagation over plain HTTP?
Choose one header, such as X-Request-Budget-Ms, and carry the remaining milliseconds rather than an absolute timestamp. An absolute deadline is only as accurate as the clock agreement between two hosts. If the receiver's clock runs 300 ms ahead, it has already lost 300 ms of the budget. A relative value avoids that problem. The cost is that time spent in transit, and time the request waits in a socket or accept queue before your code parses it, goes uncounted. Treat that drift as a known error and absorb it in headroom.
Ingress middleware reads the header and clamps it to [floor, max_budget]. External callers can send any value, so an unclamped header lets anyone hold your workers for minutes. Requests with no header get a default. The middleware computes local_deadline = now_monotonic + budget and attaches it to the request context. It should reject the request immediately if the budget is already below the floor.
Egress middleware sends remaining = local_deadline − now − headroom and sets the HTTP client timeout to the same value. Database statement timeouts and queue or pool waits must read that deadline too. Otherwise one unbounded call undoes the work.
At the edge, derive the top-level budget from the user-facing latency objective.
Illustrative estimate (assumed numbers, not a benchmark): with a 1,000 ms objective, 100 ms reserved for edge response handling, and 20 ms of headroom per hop for serialization and transit, the third hop receives 1000 − 100 − 3 × 20 − upstream_work ms.
import contextvars, time, urllib.request
HEADER = "X-Request-Budget-Ms"
DEFAULT_MS = 900 # edge: 1000 ms objective minus 100 ms response handling
MAX_MS = 5000 # clamp: external callers cannot pin workers for minutes
FLOOR_MS = 50 # below this, starting work is pointless
HEADROOM_MS = 20 # per hop: serialization, transit, response handling
_deadline = contextvars.ContextVar("deadline")
class BudgetExhausted(Exception):
pass # map to 503/504 at the framework layer
def ingress(headers):
try:
budget = int(headers.get(HEADER, DEFAULT_MS))
except ValueError:
budget = DEFAULT_MS
budget = min(budget, MAX_MS)
if budget < FLOOR_MS:
raise BudgetExhausted(budget)
# Monotonic clock is immune to wall-clock jumps. Transit and accept-queue
# time before this line is already lost and uncounted; headroom absorbs it.
_deadline.set(time.monotonic() + budget / 1000)
def remaining_ms(headroom=HEADROOM_MS):
left = int((_deadline.get() - time.monotonic()) * 1000) - headroom
if left < FLOOR_MS:
raise BudgetExhausted(left)
return left
def call(url):
ms = remaining_ms()
req = urllib.request.Request(url, headers={HEADER: str(ms)})
# Same value bounds our own wait, so we never outlive the callee's budget.
return urllib.request.urlopen(req, timeout=ms / 1000)
def query(cur, sql, args):
# PostgreSQL: SET LOCAL scopes the timeout to the current transaction.
cur.execute(f"SET LOCAL statement_timeout = {remaining_ms()}")
cur.execute(sql, args)
Framework-neutral Python middleware that parses and clamps the remaining-milliseconds header into a monotonic local deadline, then derives the outbound header, HTTP client timeout and PostgreSQL statement timeout from that same deadline minus per-hop headroom.
How should services fail fast, retry, and cancel within a budget?
Check the remaining budget before doing any work, not after. Each operation has a minimum useful time: the shortest window in which it can realistically succeed. If the remaining budget is below that floor, reject immediately with a deadline-exceeded error. Starting work you cannot finish only consumes connections and CPU that a request with a real chance of success could have used.
Run the same check when a job is pulled from a queue. A request can spend most of its budget waiting there, and under load that is when the wait is longest. Without the check, workers process requests whose callers have already given up.
Retries must draw from the same remaining budget, not start a fresh one. Before each attempt, set the attempt timeout to the smaller of the per-attempt cap and the time left. Add jittered backoff so that clients failing at the same moment do not retry in lockstep. Stop when the remaining budget drops below the minimum useful time. Retry at only one layer. If each of three layers retries three times, a single failing call at the bottom can be attempted 27 times.
Cancellation is cooperative. A deadline only takes effect if the cancellation context reaches database drivers, HTTP clients, and worker goroutines or threads. Confirm in tests that cancelled queries actually stop and that connections return to the pool.
Non-idempotent writes are the deliberate exception. If a request is cancelled partway through a write, aborting can leave inconsistent state. Detach the write from the request deadline and give it its own bounded timeout.
import asyncio, random, time
MIN_USEFUL_S = 0.05 # shortest window in which the call can realistically succeed
PER_ATTEMPT_CAP_S = 0.3 # upper bound for any single attempt
class DeadlineExceeded(Exception):
pass
async def call_with_budget(op, deadline, max_attempts=3):
# deadline is time.monotonic()-based. This is the ONLY layer that retries.
backoff = 0.02
for attempt in range(max_attempts):
left = deadline - time.monotonic()
if left < MIN_USEFUL_S: # fail fast: not enough time to succeed
raise DeadlineExceeded(f"{left:.3f}s left before attempt {attempt + 1}")
try:
# Attempt timeout = min(cap, remaining). wait_for cancels op() on timeout;
# op must pass cancellation through to its DB driver or HTTP client.
return await asyncio.wait_for(op(), timeout=min(PER_ATTEMPT_CAP_S, left))
except (asyncio.TimeoutError, ConnectionError):
if attempt == max_attempts - 1:
raise
pause = random.uniform(0, backoff) # full jitter avoids lockstep retries
if deadline - time.monotonic() - pause < MIN_USEFUL_S:
raise DeadlineExceeded("budget exhausted during backoff")
await asyncio.sleep(pause)
backoff *= 2
async def worker(queue, handle):
while True:
job = await queue.get()
try:
# Producer stored an absolute wall-clock deadline; convert to monotonic.
deadline = time.monotonic() + (job["deadline_epoch"] - time.time())
if deadline - time.monotonic() < MIN_USEFUL_S:
continue # caller has likely given up; drop instead of doing dead work
await call_with_budget(lambda: handle(job), deadline)
finally:
queue.task_done()
An asyncio retry loop where every attempt is capped by the remaining shared budget with jittered backoff, plus a worker that discards queued jobs whose budget fell below the minimum useful time while they waited.
When should you avoid or adapt deadline propagation?
Deadline propagation is a contract, and contracts only hold when every party honours them. A single proxy, SDK, or legacy service that drops the deadline header silently reverts everything downstream to static timeouts. Partial adoption can be harder to debug than none: some hops cancel early, others run to completion, and traces show work continuing after the caller gave up. Roll it out edge-first along whole call paths, and alert when a request arrives with no deadline where one is expected.
Asynchronous boundaries break the model. Once a message sits in a queue, the original caller's deadline no longer describes anyone's wait. Carrying it causes workers to drop jobs that were meant to finish later. Dropping it lets stale work run indefinitely. Often the better fit is a staleness limit, such as "discard if older than N minutes," based on what the job's result is worth.
Fan-out shares one budget across many calls, so a single slow shard can consume it for all of them. Hedged requests cut the tail but add backend load. Returning partial results keeps latency bounded but only works if callers can tolerate gaps.
Tight budgets also turn normal tail latency into visible errors. Set them from observed latency distributions, not from round numbers.
| Scenario | Risk | Recommended handling |
|---|---|---|
| Partial adoption | Silent fallback to static timeouts | Adopt per call path; alert on missing deadlines |
| Queues and background jobs | Wrong lifetime for deferred work | Replace deadline with a staleness limit |
| Fan-out | One shard consumes the budget | Hedge selectively or return partial results |
| Tight budgets | Errors during normal tail latency | Derive budgets from measured percentiles |
What are the alternatives to deadline propagation?
Deadline propagation addresses one problem: work outliving its caller. Other techniques address related problems, and most systems combine several of them.
- Top-down static timeouts. Set each hop's timeout, including its retries, to fit strictly inside its caller's timeout. This needs no protocol changes and works for shallow, stable chains owned by one team. It breaks down when chains grow, because every change to one timeout requires re-checking the callers and callees around it, and it still cannot subtract time already spent upstream.
- Circuit breakers. Stop calling a dependency that is failing or slow, and fail fast for a cooling-off period. This limits load on a struggling service but does not stop work already in flight for callers that have left.
- Bulkheads. Give each dependency its own bounded pool of connections or workers, so one slow dependency cannot exhaust capacity shared with others. This contains the damage but does not shorten the wasted work inside the pool.
- Load shedding and concurrency limits. Reject requests at a service when in-flight work or queue length exceeds a limit. This protects the service from overload but does not know whether a given request's caller is still waiting.
- Retry budgets. Cap retries as a fraction of normal traffic instead of a fixed count per request. This prevents retry amplification but does not bound how long each attempt runs.
- Asynchronous messaging. Replace a synchronous chain with queued work and a later notification. This removes the waiting caller entirely, which is the right choice when the user does not need the result immediately, but it changes the API contract and needs staleness limits instead of deadlines.
These are complements rather than substitutes. Deadline propagation tells each hop how long the work is still worth doing. Circuit breakers, bulkheads, shedding and retry budgets limit how much damage a slow dependency can do. A robust design usually uses deadlines for synchronous paths together with at least a bulkhead and a retry limit.
How do you observe, test, and justify the cost of deadline propagation?
Observe where the budget goes
Log and trace the remaining budget twice per hop: at ingress, when the request arrives, and at egress, just before each downstream call. The difference between those two values is the time the service spent locally. The difference between one hop's egress and the next hop's ingress is network and queueing time. Put both values on spans as attributes so a single trace shows which hop used up the budget.
In metrics, count deadline-exceeded separately from other failures. Use gRPC's DEADLINE_EXCEEDED or a distinct label for HTTP 504s and locally enforced expiries. If these are merged with generic 5xx counts, you lose the signal that tells you a dependency is slow rather than broken. Alert on deadline-exceeded rate per dependency. A rise on one edge points to where the budget is being consumed.
Test it in staging
Inject latency into one dependency and check three things:
- Upstream callers give up at the expected time, not at some leftover static timeout.
- The downstream work is actually cancelled. Look for queries or jobs that keep running after the caller has disconnected.
- Retries stop once the budget is exhausted and do not multiply load.
Weigh the cost
The cost is real. It includes shared middleware, auditing client libraries that ignore context or cancellation, getting each team to adopt it, and ongoing enforcement so new services don't regress.
The return is engineering time. You stop tuning dozens of unrelated timeout constants during incidents, latency is attributed clearly to a hop, and the latency objective is changed in one place.
Roll out edge-first with shared libraries. Each newly covered hop can then be verified on its own, without waiting for every team to finish.
How can deadline propagation save engineering time?
The savings come from removing recurring manual work, not from making any single request faster.
- Fewer values to keep consistent. With static timeouts, every downstream call has its own constant, and correctness depends on all of them fitting together. With propagation, most of those constants become derived values, and the remaining knobs are the edge budget, per-operation floors, and per-attempt caps.
- Shorter incident diagnosis. When ingress and egress budgets are on trace spans, the hop that consumed the time is visible in one trace. Engineers do not have to reconstruct which timeout fired first from logs across several services.
- Less retuning when the architecture changes. Adding a hop, moving a call, or tightening the latency objective does not require renegotiating timeouts between teams. The edge budget changes, and every hop inherits the new value.
- Fewer retry storms to clean up. Retries that draw from a shared budget stop on their own, which removes a common cause of incidents that outlast the original slowdown.
- Shared code instead of per-service logic. Budget parsing, clamping, outbound headers and database timeouts live in one middleware library, so a fix or improvement lands everywhere at once.
Illustrative estimate (assumed numbers, not a measurement). Assume 30 services, each making an average of 3 synchronous downstream calls, each with its own timeout and retry count. That is 90 timeout constants and 90 retry settings whose combinations must fit inside their callers. After adopting propagation with a single retrying layer, the assumed configuration surface is one edge budget per entry point plus a minimum useful time and per-attempt cap for each operation type. The exact reduction depends on your system, but the shift is from checking pairwise consistency across teams to reviewing a small set of values in one place.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to microservice timeout propagation, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.