Deadline Propagation in Microservices: Timeout Budgets That Shrink

Why per-hop static timeouts cause cascading timeouts

Consider a synchronous chain where each hop sets its own timeout without regard to the others: the edge waits 2 s for service A, A waits 3 s for B, and B allows itself 5 s against its database. These numbers look reasonable one at a time, but together they are inverted. Each callee is allowed to run longer than its caller will wait.

Illustrative estimate (assumed numbers, not a benchmark). Assume B's database degrades and every query takes 4 s.

• At t = 2 s the edge times out and returns an error to the user.
• A keeps waiting until t = 3 s, then gives up.
• B finishes at t = 4 s and sends a response nobody is waiting for.

Wasted work per request = B_finish − edge_deadline = 4 − 2 = 2 s of B's time spent after the caller had already given up.

The cost is concurrency, not only CPU. By Little's law, in-flight requests = arrival_rate × time_held. At an assumed 100 req/s, B holds 100 × 4 = 400 concurrent requests. If work stopped at the edge's 2 s deadline, B would hold 100 × 2 = 200. The extra 200 slots tie up connections, threads and database sessions during exactly the period when the database is struggling. Client retries make this worse, because each retry starts a new chain while the abandoned ones are still running. That is how a slow dependency turns into cascading timeouts.

The alternative this article argues for is to set one deadline at the edge and pass it down the chain. Each hop sees how much of that deadline remains and stops when it runs out. This matters most to backend and platform engineers who run synchronous call chains of three or more hops.

CopySTATIC PER-HOP TIMEOUTS (illustrative estimate: every DB query takes 4 s)

+--------+  waits 2 s  +-----------+  waits 3 s  +-----------+ 5 s +------+
|  Edge  |------------>| Service A |------------>| Service B |---->|  DB  |
+--------+             +-----------+             +-----------+     +------+

 t(s)  0         1         2         3         4
       |---------|---------|---------|---------|
 Edge  [===================X  times out, user gets error
 A     [=============================X  gives up
 B     [=======================================]  reply discarded
                           |<-- 2 s wasted --->|

 B in flight at 100 req/s:  100 x 4 s = 400   vs   100 x 2 s = 200
 Retries start new chains while abandoned ones still hold B's slots

SHRINKING DEADLINE: one deadline set at the edge, passed down every hop

+--------+  deadline   +-----------+  remaining  +-----------+ rem +------+
|  Edge  |------------>| Service A |------------>| Service B |---->|  DB  |
+--------+  now + 2 s  +-----------+  < 2 s      +-----------+     +------+

 Every hop stops at t = 2 s; B frees its slot instead of holding it to 4 s

With inverted static timeouts (2 s, 3 s, 5 s), B keeps working 2 s after the edge has given up in this illustrative estimate, while a single deadline set at the edge stops every hop at t = 2 s.

What deadline propagation is: deadlines vs. timeouts

A timeout is a relative duration, such as "wait up to 500 ms for this call." A deadline is an absolute point in time, such as "this work is worthless after 12:00:00.750." When each hop sets its own timeout, it starts a fresh clock, so a hop has no way to know how much of the caller's patience is already used up. A deadline fixes one end point for the whole request. Each hop computes remaining = deadline − now, so the allowance shrinks automatically as time passes, and no hop can grant more time than the original caller has left.

Sending the absolute timestamp across the network depends on clocks agreeing. If a server's clock runs 200 ms behind the caller's, it believes it has 200 ms more than it does. For this reason, many systems, gRPC among them, send the remaining duration on the wire. The receiver adds that duration to its own clock to get a local deadline, so the comparison happens against a single clock and skew between machines drops out. The trade-off is that a relative value ignores time spent in transit and queues. A sender can subtract headroom to cover that, but the result is still an approximation.

Illustrative estimate (assumed numbers, not a benchmark):

sent = (deadline − now) − headroom

Assume 1,000 ms remaining when the caller sends, 50 ms of headroom, and 30 ms of actual transit. The caller sends 950 ms. The receiver starts with 950 ms and the true remaining time is 970 ms, so the receiver errs early by 20 ms.

How it works in gRPC and over plain HTTP

gRPC: deadlines built into the protocol

In gRPC, the client attaches a deadline to the call. On the wire it travels as a remaining-time value in the grpc-timeout request header, not as an absolute timestamp. The receiving server converts that value back into a local deadline, so clock skew between hosts doesn't matter. Transit time does, as the table above notes. The server's request context carries this deadline. Any outbound gRPC call made with that same context sends the reduced remaining time downstream, so each hop's budget shrinks automatically. When time runs out, the call fails with DEADLINE_EXCEEDED, and a well-behaved server stops work it can no longer deliver.

The catch is that propagation only happens if your code passes the incoming context to outbound calls. If a handler creates a fresh background context, for example to "simplify" a helper function, the chain breaks without any error. The downstream call gets no deadline or a new local one, and the budget stops shrinking.

Plain HTTP: you define the convention

HTTP has no single universal deadline header, so teams define their own. A common pattern is a header such as X-Team-Deadline-Ms that carries the remaining milliseconds. Middleware handles both ends:

• Ingress: read the header, subtract a small safety margin, and create a context or cancellation token with that timeout.
• Egress: a wrapped HTTP client computes the time left on the current context and writes it into the header of every outbound request.

Keeping both steps in shared middleware means handler code never touches the header directly.

Copyimport { createServer, IncomingMessage, ServerResponse } from "node:http";
import { AsyncLocalStorage } from "node:async_hooks";

const HEADER = "x-team-deadline-ms"; // team-defined convention, not an HTTP standard
const SAFETY_MARGIN_MS = 20;          // absorb transit time and local overhead
const DEFAULT_BUDGET_MS = 2000;       // used only when the caller sent no budget

// Holds this request's absolute local deadline (ms since epoch) across awaits.
const deadlineStore = new AsyncLocalStorage<number>();

// Ingress: convert the remaining-ms header into a local deadline for this request.
function withDeadline(handler: (req: IncomingMessage, res: ServerResponse) => Promise<void>) {
  return (req: IncomingMessage, res: ServerResponse) => {
    const raw = Number(req.headers[HEADER]);
    const budget = Number.isFinite(raw) && raw > 0 ? raw : DEFAULT_BUDGET_MS;
    const deadline = Date.now() + budget - SAFETY_MARGIN_MS;
    deadlineStore.run(deadline, () =>
      handler(req, res).catch((err) => {
        const timedOut = err?.name === "TimeoutError" || err?.name === "DeadlineExceeded";
        res.writeHead(timedOut ? 504 : 500).end();
      })
    );
  };
}

// Egress: every outbound call forwards the *remaining* budget and enforces it locally.
export async function callDownstream(url: string, init: RequestInit = {}) {
  const deadline = deadlineStore.getStore() ?? Date.now() + DEFAULT_BUDGET_MS;
  const remaining = Math.floor(deadline - Date.now());
  if (remaining <= 0) throw Object.assign(new Error("budget spent"), { name: "DeadlineExceeded" });
  const headers = new Headers(init.headers);
  headers.set(HEADER, String(remaining));
  return fetch(url, { ...init, headers, signal: AbortSignal.timeout(remaining) });
}

createServer(withDeadline(async (_req, res) => {
  const upstream = await callDownstream("");
  res.writeHead(200, { "content-type": "application/json" }).end(await upstream.text());
})).listen(3000);

Node.js middleware that turns a team-defined X-Team-Deadline-Ms header into a per-request deadline at ingress, then forwards the shrinking remaining budget on outbound fetch calls. It uses AbortSignal.timeout to stop waiting on downstream calls once the budget runs out.

How to implement it: edge budgets, retries, and server-side enforcement

Choose the edge budget

Start from the user-facing latency SLO. Subtract headroom for the client's network path, TLS, and response serialization, which the backend never measures. The edge stamps the remainder on the request as its deadline.

Illustrative estimate (assumed numbers, not a benchmark): SLO 800 ms, headroom 100 ms, so budget = SLO − headroom = 700 ms. A sequential auth call capped at 100 ms leaves 600 ms. A parallel fan-out to three services costs max(child latencies), not their sum. Each child can therefore get 600 ms minus a 50 ms reserve for merging, or 550 ms.

Full or capped. If a hop gets the full remaining budget, a slow early hop can starve every later one. A capped slice, min(remaining, cap), protects downstream steps but can fail requests that would have succeeded. Cap hops early in a chain and give the last hop whatever is left.

Retry inside the deadline

• Check before each attempt. If the remaining time is less than a typical attempt, fail immediately. Starting an attempt that cannot finish only adds load.
• Fit the backoff into the budget. Sleep a jittered interval only if the time left after sleeping still covers an attempt.
• Bound each attempt by both limits. Set the per-attempt timeout to min(per_attempt_cap, remaining).
• Retry at one layer only. Three layers that each try three times can produce 27 calls at the bottom.

Enforce on the server

Check the deadline before expensive work such as database queries, enqueues, or fan-out. If it has passed, return a deadline-exceeded error instead of computing a result no caller will read. Propagate cancellation to child calls. Where the driver supports it, map the remaining time to the database or cache client's statement or operation timeout.

Copyimport random, time

class DeadlineExceeded(Exception): pass

def remaining(deadline):
    return deadline - time.monotonic()  # seconds left on the propagated deadline

def call_with_retries(call, deadline, attempts=3, per_attempt_cap=0.2,
                      typical_attempt=0.05, base_backoff=0.02):
    for attempt in range(attempts):
        left = remaining(deadline)
        if left < typical_attempt:  # don't start an attempt that cannot finish
            raise DeadlineExceeded("not enough budget for another attempt")
        try:
            # Per-attempt timeout is capped and never outlives the overall deadline.
            return call(timeout=min(per_attempt_cap, left))
        except TimeoutError:
            if attempt == attempts - 1:
                raise
        # Full jitter, but only sleep if an attempt still fits afterwards.
        pause = random.uniform(0, base_backoff * 2 ** attempt)
        if remaining(deadline) - pause < typical_attempt:
            raise DeadlineExceeded("backoff would exhaust the budget")
        time.sleep(pause)

def handle(request, cur):
    # Server side: check the deadline before expensive work and fail fast.
    left = remaining(request.deadline)
    if left <= 0:
        raise DeadlineExceeded("deadline passed before query")
    # Map the remaining budget to PostgreSQL's per-transaction statement timeout (ms).
    cur.execute("SELECT set_config('statement_timeout', %s, true)",
                (str(int(left * 1000)),))
    cur.execute("SELECT id, total FROM orders WHERE customer_id = %s",
                (request.customer_id,))
    return cur.fetchall()

A retry loop that skips attempts and backoff sleeps that cannot fit in the remaining deadline and bounds each attempt by min(per-attempt cap, remaining), plus a server handler that rejects expired requests before querying and passes the remaining budget to PostgreSQL as statement_timeout.

When to use it and when to avoid it

Where it fits

Deadline propagation pays off on synchronous, latency-sensitive paths with a user-facing SLO. A checkout or search request fans out to several services, and each one blocks the caller. The deeper the chain or the wider the fan-out, the more work a static timeout wastes after the user has already given up. It also matters for shared dependencies such as a primary database or an auth service. During an incident, abandoned requests that keep running are what push those dependencies over the edge.

Where to drop it

Stop propagating at asynchronous boundaries. When a request enqueues a job, the producer's remaining 300 ms says nothing about when a worker should finish. If the work is only useful within a window, convert the deadline into a message expiry or TTL that holds an absolute timestamp. Consumers then check it before starting and discard expired messages instead of processing stale work. A password-reset email sent an hour late is worse than none.

Treat non-idempotent side effects carefully. Cancellation is not rollback. A payment call cancelled mid-flight may still have charged the card. Writes, payments, and outbound messages need idempotency keys so a retry is safe. Alternatively, shield the critical section from cancellation and let it finish once it has started.

The premature-failure trade-off

An aggressive edge deadline leaves too little time for deep hops. They fail fast with deadline-exceeded errors on requests that would have succeeded slightly later. You have turned a latency problem into an availability problem, so set edge budgets from measured tail latency across the whole chain.

Alternatives and complementary techniques

Deadline propagation answers one question: how much time is left for this request? Most neighbouring techniques answer different questions, so they sit alongside it rather than replacing it.

Independently tuned per-hop timeouts are the only true alternative. They work when chains are shallow and callers are few. Once a service sits behind several callers with different latency expectations, no single number fits all of them.

Caps and floors are worth having, but they bring back the static numbers that propagation was meant to remove. A floor of, say, 50 ms is right until a downstream store gets slower or a new hop is inserted, and then it silently rejects valid work. Treat these values like SLOs: tie them to measured percentiles and review them when latency profiles shift.

Impact on development cost and engineering time

That review is one recurring cost among several, and teams should plan for all of them up front.

Middleware is a product, not a patch. Someone has to build and maintain the interceptors that read, shrink, and forward deadlines for every protocol and language in the fleet. Partial adoption fails silently. A single hop that drops the header turns every service behind it into an unbounded call, and nothing alerts. Enforce propagation with shared client libraries, lint rules that flag raw HTTP clients, and integration tests that assert the deadline arrives downstream. Convention alone does not hold.

Errors point to the wrong place. A deadline fails where time ran out, not where it was spent. Without extra data, on-call engineers start at the last service, which is often the healthy one that received a nearly empty budget.

Per-hop budget telemetry fixes this. Each span or log line records three fields, for example:

• deadline.remaining_in_ms: the budget left when the request arrived.
• deadline.local_ms: the time this hop spent itself.
• deadline.remaining_out_ms: the budget passed to each downstream call.

With these fields you can tell two cases apart. "Deadline exceeded here" means the hop had a reasonable incoming budget and spent it. "Expired on arrival" means remaining_in_ms was already near zero. Reading local_ms back up the chain points to the slow hop directly, so on-call starts there.

During incidents, this enforcement also saves capacity. Servers refuse work whose caller has already given up, so CPU and connections go to requests that can still succeed.

Copyimport time
from opentelemetry import trace

tracer = trace.get_tracer("checkout-service")
MIN_USEFUL_MS = 20      # below this, no downstream call can realistically succeed
RESPONSE_RESERVE_MS = 10  # keep some budget to serialize and return the reply

def handle(request, do_local_work, call_inventory):
    arrived = time.monotonic()
    # Budget the caller says is left. Missing header means an upstream hop broke the chain.
    remaining_in = int(request.headers.get("x-deadline-remaining-ms", "1000"))

    with tracer.start_as_current_span("checkout.handle") as span:
        span.set_attribute("deadline.remaining_in_ms", remaining_in)

        if remaining_in < MIN_USEFUL_MS:
            # Expired on arrival: the slow hop is upstream, not here.
            # Refuse so CPU and connections go to requests that can still succeed.
            span.set_attribute("deadline.outcome", "expired_on_arrival")
            raise TimeoutError("deadline expired on arrival")

        do_local_work(request)
        local_ms = int((time.monotonic() - arrived) * 1000)
        remaining_out = remaining_in - local_ms - RESPONSE_RESERVE_MS
        span.set_attribute("deadline.local_ms", local_ms)
        span.set_attribute("deadline.remaining_out_ms", remaining_out)

        if remaining_out < MIN_USEFUL_MS:
            # Deadline exceeded here: this hop had a usable budget and spent it.
            span.set_attribute("deadline.outcome", "exceeded_here")
            raise TimeoutError("deadline exceeded in checkout-service")

        # Forward the shrunk budget so the next hop records its own remaining_in_ms.
        return call_inventory(
            request,
            headers={"x-deadline-remaining-ms": str(remaining_out)},
        )

A request handler that records incoming budget, local time spent, and budget passed downstream as OpenTelemetry span attributes, and labels failures as either expired on arrival or exceeded at this hop.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to deadline propagation microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.