Request Hedging in Microservices: Cutting p99 Without Overload

What request hedging is and why tail latency compounds in fan-out

Request hedging means sending a request to one replica and, if no response arrives within a short delay, sending a duplicate to a second replica. The caller uses whichever response returns first and cancels the other. The goal is to cut the long tail of slow responses, not to improve the median. Most slow requests are not slow because the work is hard. They are slow because one replica hit a garbage-collection pause, a noisy neighbour, a cold cache or a queue spike, and a different replica would have answered quickly.

This matters most in fan-out. When a request handler calls many backends in parallel and waits for all of them, the slowest call sets the response time. A latency spike that is rare for any single service becomes common for the user-facing request.

Illustrative estimate. Assume each backend call independently has a 1% chance of exceeding that backend's p99, and the handler waits for all N calls. The probability that at least one call is slow is 1 − 0.99^N:

• N = 1: 1%
• N = 10: about 9.6%
• N = 50: about 39.5%
• N = 100: about 63.4%

At a fan-out of 100, the backend's p99 becomes the typical user experience. Real slowdowns are often correlated, so actual figures will differ. The direction of the effect holds.

Hedging helps most for read-heavy, fan-out services backed by interchangeable replicas, where any healthy copy can answer a request.

How hedged requests work compared with plain retries

A hedged call starts like any other request. The client sends the primary to one replica and starts a timer. If no response arrives before the hedge delay expires, it sends a duplicate to a different replica. Whichever response comes back successfully first is returned to the caller, and the client cancels the outstanding request so the slower replica stops spending work on it.

Copyt=0        client ──req──▶ replica A   (stalled: GC pause, cold cache)
t=hedge    client ──req──▶ replica B
t=hedge+δ  client ◀──ok─── replica B   ← winner, returned to caller
           client ──cancel─▶ replica A

A plain retry waits for an error or a timeout. Timeouts are usually set well above p99 so that normal slow calls are not abandoned, which means a retry rarely rescues a slow request in time. It only limits how bad the worst case gets. Hedging fires at a point closer to the latency the service normally achieves, and it does not discard the first attempt. Both requests race.

Distributed data stores use the same idea under the name speculative retry. When a read to one replica is slow, the coordinator issues the same read to another replica holding the same data.

Hedging assumes interchangeable replicas. If only one node can serve a key, or replicas return different results, there is nothing to race.

Copy        +--------+          +-----------+               +-----------+
        | Client |          | Replica A |               | Replica B |
        +--------+          +-----------+               +-----------+
            |                     |                           |
t=0         |---- primary req --->|                           |
            |                     |  stalled: GC pause,       |
  ^         |                     |  cold cache               |
  |         |                     |                           |
  hedge     |                     |                           |
  delay     |                     |                           |
  |         |                     |                           |
  v         |                     |                           |
t=hedge     |---- hedged duplicate--------------------------->|
            |                     |                           |
t=hedge+d   |<-- OK (winner) ---------------------------------|
caller <----|                     |                           |
            |                     |                           |
            |---- cancel -------->|                           |
            |                     X  work stopped             |
            |                                                 |
            v                                                 v

A hedged request: the client sends the primary to replica A, sends a duplicate to replica B once the hedge delay expires, returns B's first successful response to the caller, and cancels the outstanding request on A.

Choosing the hedge delay and estimating the extra load

The hedge delay is how long the client waits for the first response before it sends a second copy. A common starting point is a high percentile of recent latency for that specific method and backend, such as p95. The reasoning is simple. Most requests finish before that point and never get duplicated. The slowest few are the ones a second attempt is most likely to beat.

The delay should come from live data, such as a rolling latency histogram per endpoint, and not from a constant in config. Latency shifts with deploys, traffic mix, cache warmth and time of day. A delay that matched p95 last month may now fire on 20% of requests, or almost never.

Illustrative estimate, not measured. Assume a 1,000 RPS endpoint with a stable distribution and a p95 trigger. About 5% of calls get hedged, which adds roughly 50 RPS. Cancellation is not instant, so some calls that lose the race still do work before the cancel arrives. Budget a little above 5%.

A delay set too low duplicates healthy requests and spends capacity for no gain. A delay set too high fires after the user has already waited, so it barely moves the tail. Because both failure modes drift over time, track the hedge rate and the hedge win rate, and retune the trigger as conditions change.

Copyimport collections
import statistics

class HedgeDelay:
    """Derives the hedge delay from a rolling window of observed latencies."""

    def __init__(self, percentile=95, window=5000, floor_ms=5.0, ceiling_ms=500.0):
        self.percentile = percentile          # trigger point, e.g. 90, 95, 99
        self.samples = collections.deque(maxlen=window)  # recent latencies only
        self.floor_ms = floor_ms              # never hedge healthy fast calls
        self.ceiling_ms = ceiling_ms          # cap so a hedge still helps the user
        self.requests = self.hedges = self.hedge_wins = 0

    def record(self, latency_ms, hedged=False, hedge_won=False):
        # Record the latency of the first attempt, so hedge wins do not
        # drag the distribution down and make the trigger fire more often.
        self.samples.append(latency_ms)
        self.requests += 1
        self.hedges += hedged
        self.hedge_wins += hedge_won

    def delay_ms(self):
        if len(self.samples) < 100:
            return self.ceiling_ms            # too little data: hedge rarely
        cuts = statistics.quantiles(self.samples, n=100)  # 99 cut points
        value = cuts[self.percentile - 1]
        return min(max(value, self.floor_ms), self.ceiling_ms)

    def stats(self):
        # Hedge rate should sit near (100 - percentile)%; drift means retune.
        # A low win rate means hedges add load without cutting the tail.
        n = max(self.requests, 1)
        return {
            "hedge_rate": self.hedges / n,
            "hedge_win_rate": self.hedge_wins / max(self.hedges, 1),
            "current_delay_ms": self.delay_ms(),
        }

# One tracker per method and backend, never one global constant.
trackers = {("GetOrder", "orders-v2"): HedgeDelay(percentile=95)}

A per-endpoint tracker that recomputes the hedge delay from a rolling latency window, clamps it between a floor and a ceiling, and reports hedge rate and hedge win rate to guide retuning.

Implementing hedged requests in gRPC and application code

Configuring hedging in gRPC

gRPC defines hedging in the service config as a hedgingPolicy on a methodConfig entry, as an alternative to retryPolicy. A method can have one or the other, not both. The policy has three main fields. maxAttempts caps total copies, including the original. hedgingDelay sets the gap before each additional copy is sent. nonFatalStatusCodes lists the codes, such as UNAVAILABLE, that trigger the next hedge immediately. Any other error code is treated as final: it cancels outstanding attempts and returns to the caller.

Pair the policy with a top-level retryThrottling block (maxTokens, tokenRatio). This is a client-side token bucket. Failures drain tokens and successes refill them, and hedging stops once the bucket falls below half. That way, a struggling backend sees less extra traffic instead of more.

In the separate JSON example, the hedging delay is set near the measured p95, and maxAttempts is kept at 2 or 3.

Hand-rolled hedging

Outside gRPC's built-in support, the pattern is short. Derive a cancellable context from the caller's context, which carries the deadline. Start the first call and arm a timer. If the timer fires before a response arrives, start a second call with the same context. Take the first successful result, then cancel the context so every losing attempt is aborted.

Propagation and its limits

Cancellation only saves work if it travels downstream. Pass the context or deadline to every outbound call, so gRPC can propagate the deadline and send stream resets. Servers must also check for cancellation. Even then, cancellation is best-effort: a query already running in the database, or code that ignores the context, will finish anyway.

Hedging, throttling and cancellation support differ across language implementations, libraries and meshes. Verify how your specific version behaves before relying on the spec.

Copy{
  "methodConfig": [
    {
      "name": [{ "service": "inventory.v1.InventoryService", "method": "GetItem" }],
      "timeout": "0.500s",
      "hedgingPolicy": {
        "maxAttempts": 3,
        "hedgingDelay": "0.040s",
        "nonFatalStatusCodes": ["UNAVAILABLE"]
      }
    }
  ],
  "retryThrottling": {
    "maxTokens": 10,
    "tokenRatio": 0.1
  }
}

A gRPC service config that hedges an idempotent read: up to 3 attempts, 40 ms apart (a placeholder near the measured p95), with retry throttling that stops hedges when failures drain the token bucket.

Safety rules: idempotency, load budgets and throttling

Every hedge is a duplicate request your backend did not ask for, so the first rule is about what you hedge at all. Reads and truly idempotent operations are safe because running them twice changes nothing. Non-idempotent mutations, such as charging a card, appending to a ledger or incrementing a counter, must never be hedged. Both copies can succeed, and cancelling the loser does not undo work the server has already committed. If a write path genuinely needs hedging, attach an idempotency key so the server can detect the duplicate and return the stored result. That requires a deduplication store, a retention window and careful handling of two requests with the same key arriving at the same moment. Each of those adds storage and code you now have to maintain.

Second, cap hedges explicitly. Limit them to a fixed share of traffic, for example 5%, or draw from a token bucket that refills with successful primary responses. When the bucket is empty, the client waits for the primary instead of hedging. Turn hedging off automatically when error rates or backend saturation signals rise. A slow backend is usually an overloaded one, and extra copies make that worse.

Third, hedge at one layer only. Hedges at several hops multiply the same way stacked retries do.

Illustrative estimate (assumed numbers, not a benchmark): worst-case calls = (max attempts per hop)^(hops). With 2 attempts at each of 3 hops, that is 2³ = 8 calls to the deepest service for a single user request. Pick the hop closest to the latency source and turn hedging off everywhere else.

Copyimport threading

class HedgeBudget:
    """Token bucket that limits hedges to a share of successful traffic."""

    def __init__(self, hedge_ratio=0.05, max_tokens=10.0, max_error_rate=0.05):
        self.hedge_ratio = hedge_ratio        # about 1 hedge per 20 successful primaries
        self.max_tokens = max_tokens          # caps bursts after a long quiet period
        self.max_error_rate = max_error_rate  # above this, hedging switches off
        self.tokens = max_tokens
        self.error_rate = 0.0                 # exponentially weighted moving average
        self.lock = threading.Lock()

    def record_primary(self, ok: bool) -> None:
        with self.lock:
            # Smooth the error signal so one failure does not flip the switch.
            self.error_rate = 0.95 * self.error_rate + 0.05 * (0.0 if ok else 1.0)
            if ok:
                # Only successful primaries refill the bucket, so a failing
                # backend stops earning hedges.
                self.tokens = min(self.max_tokens, self.tokens + self.hedge_ratio)

    def try_hedge(self, idempotent: bool, backend_saturated: bool) -> bool:
        # Never hedge non-idempotent mutations, whatever the budget says.
        if not idempotent:
            return False
        with self.lock:
            # Back off automatically when the backend shows signs of overload.
            if backend_saturated or self.error_rate > self.max_error_rate:
                return False
            if self.tokens < 1.0:
                return False  # budget spent: wait for the primary instead
            self.tokens -= 1.0
            return True

# Usage: create one budget per downstream service, at the single hedging layer.
# if budget.try_hedge(idempotent=True, backend_saturated=queue_depth > 100):
#     send_hedge()

A thread-safe hedge budget that refills only on successful primary responses, refuses to hedge non-idempotent calls, and turns hedging off when the smoothed error rate or a saturation signal rises.

When to use hedging and when to avoid it

Hedging pays off when two conditions hold. The call must be safe to send twice: idempotent reads, or writes with an idempotency key the server enforces. The slowness must also be independent across replicas. A GC pause, a noisy neighbour on one host, or one instance that got an unlucky share of traffic makes that replica slow while the others stay fast. A second attempt to a different replica then has a good chance of returning quickly. That independence is the whole mechanism.

When slowness is shared, hedging only adds load. A slow shared database, a hot partition that every replica must hit, or a degraded downstream dependency slows every attempt equally. The hedge adds work to the component that is already struggling. The same applies during overload or brownouts: extra attempts feed the queueing that caused the latency. Hedging can also hide a chronically bad replica or a GC configuration problem. Your p99 looks fine, so nobody investigates, and the root cause stays in place while you keep paying for duplicate work.

Observability, cost and engineering time

Tag every attempt with whether it is the original or a hedge, and record which one returned first. Without that tag, hedges look like ordinary traffic and you cannot tell whether they help.

Measure the benefit by enabling hedging on a fraction of traffic and comparing the cohorts over the same period. Before-and-after charts mix the effect of hedging with deploys and changes in load.

Illustrative estimate (assumed numbers, not a benchmark): extra RPS = base RPS × hedge rate. Assume 10,000 RPS and a 5% hedge rate. That adds 500 RPS. If the win rate is 30%, about 150 of those hedges per second shorten a response. The other 350 cost capacity and returned nothing, which is why cancellation plumbing matters.

The costs go beyond capacity. You need idempotency work, cancellation that propagates end to end, and delay retuning as latency drifts. You also need to re-verify behaviour when client libraries change.

Used narrowly, hedging saves engineering time. A config-level policy on a few read paths can cut the tail sooner than tracking down every source of replica variance. Alert when the hedge rate rises. A rising rate means hedges are masking a fault that still needs fixing.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to request hedging microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.