What is request hedging and why does tail latency compound under fan-out?

Request hedging means sending a second copy of a request to another replica when the first has not answered within a set delay. The client uses whichever response arrives first and cancels the other. The goal is not a lower average latency. The goal is to stop a few slow replicas from setting the latency your users actually feel.

Why p99 matters more than the mean

In a single service, a slow p99 affects one request in a hundred. In a microservice architecture, one user-facing request often waits on many backend calls in parallel. It cannot finish until the slowest of them returns, so the tail of each backend becomes the typical latency of the whole request.

Illustrative estimate: assume each backend call has a 1% chance of being slow, and the chances are independent. A request that fans out to 100 backends avoids every slow call with probability 0.99^100, or about 0.37. That means roughly 63% of requests hit at least one slow call. A 1-in-100 event at the leaf becomes the common case at the root.

What hedging targets

Hedging works when slowness is transient and specific to one replica. Typical causes are a garbage-collection pause, a noisy neighbor, a cold cache, or a brief network hiccup. In those cases a different replica will probably answer quickly. Hedging does not help when the slowness is systemic, such as an overloaded database or a slow query, because every replica is slow at once.

The teams that benefit most run latency-sensitive, read-heavy services that fan out to replicated backends where requests are safe to duplicate.

How does request hedging work compared with retries and broadcast requests?

A hedged call starts like any other: the client sends the request to one replica and starts a timer set to the hedge delay. If a response arrives before the timer fires, nothing else happens, which is the case for most requests. If the timer fires first, the client sends a duplicate to a different replica. Whichever response arrives first is returned to the caller, and the client cancels the other in-flight attempt so the slower replica stops spending work on an answer nobody will read.

t=0 client ──► replica A (timer starts) t=delay no reply ──► replica B (hedge sent) t=x B replies first ──► return; cancel A

The difference from a retry is the trigger. A retry waits for a failure or a timeout, so a slow but healthy replica can still hold the caller for the full timeout. A hedge fires on slowness alone, while the first attempt is still running.

Strategy When extra requests are sent Extra load Tail latency effect
Retry After an error or timeout Low, only on failures Little help when requests are slow but not failing
Hedge After the hedge delay with no reply Small, only on slow requests Cuts off the slow tail
Broadcast Immediately, to all replicas Multiplies load by replica count Best case, at a high cost

Routing matters. Send the duplicate to a different replica, because the original one may be the cause of the delay. Prefer a replica in the same zone so the hedge doesn't add cross-zone network time. If only one healthy replica exists, skip hedging, since a second request to the same struggling host adds load without improving the odds.

                 Zone 1 (client's zone)               Zone 2
+--------+     +-----------+     +-----------+     +-----------+
| Client |     | Replica A |     | Replica B |     | Replica C |
+---+----+     +-----+-----+     +-----+-----+     +-----+-----+
    |                |                 |                 |
t=0 |---- request -->|                 |                 |
    | (timer starts) |                 |                 |
    |                | still working   |                 |
t=d |----------- hedge --------------->|                 |
    |  (no reply yet; pick another     |   not chosen:   |
    |   replica in the same zone)      |   cross-zone    |
t=x |<---------- first reply ----------|                 |
    |---- cancel --->|                 |                 |
    |                X                 |                 |
    v                v                 v                 v
 return B's reply to the caller; A stops work

 Only one healthy replica?
 +--------+          +-----------+
 | Client |--------->| Replica A |   no hedge sent
 +--------+          +-----------+

A hedged request: the client sends to replica A, and when the hedge delay passes with no reply it sends a duplicate to replica B in the same zone. It returns B's first reply and cancels A, and it skips hedging when only one healthy replica exists.

When should you use request hedging, and when should you avoid it?

Hedging pays off only when slowness is local to one replica and short-lived. A second copy then has a good chance of landing on a host that isn't stalled. If every replica is slow for the same reason, the duplicate just waits in a different place and adds load.

Cause of slowness Does hedging help? Why
GC pause on one instance Yes Other replicas are unlikely to pause at the same moment
Noisy neighbor on shared hardware Yes Contention is tied to one host
Cold cache after a restart or deploy Yes Warm replicas answer from memory
Queue buildup on a single replica Yes The hedge skips that queue
Slow shared dependency (database, downstream API) No Both copies wait on the same bottleneck
Cluster-wide overload No, harmful Extra requests deepen the overload
Inherently expensive query No Every replica takes just as long

Idempotency is a hard requirement

A hedged request can run to completion on two servers. Cancellation is best-effort, and the first copy may already have committed. Limit hedging to reads or to writes protected by an idempotency key that the server deduplicates. Without one, a hedged payment, insert or publish can charge twice, create duplicate rows or emit duplicate messages.

Hedging can hide real faults

Because hedging smooths over slow replicas, it can mask the problems that cause them. A replica with a failing disk, a service with poor GC settings, or a hot shard can all look healthy in p99 latency while they quietly absorb the extra traffic. Track which backend lost each hedge race, and alert when one host or shard keeps losing.

How do you choose a hedge delay and budget the extra load?

Set the hedge delay near a high percentile of observed latency, typically p95. The delay determines how many requests get a second copy, so it is the main lever on cost. A delay at p95 means only the slowest 5% of requests are hedged, and those are the requests that cause tail latency.

Formula: extra requests/sec ≈ baseline RPS × (1 − q), where q is the percentile chosen as the delay. This assumes at most one hedge per request and ignores work saved by cancelling the losing request.

Illustrative estimate (assumed numbers, not a benchmark): assume 10,000 RPS baseline and a maximum of one hedge per request.

Delay set at Fraction hedged Extra RPS Added load
p50 50% 5,000 +50%
p90 10% 1,000 +10%
p95 5% 500 +5%
p99 1% 100 +1%

A p50 delay roughly behaves like broadcasting half your traffic. A p99 delay costs little, but it fires too late to help p99 itself.

Static versus adaptive delays

A static delay is easy to reason about, but it drifts out of date when deploys, traffic mix or dependencies change latency. If latency improves, the static delay fires too rarely to help. If latency regresses, it fires more often and raises load during a period when the service is already under stress.

An adaptive delay is computed from a rolling percentile and tracks these changes. It needs a percentile sketch per method. It can also oscillate: hedges add load, the added load raises latency, and higher latency changes the delay. To damp this, use smoothing windows, a floor and ceiling on the delay, and a hard cap on the hedge rate.

Capacity planning

Provision for the hedge budget as if it were organic traffic. Size backends for the extra load at the worst case your rate cap allows, not the average. Hedges arrive in bursts at the moments latency is already elevated.

How do you configure a gRPC hedging policy?

gRPC moves hedging out of application code and into the service config, a JSON document attached to a channel either by the client or through name resolution. Each methodConfig entry names one or more methods by service and method name, and can carry a hedgingPolicy with three fields:

  • maxAttempts: the total number of copies, including the original. The specification caps this at a small number, and clients silently clamp higher values, so the cap acts as the upper bound on fan-out multiplication.
  • hedgingDelay: how long the client waits before sending each additional copy, written as a duration string such as "0.05s". A zero delay turns hedging into a broadcast.
  • nonFatalStatusCodes: the status codes that let outstanding copies keep running. Any other code, along with any success, ends the call and cancels the remaining attempts.

The example config that follows applies these fields to a single read-only method.

A method config can hold a hedgingPolicy or a retryPolicy, never both. This rule keeps attempt counts bounded and predictable. The top-level retryThrottling block applies to both policies. It uses a token bucket that failures drain and successes refill, and once the bucket falls below half full, the client stops sending extra attempts to that server. This is the client-side form of the rate cap described in the previous section.

Not every gRPC language implementation supports hedging. Some parse the policy and then ignore it. Confirm that your implementation supports it before you count on it, and verify with metrics that hedges are actually being sent.

When hedging works in the library, it saves real effort. Cancellation, throttling, and attempt accounting come with the library, so you don't have to rebuild them in each service.

{
  "methodConfig": [
    {
      "name": [
        { "service": "catalog.v1.ProductService", "method": "GetProduct" }
      ],
      "timeout": "0.5s",
      "hedgingPolicy": {
        "maxAttempts": 3,
        "hedgingDelay": "0.05s",
        "nonFatalStatusCodes": ["UNAVAILABLE", "RESOURCE_EXHAUSTED"]
      }
    },
    {
      "name": [
        { "service": "catalog.v1.ProductService", "method": "UpdateProduct" }
      ],
      "retryPolicy": {
        "maxAttempts": 2,
        "initialBackoff": "0.1s",
        "maxBackoff": "1s",
        "backoffMultiplier": 2,
        "retryableStatusCodes": ["UNAVAILABLE"]
      }
    }
  ],
  "retryThrottling": {
    "maxTokens": 10,
    "tokenRatio": 0.1
  }
}

A gRPC service config that hedges the read-only GetProduct method, gives the write method its own retry policy, and uses one retryThrottling block to limit extra attempts for both.

How do you implement hedging safely: cancellation, throttling and layered retries

When you build hedging yourself, cancellation is what makes it affordable. Once one attempt returns, cancel the other one. That cancellation has to travel through every hop. In Go, both attempts share a child context derived from the caller's deadline. The first response triggers cancel(), and every downstream call must take that context instead of starting a fresh one. If any service in the chain drops the context, the losing request keeps querying databases and calling dependencies. Your capacity graphs then show that waste as ordinary demand, and nobody can trace it back to hedging.

A minimal implementation has four parts:

  • Start the primary attempt.
  • Wait for the hedge delay or a response, whichever comes first.
  • Before sending a hedge, take a token from a bucket. The bucket refills as a small fraction of successful primary requests.
  • Return the first result and cancel the rest.

The token bucket protects you from positive feedback. When the whole backend slows down, a larger share of requests crosses the delay. Without a budget, hedging adds load at the moment capacity runs out. You should also disable hedging automatically when the backend sheds load, returns overload status codes, or shows rising error rates.

Count attempts across every layer. Application retries, client-side load balancers, service meshes, and proxies can each retry or hedge the same call. Three layers with two attempts each can send eight requests where you expected two. Pick one layer to own hedging.

Hand-rolled hedging also costs ongoing engineering time. It needs tests for races and cancellation leaks, metrics, and review every time dependencies change.

// Go-style sketch: hedged call with shared cancellation and a hedge budget.
type HedgeBudget interface {
	TryTake() bool // false when the bucket is empty
	OnSuccess()    // refill a fraction of a token per success, e.g. 0.05
	Disabled() bool // true while backend sheds load or error rate is high
}

func hedgedCall(ctx context.Context, call func(context.Context) (Resp, error),
	delay time.Duration, budget HedgeBudget) (Resp, error) {
	ctx, cancel := context.WithCancel(ctx) // inherits the caller's deadline
	defer cancel()                         // cancels the losing attempt end to end

	type result struct { r Resp; err error }
	results := make(chan result, 2) // buffered so the loser never blocks or leaks
	attempt := func() { r, err := call(ctx); results <- result{r, err} } // pass ctx downstream

	go attempt()
	pending := 1
	timer := time.NewTimer(delay)
	defer timer.Stop()
	select {
	case <-timer.C:
		// Hedge only if the backend is healthy and the budget allows it.
		if !budget.Disabled() && budget.TryTake() { go attempt(); pending++ }
	case res := <-results: // primary finished before the hedge delay
		if res.err == nil { budget.OnSuccess() }
		return res.r, res.err
	case <-ctx.Done():
		return Resp{}, ctx.Err()
	}
	var lastErr error
	for ; pending > 0; pending-- { // first success wins; otherwise last error
		select {
		case res := <-results:
			if res.err == nil { budget.OnSuccess(); return res.r, nil }
			lastErr = res.err
		case <-ctx.Done():
			return Resp{}, ctx.Err()
		}
	}
	return Resp{}, lastErr
}

A Go-style sketch that sends a hedge only after a delay and only when a token-bucket budget and a backend health check allow it, then returns the first success and cancels the losing attempt through the shared context.

How do you measure whether hedging works, and what are the alternatives?

Judge hedging by the whole latency distribution. Put p50, p99 and p99.9 on the same dashboard as backend request rate and CPU. A good result is a lower p99 and p99.9, a flat p50, and a small, predictable rise in backend load. If p50 creeps up or CPU climbs sharply, the duplicates are causing the queueing they were meant to route around.

Track two hedging metrics as first-class signals:

  • Hedge-trigger rate is the share of requests that sent an extra attempt. If it is high, the delay is too short.
  • Hedge-win rate is the share of hedges where the duplicate answered first. If it is low, you are paying for load that rarely helps.

Tag every attempt with an attempt number and a hedge flag, in both request metadata and span attributes. Without these tags, backends count duplicates as organic traffic and traces show parallel calls nobody can explain. Per-client rate limits can also throttle users for traffic their client library generated.

Approach What it addresses Trade-off
Tuned retries with deadlines Failures and timeouts Waits for failure, so it does little for slow successes
Fixing slow replicas or GC pauses The root cause of the tail Takes investigation and engineering time
Latency-aware load balancing Uneven replica speed Needs fresh latency signals
Outlier ejection Persistently bad hosts Can shrink capacity if thresholds are loose
Caching Repeated reads Brings invalidation and staleness concerns
Reducing fan-out Compounding tail probability May require API or data model changes

Good instrumentation tells you which of these to reach for. If hedge wins cluster on one replica or one zone, retuning the hedge delay won't help. The fix is that host, and the metrics show it before anyone spends days on it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to request hedging tail latency, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.