Why does a slow downstream service hurt more than a failed one?

Load shedding means deliberately rejecting some requests so the ones you accept still complete in time. Backpressure means signaling upstream that you cannot take more work, so callers slow down instead of piling requests onto you. Both exist because of one relationship, Little's Law:

in-flight requests = arrival rate × latency

Here is an illustrative estimate with assumed inputs. A service receives 500 requests per second, and each request waits on one downstream call that normally takes 100 ms. That gives 500 × 0.1 = 50 requests in flight. Suppose the service runs a 200-thread pool and a 200-connection client pool. If downstream latency rises to 200 ms, in-flight work doubles to 100. At 400 ms it reaches 200, and every thread and connection is occupied. At 1 s it would need 500. The arrival rate never changed. Only latency did, and the caller exhausted its threads, connections, and the memory that holds request buffers and context.

A dependency that fails fast does the opposite. If it returns an error in 5 ms, in-flight work drops to about 2.5, and the caller's resources are released almost immediately. The errors are visible, and capacity remains available.

Unbounded queues make slowness harder to see. Requests that cannot get a thread wait in a queue instead of being rejected. The server keeps completing requests at its full rate, so throughput dashboards look normal. But many of those requests have already passed their client's timeout. The client has given up, possibly retried, and the response is thrown away. Goodput, meaning responses delivered while someone is still waiting for them, collapses toward zero even though throughput looks healthy.

What is the difference between backpressure and load shedding?

Backpressure is a signal sent upstream that says "slow down." It only works if the caller can receive that signal and act on it. Inside a process, backpressure comes from bounded buffers and flow control, such as blocking writes or HTTP/2 and TCP windows. Between services, it usually takes the form of a 429 or 503 response with a Retry-After hint, so well-behaved clients back off. Load shedding is the fallback when upstream can't or won't slow down: the server rejects excess work locally, early and cheaply, before spending CPU or downstream calls on it.

The path type determines which tool does most of the work. In synchronous RPC, callers hold connections and timeouts, so a rejection is effectively shedding from their point of view, and the main win is making it fast. In queue-based systems, backpressure is the natural default, because consumers pull at their own pace and producers get blocked or rejected when the queue is full.

Bound queues by time as well as length. A thousand items is harmless at 1 ms each but stale at 500 ms each. Track how long each item has waited, drop items past a threshold, and discard any request whose propagated deadline has already expired, since nobody is waiting for that answer.

Mechanism Acts on Weakness alone
Static rate limit Request rate Doesn't track real capacity
Bounded queue Buffer size Length hides latency
Backpressure Caller behavior Needs cooperative callers
Load shedding Local admission Rejected work is lost
Autoscaling Capacity Minutes too slow for spikes

How do adaptive concurrency limits work?

An adaptive limiter treats the number of in-flight calls to a dependency as a control variable. Little's law explains why concurrency is the right variable: in-flight requests equal throughput times latency, so when a downstream slows at constant traffic, open calls pile up even though the request rate hasn't changed. The limiter compares observed latency against a baseline, usually a recent minimum or low percentile, and adjusts.

AIMD control adds a small step, such as one slot per window, while latency stays near baseline. It multiplies the limit by a factor like 0.9 when latency crosses a threshold, a timeout fires or errors appear. Gradient control scales the limit by the ratio of baseline to current latency, plus a small headroom term so the limit can keep probing upward.

A static rate limit encodes capacity measured once. Deploys, noisy neighbors and a degraded dependency change that capacity hourly. An adaptive limit rediscovers it continuously.

A minimal AIMD wrapper around a downstream call does four things:

  1. Wait for a slot for no longer than the smaller of a maximum wait and the remaining deadline, then reject.
  2. Skip the call if the remaining deadline is shorter than typical latency.
  3. Call the dependency and measure the result.
  4. Release the slot, then increase or decrease the limit.

Expect four failure modes:

  • Oscillation: aggressive decrease factors and short windows cause the limit to swing up and down. Smooth the samples and add a cooldown after each cut.
  • Slow reaction: long windows let queues grow before the limiter responds.
  • Settling too low: errors unrelated to load, such as client errors or a single bad host, keep cutting the limit. Classify errors before they count against it, and set a floor.
  • Bimodal latency: a baseline taken from cache hits makes every cache miss look like overload. Use separate limiters per path or a higher-percentile baseline.
limit = 20; MIN_LIMIT = 4; MAX_LIMIT = 200   # floor stops the limit settling at zero
MAX_WAIT = 50ms                                # never queue longer than this for a slot
TOLERANCE = 2.0                                # 'near baseline' = under 2x baseline latency
DECREASE = 0.9; COOLDOWN = 1s                  # gentle cut, at most one cut per cooldown
last_cut = 0

function call_downstream(request, deadline):
    wait_budget = min(MAX_WAIT, deadline - now())
    if not acquire_slot(timeout = wait_budget):    # succeeds only while in_flight < floor(limit)
        return reject(503, "overloaded")           # shed quickly instead of queueing
    if deadline - now() < typical_latency():       # e.g. recent p50 for this endpoint
        release_slot()
        return reject(504, "deadline too short")   # the call cannot finish in time
    start = now()
    try:
        result = dependency.call(request, timeout = deadline - now())
    finally:
        release_slot()                             # always free the slot first
    observe(now() - start, result)
    return result

function observe(latency, result):
    # Only load-related failures count. Client errors (4xx) do not cut the limit.
    overloaded = result.is_timeout or result.is_server_overload
    # Bimodal endpoints (cache hit vs miss) need a separate baseline per class.
    baseline = recent_min_latency(result.latency_class)
    if overloaded or latency > baseline * TOLERANCE:
        if now() - last_cut > COOLDOWN:            # one cut per cooldown damps oscillation
            limit = max(MIN_LIMIT, limit * DECREASE)
            last_cut = now()
    else if result.ok:
        limit = min(MAX_LIMIT, limit + 1 / limit)  # adds about one slot per 'limit' successes

Pseudocode for an AIMD concurrency limiter that bounds the wait for a slot, skips calls that cannot meet their deadline, releases the slot before adjusting the limit, and uses error classification, a floor, a cooldown and per-class baselines to guard against the failure modes described above.

Where should you shed load in a microservice architecture?

Shed as early as possible. Every layer a request passes through adds parsing, allocation, and queueing that a late rejection wastes. Each layer also protects against a different failure, so most systems need more than one checkpoint.

client
  |
[edge / LB] ---- global rate + connection limits (cheapest reject)
  |
[service ingress] -- adaptive concurrency limit
  |                  deadline check: drop if remaining budget < expected cost
  |                  reject BEFORE body parse, auth call, fan-out
  |
[handler]
  |-- bulkhead A (pool + limiter) --> downstream A (slow)
  |-- bulkhead B (pool + limiter) --> downstream B
  `-- bulkhead C (pool + limiter) --> downstream C
        ^ deadline check before each outbound call

The edge saves the most work because a request rejected there never reaches your service. Inside the service, put the ingress limiter and deadline check before you read a large body, call an auth service, or start a fan-out. Otherwise you pay most of the cost of a request only to discard it.

Per-dependency bulkheads handle a different problem. When downstream A slows down, its calls hold threads and connections longer. Without isolation they fill the shared pool, and endpoints that only use B or C start failing as well. A separate pool and limiter for each dependency keeps the damage limited to A's callers.

None of this is free. Each rejection still costs a connection accept, a TLS handshake, and some request parsing, so under extreme overload, shedding in the application alone can saturate the CPU. That is why limits at the edge and at the connection level still matter. Per-instance limiters also see only their own traffic. If the load balancer spreads requests unevenly, the hot instances shed while the cold ones sit idle. Least-loaded balancing, or limits informed by fleet-wide signals, reduces that skew.

 client
   |
   v
+------------------------------------------+  [1] global rate limit
| EDGE / LOAD BALANCER                     |      + connection limit
| (request never reaches the service)      |      -> reject: cheapest point
+------------------------------------------+
   |
   v
+------------------------------------------+  [2] adaptive concurrency limit
| SERVICE INGRESS                          |  [3] deadline check: drop if
|  accept + TLS (cost paid even on reject) |      budget < expected cost
+------------------------------------------+      -> reject BEFORE body parse,
   |                                                 auth call, fan-out
   v
+------------------------------------------+
| HANDLER: parse body, auth, fan-out       |
+------------------------------------------+
      |               |               |
     [D]             [D]             [D]   <- deadline check before each call
      v               v               v
+-----------+   +-----------+   +-----------+  [4] per-dependency bulkhead
| bulkhead A|   | bulkhead B|   | bulkhead C|      own pool + limiter
| pool+limit|   | pool+limit|   | pool+limit|
+-----------+   +-----------+   +-----------+
      |               |               |
      v               v               v
downstream A    downstream B    downstream C
  (SLOW)        only A's pool fills; B and C callers keep being served

Request flow from edge to three downstreams: rate and connection limits at the edge, a concurrency limiter and deadline check at ingress before expensive work, and a deadline check plus separate bulkhead in front of each dependency so one slow downstream cannot starve the others.

How do you prioritize which requests to shed?

Shedding at random wastes capacity. A rejected checkout and a rejected nightly report cost the same CPU to refuse, but only one of them costs revenue. Assign every request a criticality class at the edge and shed from the bottom up.

A workable scheme has three or four tiers:

  • Critical: user-facing writes such as payments and order placement.
  • Default: user-facing reads, which can often fall back to cached or degraded responses.
  • Sheddable: batch jobs, prefetching, analytics, and backfills, which can retry later without anyone noticing.
  • Health checks: handle these separately. If a liveness probe gets shed, the orchestrator may restart a healthy but busy instance and make the overload worse. Answer probes cheaply and exempt them from the limiter.

The admission check compares the current load against a threshold for each tier. Sheddable traffic is rejected when the limiter reaches about 70% of its limit, and critical traffic only when it is at 100%. This keeps headroom free for the requests that matter.

Priority and deadline both have to travel with the request. If the gateway tags a request as sheddable but a downstream service treats untagged calls as critical, the two services make conflicting decisions. Propagate the absolute deadline, and have each hop compute its timeout as the remaining time minus a small margin. If that remaining budget is already below the expected service time, reject the request immediately rather than doing work nobody will wait for.

Most of the difficulty here is organizational. Product and SLO owners have to agree in writing on what is expendable. Every team also has to honor the same header names and defaults. If only some services adopt the scheme, priorities get silently dropped at the services that don't, and the system behaves inconsistently.

import time

PRIORITY_HDR = "x-request-priority"     # critical | default | sheddable
DEADLINE_HDR = "x-request-deadline-ms"  # absolute Unix epoch, milliseconds

THRESHOLDS = {"critical": 1.0, "default": 0.9, "sheddable": 0.7}
SAFETY_MARGIN_S = 0.020     # time reserved for our own response and network
EXPECTED_SERVICE_S = 0.050  # typical time this service needs to do the work
HEALTH_PATHS = {"/healthz", "/readyz"}

def remaining_timeout(headers, now=None):
    """Seconds left for downstream calls, or None if no deadline was sent."""
    raw = headers.get(DEADLINE_HDR)
    if raw is None:
        return None
    now = time.time() if now is None else now
    # An absolute deadline assumes host clocks are reasonably synchronized.
    return int(raw) / 1000.0 - now - SAFETY_MARGIN_S

def admit(path, headers, in_flight, limit):
    """Return (admitted, reason_or_timeout)."""
    if path in HEALTH_PATHS:
        return True, None  # probes bypass the limiter entirely
    # Missing or unknown priority maps to the agreed default, never critical.
    priority = headers.get(PRIORITY_HDR, "default")
    if priority not in THRESHOLDS:
        priority = "default"
    if in_flight >= THRESHOLDS[priority] * limit:
        return False, "overloaded"  # e.g. respond 503 with Retry-After
    timeout = remaining_timeout(headers)
    if timeout is not None and timeout < EXPECTED_SERVICE_S:
        return False, "deadline"  # caller will give up before we finish
    return True, timeout  # use as the timeout on outbound calls

def outbound_headers(headers):
    # Forward priority and the same absolute deadline unchanged to the next hop.
    return {h: headers[h] for h in (PRIORITY_HDR, DEADLINE_HDR) if h in headers}

A priority-aware admission check that exempts health probes, rejects lower tiers at lower load thresholds, refuses requests whose remaining deadline is shorter than the expected service time, and forwards priority and the absolute deadline to the next hop.

How do retries, timeouts and circuit breakers interact with load shedding?

Shed load only helps if it stays shed. When a rejected request comes back as several retries, the shedding layer ends up doing more work than if it had accepted the original call. Retry discipline is part of overload protection.

Illustrative estimate (assumed numbers, not a benchmark): worst-case attempts at the deepest service = (1 + r)^L, where r is retries per call and L is the number of layers that retry. With r = 2 and L = 3, that is 3³ = 27 attempts per user request.

Four rules keep this in check:

  • Use a retry budget. Cap retries as a ratio of recent successful traffic, for example 10%, rather than a fixed count per call. When the success rate drops, the budget shrinks automatically.
  • Back off exponentially with jitter. This keeps synchronized clients from retrying in waves.
  • Retry at one layer only. Usually that is the layer closest to the caller that knows whether the operation is idempotent.
  • Treat a rejection as an answer. A 429, a 503 with Retry-After, or gRPC RESOURCE_EXHAUSTED means "not now," so honor the delay.

Derive timeouts from the request's remaining deadline minus local work. If you guess at timeouts instead, callers abandon requests that downstream services are still processing.

Mechanism Trigger signal Scope Typical failure mode Configure so it doesn't fight the others
Concurrency limiter Latency and in-flight saturation Per instance or per dependency Limit too low or too slow to recover, which throttles healthy traffic Return explicit rejections that callers don't count as retryable errors
Circuit breaker Error rate Per downstream dependency Trips on shed responses and blocks a recovering service Exclude deliberate rejections from the error count, or weight them separately
Retry policy Retryable failure Per call, budgeted per client Amplifies load Use a budget, and stop retrying once the deadline is near
Timeout Elapsed time Per call Fires before useful work finishes, or fires too late to matter Compute it from the propagated deadline

Who should use load shedding?

Load shedding is aimed at the teams that own request-serving services sitting between callers and dependencies they don't control. In practice that means:

  • Backend teams running synchronous APIs with shared thread pools, connection pools, or event loops, where a slow dependency can hold resources that every endpoint needs.
  • Teams owning aggregation or gateway services that fan out to several downstreams. These services are the first to fill up when any one dependency slows, and they decide which downstream calls are worth making.
  • Platform and SRE teams that maintain shared service frameworks, sidecars, or edge configuration. Putting the limiter, deadline propagation, and priority headers into a common library or proxy means individual service teams adopt consistent behavior instead of each writing their own.
  • Teams with explicit latency SLOs or user-facing critical paths, such as checkout, login, or search, where a fast, labeled rejection of low-priority work protects the requests that matter.

It is a weaker fit for small services with low, steady traffic, for purely asynchronous pipelines where a bounded queue and consumer-paced processing already provide backpressure, and for teams that do not yet have per-dependency latency and rejection metrics. Without those metrics, nobody can tell whether a limiter is protecting the service or throttling it by mistake.

What are the alternatives to load shedding?

Most alternatives either add capacity, reduce demand, or fail faster. Each covers part of the problem, and most production systems combine several of them with load shedding rather than replacing it.

  • Overprovisioning. Run enough spare threads, connections, and instances to absorb expected slowdowns. This is simple and needs no coordination, but Little's Law still applies: a large enough latency increase fills any fixed pool, and the spare capacity costs money all the time.
  • Autoscaling. Add instances when load or saturation rises. It handles sustained growth well but usually reacts in minutes, and it does nothing when the bottleneck is the slow downstream itself. Scaling up callers of a struggling dependency can send it more concurrent work.
  • Static rate limits and quotas. Cap requests per client or per endpoint. They are easy to reason about and good for fairness between tenants, but they encode capacity measured once and don't react when a dependency's latency changes.
  • Timeouts alone. Aggressive timeouts release resources sooner, which limits how long a slow call can hold a slot. Without admission control, though, new requests keep arriving and filling the same slots, and timed-out work is still wasted work.
  • Circuit breakers alone. A breaker stops calls to a dependency whose error rate is high. It reacts to errors, not latency, so a dependency that is slow but still succeeding may never trip it, and the caller's pools fill anyway.
  • Bulkheads without adaptive limits. Fixed per-dependency pools contain the blast radius of one slow downstream. They are a good first step, but the pool sizes are static and need manual retuning as traffic and dependency behavior change.
  • Asynchronous processing and queues. Moving work off the request path lets consumers pull at their own pace, which turns overload into backlog. This suits work that doesn't need an immediate answer, but the queue still needs time-based bounds, and users waiting on a synchronous response don't benefit.
  • Caching and degraded responses. Serving stale data or a reduced response when a dependency is slow removes demand on it entirely. This works well for reads and is often paired with shedding, but it doesn't apply to writes or to data that must be fresh.

The comparison table in the backpressure section summarizes the main weakness of each mechanism used alone. Load shedding fills the gap they leave: it reacts within seconds to the capacity actually available and decides locally which work to drop.

How can load shedding save engineering time?

The time saving comes mainly from turning unpredictable incidents into predictable, bounded degradation.

  • Smaller incidents. Without shedding and bulkheads, a slow dependency can exhaust shared pools and fail unrelated endpoints, which pulls several teams into one incident. With per-dependency limits, the impact stays on the callers of the slow service, and fewer people need to be involved.
  • Faster diagnosis. Rejections labeled by reason, priority, and dependency point directly at the cause. Engineers spend less time correlating dashboards to work out which downstream started the cascade.
  • Less manual intervention during incidents. An adaptive limiter lowers concurrency automatically when latency rises and raises it again on recovery. That replaces actions such as manually scaling pools, restarting saturated instances, or disabling features under pressure.
  • Less hand-tuning over time. Static pool sizes and rate limits need revisiting after deploys, traffic changes, and dependency changes. An adaptive limit with a sensible floor and ceiling rediscovers capacity on its own, so fewer configuration changes are needed.
  • Shared implementation. When a platform team puts the limiter, deadline propagation, and priority handling in a common library or proxy, each service team inherits the behavior instead of building and debugging its own overload handling.

Illustrative estimate (assumed inputs, not a measured result): assume a team has two dependency-slowdown incidents per month. Assume each one, without isolation, cascades into unrelated endpoints and involves four engineers for three hours, or 12 engineer-hours. Assume that with bulkheads and labeled rejections, the same slowdown is contained to one service and handled by one engineer in one hour. That is 24 engineer-hours a month reduced to 2. Against this, count the up-front cost of building, tuning, and load testing the limiters, agreeing on priority classes, and the ongoing risk of a misconfigured limit causing its own outage. The trade only works out if the metrics described in the next section are in place so limits can be tuned and verified.

When should you use load shedding, and when should you avoid it?

Load shedding pays off when one slow dependency can consume a resource that every request needs. Services with high fan-out, shared thread or connection pools, dependencies whose health varies, or strict latency SLOs benefit most. In these services, a slowdown fills the pool quickly, and rejecting early is cheaper than letting every caller time out.

Situation Recommendation Reason
Fan-out or shared pools Adopt One slow call can exhaust capacity needed by every request
Strict latency SLO Adopt A fast rejection keeps the remaining requests inside the target
Low, steady traffic Defer Little contention, and the mechanism is hard to tune with sparse data
Brief, predictable spikes Prefer a small bounded buffer Absorbing a short burst is simpler than rejecting it
No per-limit metrics Defer You cannot tell a correct rejection from a misconfiguration

The cost is real. Teams spend time tuning limits and agreeing with other teams on priority classes and on how callers should handle rejections. Each limiter is also another component that can cause a self-inflicted outage if a limit is set too low. The return is that overload behaves predictably: a dependency slowdown produces bounded, labeled rejections instead of a cascading incident that pulls in several teams.

To tune limits and tell correct rejections from misconfiguration, emit these metrics:

  • Rejected counts by reason and by priority
  • The current limit value
  • Queue wait time
  • Goodput (requests completed within their deadline) compared with raw throughput

Load tests should inject downstream latency, not only errors, because latency is what fills pools. The tests should also confirm that limits rise again and the rejection rate drops once the slowdown ends. A limiter that sheds correctly but never recovers is itself an outage.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to load shedding microservices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.