Why does a slow downstream overload your own service?

When a dependency slows down, your service does not see an error. It sees requests that take longer to finish. Little's Law, which holds for any stable system, shows what that does to you: requests in flight equal the arrival rate multiplied by the time each request spends in the system.

Illustrative estimate. Assume 500 requests per second arrive, and each request makes one synchronous call that dominates its latency. At 40 ms, about 20 requests are in flight (500 × 0.04). If the downstream slows to 400 ms while traffic holds steady, about 200 are in flight (500 × 0.4). Your load has not changed, but you now hold ten times as much concurrent work.

Each in-flight request holds something: a worker thread, a pooled connection, buffers, and request state on the heap. Once a pool is exhausted, new requests wait in a queue. Queued requests add latency of their own, so in-flight counts climb further. Memory and GC pressure rise, and requests to unrelated endpoints get stuck behind the slow ones.

A dead dependency is less dangerous. A refused connection fails in milliseconds and frees its resources at once. A slow dependency holds each resource for the full duration of the call.

Timeouts set a ceiling on how long a request can hold resources, but they still let it hold them until the timeout fires. Retries make things worse: every timed-out call is sent again to the dependency that is already struggling, which raises arrival rate exactly when latency is high.

If your service makes synchronous calls to other services, this failure mode applies to you.

What is the difference between backpressure and load shedding?

Backpressure slows the producer down. A bounded queue blocks or refuses new items when it is full. A TCP or HTTP/2 flow-control window stops the sender until the receiver has consumed data. A pull-based consumer fetches the next message only when it has a free worker. No work is lost, but the waiting moves upstream.

Load shedding drops work instead. If a request cannot finish before its deadline, rejecting it immediately with a 429 or 503 costs less than accepting it, holding a thread and memory, and timing out anyway.

The two fit different system shapes. In queue-based async pipelines, messages are durable and producers can tolerate delay, so backpressure alone is often enough. In synchronous RPC, the producer is a caller with a timeout, usually with a user behind it. Slowing that caller only moves the queue into its thread pool. The rule of thumb: if waiting only adds latency the caller can accept, apply backpressure. If waiting means the response arrives after nobody wants it, shed.

Mechanism What it solves What it does not solve
Timeouts Bounds how long a caller waits Server keeps working on abandoned requests
Retries Transient, isolated failures Overload, which retries amplify
Circuit breakers Stops calling a failing dependency Gradual slowdowns that stay under the error threshold
Autoscaling Sustained demand growth Spikes faster than new capacity can start
Caching Repeated reads Writes and cache misses
Fallbacks Degraded but usable responses The capacity problem itself

How does concurrency-limit-based load shedding work at each hop?

Every hop enforces two numbers: a maximum number of requests in flight and a bounded queue in front of that limit. Size the queue against a latency budget, not a memory budget. If the handler completes about 200 requests per second at its concurrency limit and the hop's latency budget allows 100 ms of waiting, the queue holds about 20 requests (illustrative estimate). Anything beyond that would wait longer than the caller is willing to.

An unbounded queue does not absorb overload. It stores latency until memory runs out. Each queued request is likely to time out at the client, which often retries, so the server does work nobody is waiting for.

When the queue stays non-empty, the oldest entries are the most stale. Two common fixes:

  • LIFO under load serves the newest request first, because its caller is still waiting.
  • CoDel-style dropping watches how long requests sit in the queue. If the minimum wait over an interval stays above a target, it sheds from the head.

Propagate a deadline with each request so every hop knows how much time is left. Before doing work, drop any request whose deadline has already passed.

Client → [1] edge limiter → [2] bounded queue → [3] handler → [4] downstream pools

  1. Reject with 429 when the in-flight limit and queue are both full.
  2. Drop entries that are stale, have expired deadlines, or exceed the CoDel target.
  3. Re-check the deadline before running expensive work.
  4. Fail fast when a downstream pool is exhausted instead of blocking.
  +--------------------------+
  |          Client          |  sends request + deadline
  +--------------------------+
      |                 ^
      | req + deadline  | 429 / dropped / fail fast
      v                 |
  +--------------------------+       +------------------------------+
  | [1] Edge limiter         |--X--->| REJECT 429: in-flight limit  |
  |     max in-flight = N    |       | reached and queue is full    |
  +--------------------------+       +------------------------------+
      |
      v
  +--------------------------+       +------------------------------+
  | [2] Bounded queue        |--X--->| DROP: expired deadline, or   |
  |  cap ~ rate x wait budget|       | CoDel min wait > target;     |
  |  LIFO when non-empty     |       | stale entries shed from head |
  +--------------------------+       +------------------------------+
      |
      v
  +--------------------------+       +------------------------------+
  | [3] Handler              |--X--->| DROP: deadline already passed|
  |  re-check deadline       |       | before expensive work        |
  +--------------------------+       +------------------------------+
      |  remaining deadline
      v
  +--------------------------+       +------------------------------+
  | [4] Downstream pools     |--X--->| FAIL FAST: pool exhausted,   |
  |  (DB, RPC clients)       |       | no blocking wait             |
  +--------------------------+       +------------------------------+

A request flows from the client through the edge limiter, the bounded queue and the handler to the downstream pools, carrying its deadline, and each X marks a point where it is rejected or dropped instead of waiting.

Static vs adaptive concurrency limits: how do they find the knee?

A static limit is a fixed max-in-flight per downstream, such as 64 concurrent calls to the inventory service. It is simple to reason about, but the right number depends on the downstream's capacity, your instance size, and the request mix. That number changes every time any of them changes. An adaptive limiter measures the right number continuously instead of guessing it once.

The core signal is latency inflation. The limiter tracks a baseline minimum RTT, which approximates the no-queueing cost of a request, and compares it with recent RTT. When current RTT stays near the baseline, there is headroom, so the limit grows. When RTT rises well above the baseline, requests are queueing somewhere, so the limit shrinks.

  • AIMD adds one slot per successful window and multiplies the limit down (for example, by 0.9) on timeouts or drops.
  • Gradient-style algorithms scale the limit by the ratio of baseline RTT to current RTT, so they react in proportion to how much queueing they see.

Both settle near the knee of the latency curve, the point where extra concurrency stops adding throughput and only adds wait time.

Aspect Static limit Adaptive limit
Too tight Rejects work the downstream could handle Recovers by probing upward
Too loose Lets queues grow until timeouts Shrinks when RTT inflates
Fit across deploys and instance sizes One number rarely fits all Re-derived per instance
False signals None, because it ignores latency GC pauses, noisy neighbors, and a shift to heavier requests can look like overload

Adaptive limiters still need min/max bounds to stay stable. Be careful with them: a floor that is set too high keeps traffic flowing into a sick downstream, and a ceiling hides a stale baseline. If the limiter keeps hitting a bound, investigate why rather than widening the bound.

How do you implement load shedding that rejects early and cheaply?

Reject before doing work

Run the limiter as the first middleware after routing. It should come before body deserialization, lock acquisition, connection-pool checkout or any downstream call. A rejection should cost one counter check and a small response. If you parse a large body before you shed, the overloaded process has already paid for the request. Check the propagated deadline in the same place. If the remaining budget is shorter than your typical latency, fail immediately. When you do proceed, pass the reduced deadline downstream so the next hop can make the same decision.

Shed the lowest priority first

Tag each request at the edge. User-facing requests rank above batch, first attempts rank above retries, and health checks get a reserved slot so a busy instance is not marked dead. As the limit tightens, admit only the higher tiers. Derive priority from authenticated identity or route, never from a client-supplied header that anyone can set. Watch for priority inversion, where an admitted user request waits on a lock or connection held by batch work.

Signal overload correctly

Situation HTTP gRPC Expected client action
Per-client quota exceeded 429 RESOURCE_EXHAUSTED Slow this client down
Server overloaded 503 + Retry-After UNAVAILABLE Back off, retry later or elsewhere

Clients must cooperate

Use retry budgets that cap retries at a small fraction of total requests. Add exponential backoff with jitter, and retry at only one layer. Three layers each making three attempts can turn one call into 27. Circuit breakers should count 429 and 503 as signals to back off, not as noise.

One middleware, not per-endpoint tuning

A shared component gives every service the same metrics, tests and failure behaviour. Endpoints only declare a priority, so teams do not each rediscover limits, and fixes ship everywhere at once.

import type { IncomingMessage, ServerResponse } from "node:http";

type Tier = "health" | "user" | "batch";
// Illustrative limits: health keeps a reserved slot, batch gets a small share.
const LIMIT: Record<Tier, number> = { health: 2, user: 80, batch: 20 };
const inFlight: Record<Tier, number> = { health: 0, user: 0, batch: 0 };
const TYPICAL_LATENCY_MS = 50; // assumed p50 for this service

// Priority comes from the route and the authenticated principal, never a client-set header.
// The retry-attempt header is client-supplied, so it is only ever allowed to LOWER priority.
function classify(req: IncomingMessage, principalKind: string): Tier {
  if (req.url === "/healthz") return "health";
  if (principalKind === "batch-job") return "batch";
  return Number(req.headers["x-retry-attempt"] ?? 0) > 0 ? "batch" : "user";
}

function reject(res: ServerResponse, status: 503 | 504, retryAfterSec?: number): void {
  const headers: Record<string, string> = { "content-length": "0" };
  if (retryAfterSec) headers["retry-after"] = String(retryAfterSec);
  res.writeHead(status, headers); // tiny response: the rejection must stay cheap
  res.end();
}

// Runs right after routing: before body parsing, locks, pool checkout or downstream calls.
export async function shed(req: IncomingMessage, res: ServerResponse, principalKind: string,
                           next: (deadlineUnixMs: number) => Promise<void>): Promise<void> {
  const deadline = Number(req.headers["x-deadline-unix-ms"] ?? Date.now() + 1000);
  // Not enough budget left to finish: fail now instead of doing doomed work.
  if (deadline - Date.now() < TYPICAL_LATENCY_MS) return reject(res, 504);

  const tier = classify(req, principalKind);
  if (inFlight[tier] >= LIMIT[tier]) return reject(res, 503, 2); // server overload, not quota

  inFlight[tier]++;
  try {
    await next(deadline); // handler forwards the same absolute deadline to the next hop
  } finally {
    inFlight[tier]--;
  }
}

A minimal edge middleware that checks the propagated deadline and a per-priority concurrency counter before any expensive work, then answers with a small 503 plus Retry-After when the request's tier is full; the limits are assumed values, and the header names are placeholders.

How do you observe and test load shedding?

Metrics to collect

Instrument the limiter itself, not only the endpoints behind it:

  • In-flight count and current limit, plotted together. The gap between them shows how close you are to shedding. A limit that keeps falling while in-flight stays flat points to slowing downstreams, not rising traffic.
  • Queue wait time, measured separately from service time. If you combine them, you cannot tell whether requests are slow because work is slow or because they waited too long to start.
  • Shed rate by priority and reason: limit reached, deadline already expired, or queue timeout. This confirms that low-priority traffic is dropped first.
  • Per-dependency downstream latency, so a falling limit can be traced to a specific dependency.

Count shed responses (429s or their equivalent) separately from errors. If they share an error-rate panel, sheds look like an outage and real failures get lost among them. Alert on sustained shedding, such as a meaningful shed rate held for several minutes. Single spikes are the limiter doing its job.

Load tests

Inject latency into a downstream instead of errors. Slow responses hold concurrency slots, and that is the failure mode the limiter exists to handle. Fast errors do not exercise it. Push offered load past capacity in steps and check that goodput, meaning successful responses within their deadline, stays flat instead of collapsing. Then remove the injected latency and confirm that the limit recovers smoothly without oscillating between shedding and overload.

This testing saves time during incidents. On-call engineers already know what normal shedding looks like on the dashboards, so they can quickly tell controlled degradation from a real failure.

# Prometheus rules for limiter observability.
# Metric names are placeholders for your own instrumentation:
#   limiter_shed_total{reason, priority}  - requests rejected by the limiter
#   http_requests_total{code}             - all completed requests
groups:
  - name: load-shedding
    rules:
      # Shed rate as its own series, split by priority and reason,
      # so dashboards never fold it into the error rate.
      - record: limiter:shed_rate:ratio_5m
        expr: |
          sum by (priority, reason) (rate(limiter_shed_total[5m]))
          / ignoring (priority, reason) group_left
          sum(rate(http_requests_total[5m]))

      # Error rate that excludes 429s, so real failures are not
      # hidden among intentional sheds.
      - record: http:error_rate_excluding_shed:ratio_5m
        expr: |
          sum(rate(http_requests_total{code=~"5.."}[5m]))
          / sum(rate(http_requests_total{code!="429"}[5m]))

      # Page only on sustained shedding. Short spikes are the limiter
      # doing its job; 'for' holds the alert until the condition persists.
      - alert: SustainedLoadShedding
        expr: sum(limiter:shed_rate:ratio_5m) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Shedding above 5% of requests for 10 minutes"
          description: "Check per-dependency latency to find which downstream is pulling the limit down."

Prometheus recording and alerting rules that track shed rate by priority and reason separately from 5xx errors and alert only when shedding is sustained, using placeholder metric names and thresholds.

When should you avoid or limit microservice load shedding?

Load shedding has real costs, and some of them only show up once it is running in production.

Per-instance limits only see local load. Each instance protects itself, but it knows nothing about the rest of the fleet. If load balancing is uneven, one hot instance sheds while its neighbours sit idle. A global limit coordinated through a shared store fixes that, but it adds a network hop to every admission decision, introduces shared state, and creates a new dependency that can fail.

Clients can turn rejections into more load. A 429 is only cheap if the caller backs off. Naive or third-party clients often retry immediately, sometimes several times, so each rejected request comes back as more traffic. Send Retry-After, keep retry budgets in clients you control, and assume clients you don't control will misbehave.

LIFO is unfair by design. Serving the newest requests first maximises goodput because those callers are still waiting. The cost is that older requests starve.

Control loops can interact. An adaptive limiter and a CPU-based autoscaler both react to the same overload. When the limiter sheds, CPU usage drops, the autoscaler scales in, and the pressure comes back. The two loops can keep oscillating against each other.

Shedding does not fix the downstream. It turns latency into errors. You still need fallbacks, caching, and a product decision about what users see when their request is rejected.

Skip shedding, or keep it to a static limit, for low-traffic services and for services with no synchronous dependencies. In those cases a timeout and a bounded queue are usually enough.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice load shedding, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.