What is microservice load shedding, and how does it differ from backpressure?
Microservice load shedding means deciding at admission time that a request will not be served, and rejecting it before the service spends meaningful CPU, memory, connections or downstream calls on it. Shedding is useful only when the rejection is cheap. A rejection that arrives after the request has waited 800 ms in a queue protects nothing.
Backpressure slows producers so they send less. It works through bounded buffers, protocol flow control (TCP windows, HTTP/2 stream credits) or blocking writes.
The difference matters most for synchronous calls. In HTTP or gRPC request/response, the caller is already waiting. If a server "applies backpressure" by accepting the connection and holding the request, it is queueing the request. The caller's deadline keeps running, and the queued work may finish after the caller has stopped waiting for it. Async consumers behave differently. A pull-based consumer with bounded prefetch only takes messages it has capacity for, and the backlog waits durably in the broker, where it costs nothing. In that setting backpressure comes almost for free.
Teams need explicit shedding when a request-serving service has fan-in from many callers, sits in front of a shared dependency such as a database or another service, or sees bursty traffic. In those cases no single producer can be slowed enough to protect the server.
| Load shedding | Backpressure | |
|---|---|---|
| Where it acts | Server admission point | Between producer and consumer |
| Cost of rejection | Very low if done early | No rejection; work is delayed |
| Sync HTTP/gRPC | Strong fit | Often becomes queueing |
| Async consumers | Rarely needed | Natural fit via pull/prefetch |
Why do large queues turn overload into latency collapse?
Little's Law ties the three quantities together: requests in the system = throughput × time each request spends there. A server's throughput is fixed by its capacity, so any extra requests in the system can only show up as extra latency. A queue does not add capacity. It converts overload into waiting time.
Illustrative estimate
The numbers below are assumed inputs, not measurements:
- One instance completes 200 req/s at 50 ms each, so about 10 requests are in service at once.
- Offered load is 1,000 req/s.
- The queue holds up to 5,000 requests.
- Clients time out after 2 s.
The queue grows by about 800 req/s. Queue wait is depth ÷ 200 req/s, so wait passes the 2 s timeout once depth reaches 400. That happens about 0.5 s into the overload. The queue fills in about 6 s, and from then on each request waits around 25 s before it is served.
After that first half-second, the server still finishes 200 req/s. CPU is busy and the throughput graph looks healthy. But nearly every response goes to a client that has already given up. Goodput, the rate of useful completed responses, falls toward zero.
Clients that retry on timeout make it worse. If each one retries once, offered load approaches 2,000 req/s. The extra requests join the same queue and arrive too late to be useful.
A larger queue only delays the point where you see the problem. The wait time is what breaks the system, and by the time queue length looks alarming, the latency damage has already happened. That is why the shed decision belongs at admission, based on in-flight concurrency and observed latency, before a request takes a place in line.
How does admission-time load shedding work inside a service?
Admission-time shedding has three parts that sit in front of any real work:
- A hard cap on in-flight requests per instance. Size it to what the instance can finish within its latency target, not to what the thread pool or event loop can hold.
- A short, bounded queue. It absorbs brief bursts. It should hold roughly one latency budget's worth of work, not minutes of backlog.
- A rejection check at the edge of the process. The check runs before reading large request bodies, calling an auth service or taking a database connection from the pool.
connection ──▶ listener (accept, TLS, headers)
│
▼
┌─ admission check ──────────────┐
│ 1. deadline already too short? ├──▶ REJECT (no work done)
│ 2. in-flight >= limit? │
│ └─ queue full / low priority├──▶ REJECT (429/503)
│ 3. priority class │
└───────────────┬────────────────┘
▼
short bounded queue ── wait > budget ──▶ REJECT (timed out)
▼
handler (body parse, auth)
▼
downstream calls (DB, RPC) ── budget exhausted ──▶ fail fast
Order matters because a rejection is only cheap if it happens before expensive work. If a request has already spent 40 ms on JWT validation and a pool checkout, rejecting it saves little. Rejecting at admission costs a header read and a counter comparison. The deadline check runs first because a request whose caller has already given up is waste whatever the current load.
The process cannot protect itself entirely. Accepting TCP connections, completing TLS handshakes and parsing headers all use CPU before your admission code runs. Under a large enough flood, that work alone can saturate the instance. Put connection limits, accept-rate limits, or a proxy that sheds in front of the process, so overload is cut off before your code is ever invoked.
clients
v
+--------------------------------+
| proxy / LB: connection limits |---> REJECT (conn cap, rate limit)
+--------------------------------+
v
============ service instance boundary ============
+--------------------------------+
| listener: accept, TLS, headers | <- CPU spent before shedding runs
+--------------------------------+
v
+--------------------------------+
| admission check |
| 1. deadline already too short? |---> REJECT (no work done)
| 2. in-flight >= limit? |
| queue full or low priority? |---> REJECT (429/503)
| 3. assign priority class |
+--------------------------------+
v
+--------------------------------+
| short bounded queue |---> REJECT (waited > budget)
+--------------------------------+
v
+--------------------------------+
| handler: body parse, auth |
+--------------------------------+
v
+--------------------------------+
| downstream calls: DB, RPC |---> fail fast (budget exhausted)
+--------------------------------+
Request path through an instance with admission-time shedding: every rejection exit sits before the next expensive stage, and a proxy with connection limits absorbs the accept, TLS and header-parsing cost that the process cannot avoid.
How can you implement concurrency limits, adaptive limits and deadline-aware shedding?
A semaphore sized to the in-flight limit is the simplest admission control. The middleware attempts a non-blocking acquire. If no slot is free, it returns an overload response immediately rather than waiting. If you allow a short queue, check the remaining deadline when a request is dequeued, not when it arrives. A request that waited 400 ms against a 500 ms budget will likely fail anyway, so drop it before it takes a slot.
Adaptive limits
Static limits go stale when hardware, payload mix or dependencies change. AIMD raises the limit by one while latency stays healthy and multiplies it down, for example halving it, on latency rise, timeouts or errors. Gradient approaches compare short-term latency with a tracked minimum and scale the limit by that ratio. Feed them service time excluding queue wait, in-flight count, timeouts and downstream errors. Including queue wait creates a feedback loop.
Expect two risks. Limits can oscillate. They can also converge too low when latency rises for reasons unrelated to load, such as a slow dependency, GC pauses or noisy neighbors. Set a floor.
Deadlines and queue order
Propagate deadlines. gRPC does this natively, and over HTTP you can use a header carrying the remaining budget. Drop requests whose remaining budget is below typical service time. LIFO or CoDel-style queues serve fresh requests that can still succeed. Older requests are sacrificed, which is unfair but usually the right trade.
Priority
Shed batch traffic before user-facing traffic, and retries before first attempts. Exempt health checks deliberately, or orchestrators may restart instances that are busy but healthy. Reserve some capacity for low-priority classes so they are not starved indefinitely. Derive priority from authenticated identity, never from client-supplied headers.
// Go: admission-control middleware (net/http, standard library only)
package shed
import (
"net/http"
"time"
)
// Limit caps in-flight requests and drops work that cannot finish in time.
// minBudget should be close to typical service time, e.g. 50 * time.Millisecond.
func Limit(next http.Handler, maxInFlight int, minBudget time.Duration) http.Handler {
// A buffered channel acts as a counting semaphore.
sem := make(chan struct{}, maxInFlight)
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Non-blocking acquire: if no slot is free, reject now instead of queueing.
select {
case sem <- struct{}{}:
defer func() { <-sem }()
default:
w.Header().Set("Retry-After", "1")
http.Error(w, "overloaded", http.StatusServiceUnavailable)
return
}
// Check the remaining budget at the moment work would start, not on arrival.
// The deadline comes from upstream middleware that turned a timeout header
// into a context deadline.
if dl, ok := r.Context().Deadline(); ok && time.Until(dl) < minBudget {
http.Error(w, "insufficient deadline", http.StatusServiceUnavailable)
return // deferred release frees the slot for a request that can succeed
}
next.ServeHTTP(w, r)
})
}
Go middleware that uses a buffered channel as a semaphore to reject requests immediately when the in-flight limit is reached, and that drops requests whose remaining deadline is too short to finish before they do any work.
Should you return 429 or 503, RESOURCE_EXHAUSTED or UNAVAILABLE?
The status code tells the caller who is at fault and whether retrying could help. Use 429 (gRPC RESOURCE_EXHAUSTED) when this caller exceeded a quota or rate limit. Retrying elsewhere won't help, because the limit follows the caller. Use 503 (gRPC UNAVAILABLE) when this instance is overloaded. Another replica may have headroom.
| Cause | HTTP | gRPC | Retry elsewhere? | Retry-After hint |
|---|---|---|---|---|
| Per-caller quota exceeded | 429 | RESOURCE_EXHAUSTED |
No | Time until quota window resets |
| Instance overload | 503 | UNAVAILABLE |
Yes, another replica | Short, about one to a few seconds, jittered |
| Deadline already too short | 503 (or 504 at a gateway) | DEADLINE_EXCEEDED |
No, the budget is spent | Omit |
| Dependency down | 503 | UNAVAILABLE |
Only if replicas use different dependencies | Longer; reflect expected recovery |
Retry-After accepts seconds or an HTTP date. gRPC has no direct equivalent, so put the hint in response metadata. Also check whether your gRPC retry policy treats RESOURCE_EXHAUSTED as retryable. Usually it should not.
Rejections only shed load if clients back off. Three mechanisms help:
- Exponential backoff with full jitter spreads retries out so they don't arrive as synchronized waves.
- Retry budgets cap retries at a small fraction of successful traffic, so retries cannot multiply load during an outage.
- Client-side adaptive throttling tracks recent requests and accepts. When the accept ratio drops, the client rejects some requests locally with a probability like
max(0, (requests − K·accepts) / (requests + 1)). Those requests never reach the overloaded server.
This only works when clients, proxies and SDKs honor these signals. You rarely control every caller. Third-party integrations, old SDK versions and retry-happy proxies will ignore Retry-After. Server-side shedding must therefore hold up on its own without client cooperation.
import random, time
from collections import deque
K = 2.0 # adaptive throttling: tolerate ~K requests per backend accept
WINDOW_S = 120.0 # how far back request/accept history reaches
history = deque() # (timestamp, accepted_by_backend)
budget = 10.0 # retry budget tokens; successes refill, retries spend
def reject_locally() -> bool:
now = time.monotonic()
while history and now - history[0][0] > WINDOW_S:
history.popleft()
requests = len(history)
accepts = sum(1 for _, ok in history if ok)
p = max(0.0, (requests - K * accepts) / (requests + 1))
return random.random() < p # drop before it reaches an overloaded server
def call(send, max_attempts=3, base_s=0.1, cap_s=5.0):
"""send() returns (http_status, retry_after_seconds_or_None)."""
global budget
for attempt in range(max_attempts):
if reject_locally():
raise RuntimeError("throttled client-side, request not sent")
status, retry_after = send()
history.append((time.monotonic(), status not in (429, 503)))
if status != 503:
if status < 500:
budget = min(10.0, budget + 0.1) # ~10% retry ratio
return status # includes 429: the quota follows this caller
if attempt == max_attempts - 1 or budget < 1.0:
return status # out of attempts or budget: surface the 503
budget -= 1.0
# Full jitter, but never retry sooner than the server asked.
delay = random.uniform(0, min(cap_s, base_s * 2 ** attempt))
if retry_after is not None:
delay = max(delay, retry_after + random.uniform(0, 0.5))
time.sleep(delay)
return status
A minimal client wrapper that combines adaptive client-side throttling, a retry budget, and exponential backoff with full jitter. It retries 503s while honoring Retry-After and returns 429s to the caller instead of retrying them.
When should you avoid load shedding, and what are the alternatives?
Shedding is not always the right primary defence. A low-traffic internal service with a handful of known callers gains little from adaptive limits. A fixed timeout and a conversation with the owning team usually fix overload faster. In async pipelines, consumer-side backpressure already does the work. A consumer that pulls only what it can process cannot be overrun, so the useful signal is queue lag, not rejection rate. Workloads where every request must eventually complete, such as payments ingestion, need the opposite of shedding. Accept the request durably, acknowledge it, and process it at whatever rate downstream allows.
| Mechanism | Strength | Main trade-off |
|---|---|---|
| Autoscaling | Adds real capacity | Too slow for sudden spikes. Shedding can hide demand from scaling signals. |
| Rate limiting and quotas | Enforces fairness per caller | Static limits ignore actual server health |
| Circuit breakers | Stop calls to a failing dependency | Protect the caller, not the overloaded callee |
| Bulkheads | Isolate one workload's failure from others | Partitioned capacity sits idle when one pool is quiet |
| Durable async queues | Nothing is dropped | Latency becomes unbounded. Consumers need their own limits. |
| Global or distributed limiters | See fleet-wide state | Add coordination cost and new failure modes |
Per-instance limits ignore global state and uneven load balancing. One hot instance sheds while its neighbours sit idle.
If you add shedding, export both the rejections and the offered load so autoscaling still sees real demand. Then tune the threshold in both directions. A limit that is too eager wastes capacity during harmless bursts. A limit that is too lax lets latency collapse before it fires.
How do you test and observe load shedding, and what does it cost to build?
Testing past saturation
Steady-state load tests at expected peak prove very little. Push to two or three times measured capacity, and include a step change from idle to overload, because adaptive limits behave differently under a sudden spike than under a gradual ramp. The test passes if goodput, meaning successful responses that finished within their deadline, stays roughly flat once saturation is reached. It fails if goodput falls as offered load rises. Also measure how much a rejection costs. If a 429 takes as long as a real request, or allocates as much memory, shedding only relocates the overload.
What to graph
- Rejected-request rate, broken down by reason: concurrency limit, expired deadline, or priority
- In-flight concurrency
- The current adaptive limit, so you can see it converge or oscillate
- Queue wait time, recorded separately from service time
- Goodput plotted against raw throughput
If wait time climbs while service time stays flat, requests are queuing, not running slowly.
Cost
The upfront work includes admission middleware, deadline propagation through every hop, client retry policies with backoff and retry budgets, dashboards, and several rounds of tuning thresholds against real traffic. Priority classes add complexity that never goes away: someone has to own the classification, and it drifts as endpoints change.
The return is fewer incidents where one saturated service cascades into a multi-service outage, and clearer signals during incidents, because rejections by reason tell you where the pressure is. If adaptive limits ship as shared middleware, teams also stop hand-tuning a static limit for each service after every capacity change.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to microservice load shedding, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.