What is load shedding in microservices and why does overload collapse goodput?
Load shedding is a service deciding, locally and cheaply, to reject work it cannot finish in time. The point is not to protect the server for its own sake. The point is to spend finite capacity only on requests that can still succeed.
Overload is a queueing problem. If requests arrive faster than the service completes them, the queue grows for as long as the excess lasts. Every queued request adds waiting time, so latency climbs until it passes the caller's timeout. The server keeps working through the backlog, but the callers have already given up and often retried. Throughput stays high while goodput, the rate of responses that arrive in time to be useful, collapses toward zero.
Little's Law gives you the arithmetic: concurrency = throughput × latency.
Illustrative estimate. Assume these inputs:
- 50 concurrent workers
- 100 ms per request
- arrivals of 600 req/s
- a 1 s client timeout
Capacity is 50 / 0.1 s = 500 req/s, so 100 req/s accumulate in the queue. After 5 seconds, 500 requests are queued, which adds 1 s of wait. From that point on, a newly arriving request waits past its deadline before any work starts, yet the server still spends all 500 req/s of capacity on it. Shedding the excess 100 req/s at admission would have kept about 500 req/s of on-time responses.
Engineers running latency-sensitive services need this protection most, particularly services that see bursty traffic or depend on fragile downstream systems, where a slowdown in one place can turn into a queue in another.
Where should requests be rejected to keep shedding cheap?
A rejection should cost a small fraction of what serving the request would cost. If turning a request away requires parsing its body, loading user state or calling a dependency, then shedding consumes the capacity it was supposed to protect. That is why admission checks belong before expensive work.
client
│
▼
[ingress / edge proxy] ◄── (1) shed by route, tenant, global budget
│
▼
[service admission] ◄── (2) concurrency/deadline check,
│ before deserialization and business logic
▼
[handler] ──► [internal queue / worker pool] ◄── (3) bounded queue,
│ reject on full or stale
▼
downstream calls ──► (same three points in each downstream service)
Ingress (1) is the cheapest point because it holds no application state, but it cannot see which backend instance is actually struggling. Service admission (2) knows local concurrency and latency, and it can reject using only headers and connection metadata. Internal queues (3) catch overload that builds up inside the process, such as a saturated thread pool or database connection pool.
Rejecting late in a deep call graph is the expensive case. When a request has already passed through three services and fails at the fourth, every upstream service has spent CPU, memory and connection slots on it, and all of that work is discarded. Shed as close to the entry point as the available signal allows.
Exempt health checks, admin endpoints and control-plane traffic, or reserve capacity for them. If the shedder rejects these, orchestrators may kill instances that are still healthy, and operators lose the controls they need to intervene.
client
│
▼
┌────────────────────────┐
│ ingress / edge proxy │◄── (1) shed by route, tenant, global budget
└───────────┬────────────┘
▼
┌─ service A ──────────────────────────────────────────────────┐
│ health / admin / control plane ──► exempt or reserved slots │
│ │
│ ┌─────────────────┐ │
│ │ admission check │◄── (2) concurrency/deadline check │
│ └────────┬────────┘ using headers only, no body parse │
│ ▼ │
│ deserialize, load state, business logic (expensive) │
│ ▼ │
│ ┌─────────────────┐ │
│ │ queue / workers │◄── (3) bounded queue; reject when │
│ └────────┬────────┘ full or request is stale │
└───────────┼──────────────────────────────────────────────────┘
▼
service B ───► service C ───► service D ✗ rejected late
(1)(2)(3) (1)(2)(3) (2) or (3)
└─── CPU, memory, connection slots spent upstream: lost ───┘
rejection cost: (1) lowest < (2) low < (3) higher < late
Shedding points along the request path: the edge, the service admission check before deserialization, and bounded internal queues, with control-plane traffic exempted and late rejection deep in the call graph discarding all upstream work.
How does adaptive concurrency work compared with static rate limits?
A static cap records capacity at one moment. A limit tuned on last year's instance type, before a dependency slowed down or a release added CPU cost per request, becomes too loose to protect the service or so tight that it sheds traffic the service could have handled. RPS caps also ignore per-request cost. Concurrency caps follow cost better, because slow requests hold slots longer, but someone still has to pick the right number.
Adaptive limiters make the in-flight limit the control variable and use latency as feedback. As a service nears saturation, requests queue and latency rises before throughput drops, so latency gives early warning.
AIMD. The admission check rejects a request when in-flight count is at or above the current limit. Otherwise it increments in-flight, runs the request, records latency and outcome, and decrements in-flight. Each healthy completion raises the limit slightly, for example by 1/limit. A timeout or latency above a threshold multiplies the limit by a backoff factor such as 0.9.
Gradient. The limiter tracks a baseline latency, usually the recent minimum, and compares it with smoothed current latency. When the ratio is near 1, the limit grows. As the ratio falls, the limit shrinks in proportion.
Stability depends on three things:
- Smoothing: use an exponentially weighted moving average or a windowed percentile instead of raw samples.
- Bounds: a minimum limit keeps traffic flowing so the limiter can probe for recovery. A maximum limit caps memory and connections.
- Baseline resets: reset the baseline periodically so a permanently slower dependency does not pin the limit low.
| Approach | Signal | Tuning effort | Adaptivity | Failure modes |
|---|---|---|---|---|
| Static RPS | Request count | Per deploy | None | Ignores cost; goes stale |
| Static concurrency | In-flight count | Per deploy | Indirect | Wrong number as capacity changes |
| AIMD | Timeouts, latency threshold | Threshold, backoff | Reactive | Sawtooth; threshold drift |
| Gradient | Latency vs baseline | Smoothing, bounds | Continuous | Stale baseline; noise sensitivity |
import threading, time
class AIMDLimiter:
def __init__(self, initial=20, min_limit=5, max_limit=200,
backoff=0.9, latency_threshold_s=0.250):
self.limit = float(initial)
self.min_limit = min_limit # floor: keep probing for recovery
self.max_limit = max_limit # ceiling: cap memory and connections
self.backoff = backoff # multiplicative decrease factor
self.threshold = latency_threshold_s
self.in_flight = 0
self.lock = threading.Lock()
def try_acquire(self):
# Admission check: shed when in-flight count reaches the current limit.
with self.lock:
if self.in_flight >= int(self.limit):
return False
self.in_flight += 1
return True
def release(self, latency_s, timed_out):
with self.lock:
self.in_flight -= 1
if timed_out or latency_s > self.threshold:
# Overload signal: back off multiplicatively.
self.limit = max(self.min_limit, self.limit * self.backoff)
else:
# Healthy completion: grow by ~1 per full window of requests.
self.limit = min(self.max_limit, self.limit + 1.0 / self.limit)
def handle(limiter, request, process, timeout_s=1.0):
if not limiter.try_acquire():
return 503 # reject fast instead of queueing
start = time.monotonic()
timed_out = False
try:
return process(request, timeout_s)
except TimeoutError:
timed_out = True
return 504
finally:
limiter.release(time.monotonic() - start, timed_out)
A minimal AIMD concurrency limiter around an admission check: rejects at the limit, adds 1/limit on healthy completions, multiplies by 0.9 on timeouts or slow responses, bounded by min and max limits.
How do you implement deadlines, time-bounded queues and priority-aware shedding?
A queue capped at 1,000 entries says nothing about how long the last entry has waited. Bound queues by time as well as length. Attach an absolute deadline to each request at the edge. Propagate the remaining budget downstream rather than a fixed per-hop timeout, and subtract local work before each outbound call. At dequeue time, check the deadline. If it has passed, or if the remaining time is shorter than the expected service time, drop the request before doing any work. The caller has already given up or soon will, so serving it only consumes capacity.
FIFO is fair under normal load. Under overload it serves the oldest requests first, and those are the ones most likely to expire. Switch to LIFO, or to CoDel-style dropping (shed when the minimum queueing delay over an interval stays above a target), only while overload is detected. The trade-off is that requests at the back of the queue starve and tail latency rises sharply for those callers. Keep the mode temporary and revert automatically.
Priority-aware admission
Classify traffic into four groups:
- Interactive, user-facing: shed last.
- Background and batch: shed first, but guarantee a minimum share so this work cannot starve indefinitely.
- Retries: rank below first attempts, because they amplify the overload.
- Health checks: usually exempt them. Shedding health checks can cause load balancers to eject healthy instances and shift load onto the remaining ones.
Give each class its own admission threshold as a fraction of the adaptive concurrency limit. For example, batch might be admitted only below 50% utilization and interactive up to 100%. Carry the criticality in request metadata so that every downstream service sheds the same requests instead of making independent, inconsistent decisions. The code sketch shows both the dequeue-time deadline check and the per-class thresholds.
import time
from enum import IntEnum
class Priority(IntEnum): # read from request metadata, set at the edge
HEALTH = 0
INTERACTIVE = 1
RETRY = 2
BATCH = 3
# Share of the adaptive concurrency limit each class may fill before it is shed.
ADMIT_BELOW = {Priority.INTERACTIVE: 1.0, Priority.RETRY: 0.75, Priority.BATCH: 0.5}
MIN_BATCH_SLOTS = 2 # guaranteed share so batch work never starves completely
def admit(prio, in_flight, batch_in_flight, limit):
if prio == Priority.HEALTH:
return True # exempt: shedding probes gets healthy instances ejected
if prio == Priority.BATCH and batch_in_flight < MIN_BATCH_SLOTS:
return True
return in_flight < limit * ADMIT_BELOW[prio]
def next_runnable(queue, expected_service_s, overloaded):
"""queue is a collections.deque; req.deadline is a local time.monotonic() value
computed on arrival from the remaining budget the caller propagated."""
while queue:
# FIFO normally; LIFO only while overload is detected (unfair to old entries).
req = queue.pop() if overloaded else queue.popleft()
remaining = req.deadline - time.monotonic()
if remaining < expected_service_s:
req.reject("deadline_exceeded") # caller has given up or will before we finish
continue
return req
return None
def outbound_budget(req, local_overhead_s):
# Propagate what is left, minus our own remaining work, not a fixed per-hop timeout.
return max(0.0, req.deadline - time.monotonic() - local_overhead_s)
A sketch of per-priority admission thresholds with a batch minimum share and exempt health checks, a dequeue-time deadline check that switches to LIFO only under overload, and remaining-budget propagation for downstream calls.
How should services signal rejection and how should clients respond?
A rejection only protects the server if it is cheap and unambiguous. Return it before parsing the body or touching downstream dependencies, and make the reason explicit. A timeout or reset connection looks like a network fault, so the client cannot tell it should slow down.
| Rejection type | HTTP | gRPC | Retryable? | Expected client behavior |
|---|---|---|---|---|
| Instance overloaded | 503 + Retry-After |
UNAVAILABLE |
Yes, on another instance | Back off with jitter and spend from the retry budget |
| Caller over quota | 429 + Retry-After |
RESOURCE_EXHAUSTED |
Only after the stated delay | Honour Retry-After and do not fail over to evade the limit |
| Deadline already expired | 503 or 504 | DEADLINE_EXCEEDED |
No | Fail upward, because the caller has already given up |
Shedding fails if every rejected request comes straight back. Clients need three controls:
- Retry budget. Cap retries at a small fraction of successful requests, so retry traffic stays bounded no matter how many calls fail.
- Exponential backoff with full jitter. Jitter spreads retries out so clients do not fire again in synchronized waves.
- Circuit breaker. When the rejection rate stays high, stop sending requests for a cooling period so the server can recover.
Load balancers should count rejections as a load signal and steer new requests toward instances with spare capacity. They should not eject the instance that sent the rejection.
Health checks need the same distinction. An instance that is shedding is protecting itself, which is correct behavior, and it is still healthy. If liveness or readiness probes fail under load, the orchestrator removes that instance. Its traffic then moves to the remaining instances, which become overloaded in turn, and the failure spreads through the fleet. Keep probes on a path that bypasses the concurrency limiter, and have them report only on whether the process can do work at all.
When should you avoid load shedding, and what are the alternatives?
Shedding is a safety valve, not a capacity plan. If a service rejects traffic every weekday peak, the limiter is masking chronic under-provisioning, so alert on sustained rejection rate as well as errors. It suits work that must never be dropped, such as payment captures or audit writes, only when rejected requests land in a durable queue. On low-traffic services, a few rejections hit a large share of users, and the limiter has too few samples to estimate anything useful.
Adaptive limiters assume latency tracks local load. That assumption breaks when latency is noisy, when request costs are bimodal (a cheap lookup and a heavy export share one limit), or when GC pauses inflate every sample at once. In those cases the limit oscillates or settles well below real capacity. A slow dependency also raises local latency, so the service sheds its own traffic for someone else's fault.
| Mechanism | What it adds | Cost or limitation |
|---|---|---|
| Autoscaling | Real capacity | Takes minutes; shedding covers the gap until it arrives |
| Global rate limiting | Fleet-wide view of quotas | Coordination latency; a new failure point |
| Durable queue with backpressure | No dropped work | Higher latency; the queue needs its own limits |
| Bulkheads | Isolate pools per workload | Idle capacity in each partition |
| Circuit breakers | Stop calling a failing dependency | Need tuning; can trip on transient blips |
Priority tiers invite gaming. If callers set their own criticality, every caller marks its traffic critical. Assign tiers server-side from authenticated caller identity, cap each caller's share of the top tier, and review classification changes the same way you review config changes.
How do you observe, test and justify the engineering cost of load shedding?
Emit five signals per service instance:
- Rejected requests, labeled by
reason(limit reached, deadline expired, queue timeout) andpriority. Without labels you cannot tell healthy shedding of batch traffic from rejecting critical calls. - Current concurrency limit, so you can see the adaptive algorithm moving.
- In-flight count, compared against the limit.
- Queue wait time as a histogram, because averages hide the tail that breaks deadlines.
- Goodput: requests completed successfully within their deadline.
Treat rejections as a capacity signal. Shedding during a short spike is correct. Shedding that continues for hours means the service is under-provisioned. An example alert: fire when critical-tier rejections exceed 0.1% of critical requests for 10 minutes, or when any-tier rejections exceed 5% for 30 minutes. Route the alert to whoever owns capacity, not only to on-call.
Validate with load tests that ramp traffic past measured capacity and inject latency into a dependency. Plot goodput against offered load. It should rise, then stay flat. If it falls, the service is still doing work it later throws away.
The cost is real. You have to integrate the mechanism, then tune smoothing and limit bounds over several iterations. Bounds set too tight will reject traffic the service could have served, so compare the limit with observed latency headroom. The return is fewer cascading-failure incidents and no hand-retuning of static limits after each deployment. A shared middleware library keeps metric names and behavior the same across services, so one dashboard and one alert definition work for all of them.
# Prometheus rules. Metric names come from the shared load-shedding middleware:
# loadshed_requests_total{service,priority} all arriving requests
# loadshed_rejected_total{service,priority,reason} shed requests
# loadshed_goodput_total{service,priority} succeeded within deadline
# loadshed_concurrency_limit, loadshed_inflight gauges per instance
# loadshed_queue_wait_seconds histogram
groups:
- name: load-shedding
rules:
# Goodput as requests/sec, for plotting against offered load in tests.
- record: service:loadshed_goodput:rate5m
expr: sum by (service) (rate(loadshed_goodput_total[5m]))
# Shedding critical traffic is never steady state: page quickly.
- alert: CriticalTrafficShed
expr: |
sum by (service) (rate(loadshed_rejected_total{priority="critical"}[5m]))
/ sum by (service) (rate(loadshed_requests_total{priority="critical"}[5m]))
> 0.001
for: 10m
labels:
severity: page
annotations:
summary: "{{ $labels.service }} is rejecting >0.1% of critical requests"
# Sustained shedding of any tier means under-provisioning: route to capacity owner.
- alert: SustainedLoadShedding
expr: |
sum by (service) (rate(loadshed_rejected_total[5m]))
/ sum by (service) (rate(loadshed_requests_total[5m]))
> 0.05
for: 30m
labels:
severity: ticket
team: capacity-planning
annotations:
summary: "{{ $labels.service }} has shed >5% of traffic for 30m; add capacity"
Prometheus recording and alerting rules that track goodput and treat critical-tier and sustained rejections as capacity signals, with metric names standardized by a shared middleware library.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to load shedding microservices, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.