Microservice Load Shedding: Adaptive Admission Control Under Overload

Why overload is a queueing problem, not a rate problem

Little's Law states that the average number of requests in flight equals throughput multiplied by latency. A service completing 500 requests per second at 20 ms each holds about 10 requests in flight. When arrivals stay below capacity, that number is stable. Overload is the condition where it is not.

When the arrival rate exceeds the service rate, the excess goes into a queue, and the queue keeps growing for as long as the imbalance lasts. Illustrative estimate: assume a capacity of 500 req/s and arrivals of 600 req/s. The queue grows by 100 requests every second. After 10 seconds, 1,000 requests are waiting, so a new arrival waits about 2 seconds before any work on it starts.

Goodput collapses because callers have deadlines. If clients time out at 1 second, the server still completes 500 req/s, but it spends that capacity on requests whose callers have already given up. Useful throughput falls toward zero, and client retries add more arrivals to the queue.

The fix is to bound in-flight work. That keeps latency near its uncontended value and rejects excess requests early, while rejecting them is still cheap. Three related mechanisms do this:

This matters most for backend and platform engineers running request-serving services whose capacity shifts with deploys, cache state, noisy neighbours or the health of downstream services.

Who should use adaptive admission control

Adaptive admission control is for teams that own synchronous, request-serving services and are responsible for how those services behave when demand exceeds capacity. In practice that means:

• Backend service owners whose services sit on user-facing paths with latency targets, where a slow success is worth less than a fast, retryable rejection.
• Platform and infrastructure teams that maintain shared gateways, sidecars or RPC libraries. They are best placed to build the limiter once and give every service the same semantics and metrics.
• Teams running shared or multi-tenant services, where one caller's burst can consume capacity that other callers depend on.
• Teams with deep call graphs or fan-out, where a slowdown in one dependency spreads upstream unless something bounds the work waiting on it.
• SREs and on-call engineers who have been through retry storms or cascading failures and want overload to degrade service gradually instead of taking it down.

It is a weaker fit for teams whose workloads are mostly asynchronous batch jobs, whose traffic never approaches saturation, or who do not yet have per-endpoint latency and rejection metrics. The section on when to avoid load shedding covers those cases in more detail.

Why static rate limits and CPU thresholds fail, and what the alternatives are

A requests-per-second limit encodes a capacity measurement taken on one day, against one build and one traffic mix. That number drifts. A deploy adds a serialization step. A cache flush drops hit rates and multiplies database reads. A noisy neighbour steals CPU or disk bandwidth. A downstream dependency slows from 20 ms to 200 ms. The same RPS figure is then either far too generous or needlessly strict.

CPU thresholds fail differently. Utilisation is averaged over a sampling window, so it reports saturation after queues have already formed. It also stays low while a service is stuck waiting on I/O. A service whose threads are all blocked on a slow database can sit at 30% CPU while its latency explodes.

How it works: the control loop in brief

Adaptive load shedding is a feedback loop that runs inside each service instance on every request:

1. Measure. For each completed request, the service records its latency and how many requests were in flight. It also tracks how long admitted requests waited in the queue before work began.
2. Estimate capacity. A limiter compares current latency with a baseline that represents the service without queueing. Latency close to baseline means there is spare capacity, so the concurrency limit can rise. Latency well above baseline means requests are queueing, so the limit falls.
3. Decide at the door. When a request arrives, the admission controller checks the in-flight count against the current limit. It also checks the request's deadline and priority, and how long requests are currently waiting. Then it admits, queues or rejects the request.
4. Reject cheaply and visibly. A rejected request receives a fast 503 or 429 with a Retry-After hint before the body is parsed or any downstream call is made. The rejection is labelled with the layer and reason.
5. Clients cooperate. Callers respond to rejections by backing off with jitter and spending a bounded retry budget, so shedding lowers total load instead of turning into a retry storm.

The sections below cover each step: the limiter algorithms, the admission middleware and queue policy, and the deadline, priority and client-side mechanisms that keep the loop stable.

How adaptive concurrency limits work

The limiter treats in-flight requests the way TCP treats its congestion window. It probes upward while latency stays flat and backs off when queueing shows up. Concurrency is the variable to control because, by Little's law, extra in-flight work beyond what the service can process concurrently only lengthens the queue.

AIMD raises the limit by a small constant after each successful window and multiplies it by a factor such as 0.9 on a timeout, rejection or latency breach. The asymmetry is intentional. Slow growth avoids overshooting, and fast cuts drain the queue before deadlines expire.

Gradient limiters compare the observed latency against a tracked minimum, which represents the latency of the service with no queue. The ratio baseline / observed scales the limit down as queueing grows. A small headroom term, often near the square root of the limit, lets the limit keep probing upward.

Worked example (illustrative estimate). Assume a baseline of 20 ms, an observed latency of 40 ms and a current limit of 100.

• The gradient is 0.5, so the raw target is 100 × 0.5 + 10 = 60.
• With a smoothing weight of 0.2, the new limit is 0.8 × 100 + 0.2 × 60 = 92.
• That result is then clamped to the configured bounds, for example 10 to 500.

The trade-offs:

• Oscillation and slow convergence. Heavy smoothing settles slowly. Light smoothing chases noise.
• Aggressive tuning. The limiter flaps and sheds traffic a healthy service could have absorbed.
• Conservative tuning. The limiter sheds only after the queue has already blown through deadlines.
• Baseline drift. When the request mix shifts toward expensive endpoints, the old minimum makes normal latency look like congestion. Limiters therefore decay or periodically re-probe the baseline, or keep separate limits per request class.

Copyimport math

class GradientLimiter:
    # Single-threaded sketch; a real limiter needs a lock or atomics.
    def __init__(self, initial=100, min_limit=10, max_limit=500,
                 smoothing=0.2, baseline_decay=0.01):
        self.limit = float(initial)
        self.min_limit, self.max_limit = min_limit, max_limit
        self.smoothing = smoothing            # weight given to each new target
        self.baseline_decay = baseline_decay  # lets the baseline follow mix changes
        self.baseline_ms = None               # tracked no-queue latency
        self.in_flight = 0

    def try_acquire(self):
        if self.in_flight >= int(self.limit):
            return False                      # shed: caller returns 503/429 fast
        self.in_flight += 1
        return True

    def on_complete(self, observed_ms):
        self.in_flight -= 1
        if self.baseline_ms is None or observed_ms < self.baseline_ms:
            self.baseline_ms = observed_ms
        # Ratio < 1 means queueing. The floor at 0.5 caps how hard one sample cuts.
        gradient = max(0.5, min(1.0, self.baseline_ms / observed_ms))
        headroom = math.sqrt(self.limit)      # keeps probing upward when healthy
        target = self.limit * gradient + headroom
        smoothed = (1 - self.smoothing) * self.limit + self.smoothing * target
        self.limit = max(self.min_limit, min(self.max_limit, smoothed))
        # Drift the baseline slowly upward so a costlier request mix is not
        # read as permanent congestion.
        self.baseline_ms += self.baseline_decay * (observed_ms - self.baseline_ms)

lim = GradientLimiter()
lim.baseline_ms = 20.0
lim.in_flight = 1
lim.on_complete(observed_ms=40.0)
print(round(lim.limit, 1))  # 92.0  (target 60, smoothed toward it, within 10..500)

A minimal gradient concurrency limiter that applies the baseline/observed ratio, sqrt headroom, exponential smoothing, min/max clamping and slow baseline decay, and reproduces the worked example's step from 100 to 92.

Implementing admission control at the service edge

The admission decision belongs at the first point where the service knows its own concurrency, and before any expensive work. A request rejected after parsing a large JSON body, calling an auth service, and taking a connection from the database pool has already consumed most of what it would cost to serve.

Copyclient -> [gateway]* -> [mesh sidecar]* -> [admission controller]* -> [bounded queue]* -> handlers -> DB pool* / downstreams
                                            in-flight limit           wait-time + deadline check
                                            (* = shedding point)

Queue management

Bound the queue, but shed based on time waited, not queue length. This is the CoDel approach. Ten queued requests are harmless when each takes 2 ms and fatal when each takes 2 s. If the minimum wait over a recent interval stays above a target, the queue is standing rather than absorbing a burst, so start dropping.

Under sustained overload, switch to adaptive LIFO. Newest-first serves requests whose clients are still waiting, while the oldest entries are the ones most likely to have given up. Use FIFO otherwise to keep ordering fair. On dequeue, drop any request whose propagated deadline has already passed.

The middleware shown below checks in-flight count and wait time. It returns 503 with Retry-After before reading the request body.

Double limiting

When the gateway, mesh, service and DB pool each shed independently, their limits compound. The tightest layer sets throughput, and retries at outer layers multiply load. Make one layer authoritative, usually the service, and set the others as loose backstops. Tag every rejection with the layer that made it, for example in a response header and a metric label, so operators can tell which limit actually fired.

Copyimport http from "node:http";

const MAX_IN_FLIGHT = 64;     // concurrency the service can actually sustain
const MAX_QUEUE = 128;        // hard bound; never queue unboundedly
const MAX_WAIT_MS = 50;       // shed on time waited, not on queue length
type Item = { enq: number; deadline: number; run: () => void; shed: (r: string) => void };
let inFlight = 0;
const queue: Item[] = [];

function reject(res: http.ServerResponse, reason: string) {
  // Cheap rejection: headers only, request body is never read or parsed.
  res.writeHead(503, { "Retry-After": "1", "X-Shed-Layer": "service-admission", "X-Shed-Reason": reason, Connection: "close" });
  res.end();
}

function drain() {
  while (inFlight < MAX_IN_FLIGHT && queue.length > 0) {
    // Adaptive LIFO: newest-first only when the queue is filling up.
    const item = queue.length > MAX_QUEUE / 2 ? queue.pop()! : queue.shift()!;
    const now = Date.now();
    if (now > item.deadline) { item.shed("deadline"); continue; } // client gave up
    if (now - item.enq > MAX_WAIT_MS) { item.shed("wait"); continue; } // standing queue
    inFlight++;
    item.run();
  }
}

export function admit(handler: http.RequestListener): http.RequestListener {
  return (req, res) => {
    if (queue.length >= MAX_QUEUE) return reject(res, "queue-full");
    // Deadline propagated by the caller as epoch ms; default to 1s from now.
    const hdr = Number(req.headers["x-request-deadline"]);
    const deadline = Number.isFinite(hdr) && hdr > 0 ? hdr : Date.now() + 1000;
    const run = () => {
      res.on("close", () => { inFlight--; drain(); }); // release slot however the request ends
      handler(req, res); // deserialization, auth and DB access happen only after admission
    };
    queue.push({ enq: Date.now(), deadline, run, shed: (r) => reject(res, r) });
    drain();
  };
}

A Node.js admission middleware that enforces an in-flight limit with a bounded queue, switches to LIFO when the queue fills, sheds requests that waited too long or whose deadline has passed, and returns a header-only 503 with Retry-After and a label naming the layer that rejected the request.

Deadlines, priorities and client cooperation

Admission control makes better decisions when each request carries context. Propagate a deadline, not a timeout. Each hop forwards the absolute remaining budget minus its own expected overhead. If the budget is smaller than the downstream's typical latency, reject at once, because the result would arrive after the caller stopped waiting. When the caller cancels or the deadline passes, cancel in-flight downstream calls and drop queued work for that request.

Criticality tagging

Assign tags server-side from caller identity. Clients will mark everything as critical if they are allowed to. Add per-tenant sub-limits so one noisy client cannot take the whole concurrency limit.

Client duties

• Retry budgets: cap retries as a ratio of successful requests (for example, 10%) instead of a fixed count per request.
• Backoff: use exponential backoff with full jitter.
• Server signals: honor 429/503 responses and Retry-After.
• Client-side throttling: reject locally with a probability that rises as the server's acceptance ratio falls.

Trade-offs

LIFO queues keep latency low for fresh requests but can starve older ones. Priority tags drift across teams unless they are enforced centrally.

Without retry budgets, shedding multiplies load across layers. Consider an illustrative estimate: if three layers each retry three times, one user request can turn into up to 4³ = 64 attempts at the bottom layer.

Copyimport random
import time

class RetryBudget:
    """Allow retries only up to a fraction of recent successful requests."""
    def __init__(self, ratio=0.1, min_tokens=10.0):
        self.ratio = ratio            # e.g. 10% extra load from retries
        self.tokens = min_tokens      # small reserve so low traffic can still retry
        self.max_tokens = min_tokens * 10

    def on_success(self):
        # Each success earns a fraction of a retry token.
        self.tokens = min(self.max_tokens, self.tokens + self.ratio)

    def try_spend(self):
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True
        return False                  # budget exhausted: fail fast, do not retry

def call_with_retries(send, budget, base=0.1, cap=5.0, max_attempts=4):
    for attempt in range(max_attempts):
        status, retry_after = send()  # retry_after: seconds parsed from header, or None
        if status < 400:
            budget.on_success()
            return status
        if status not in (429, 503) or not budget.try_spend():
            return status             # non-retryable, or no budget left
        if attempt == max_attempts - 1:
            break                     # no attempts left; don't sleep for nothing
        if retry_after is not None:
            delay = retry_after       # server's hint wins over our own schedule
        else:
            # Full jitter: uniform in [0, min(cap, base * 2^attempt)].
            delay = random.uniform(0, min(cap, base * 2 ** attempt))
        time.sleep(delay)
    return status

A client retry budget that earns retry tokens from successes, retries only on 429/503, honors Retry-After, and otherwise waits with capped exponential backoff and full jitter.

When to use load shedding and when to avoid it

Adaptive admission control pays off where capacity is unpredictable and the cost of queueing is high. Shared services fit this: their effective capacity shifts with noisy neighbors, cache hit rates and downstream health. Fan-out architectures are another case, because one slow leaf stalls every parent waiting on it. Latency-sensitive user paths benefit because a fast rejection is more useful than a slow success. Multi-tenant systems need it too, since one tenant's burst should not consume everyone's concurrency.

It is overkill elsewhere. A low-traffic internal tool rarely reaches saturation, and a limiter that never trips still has to be maintained. Batch pipelines are better served by durable queues and pull-based consumers, where workers take only what they can process and backpressure comes from the design. When load and capacity are both stable, a fixed concurrency cap set from a load test is easier to reason about. If a team cannot see per-endpoint latency, rejection rates and limit values, it cannot tell a well-tuned limiter from one that sheds healthy traffic.

Shedding protects a service but adds no capacity. Pair it with autoscaling or deliberate capacity planning. Be careful which signal the autoscaler reads: rejected requests consume little CPU, so a CPU-driven scaler can read heavy shedding as falling demand and scale in. Scale on offered load or rejection rate instead.

Per-instance limits also do not add up to a global view. Each replica knows only its own load. Coordinating limits across replicas brings back shared state, propagation lag and a new dependency that can fail during the incident you are trying to survive.

Observability, testing and impact on engineering cost

Because each replica decides on its own, its decisions have to be visible per replica before you aggregate them.

Signals to track

• In-flight count and current limit, plotted together. A limit that keeps shrinking while in-flight stays pinned to it means the service is degrading, even if errors look flat.
• Queue wait time, kept separate from service time. Rising wait time is the earliest sign of saturation.
• Shed rate by priority. Low-priority traffic should be shed first. Critical requests being rejected is a page-worthy event.
• Goodput, meaning successful responses completed within their deadline. Raw throughput can stay high while every response arrives too late to use.
• Rejections as their own status, never counted as failures. If sheds feed the error-rate alert, it pages on correct behavior, and engineers learn to ignore it.

Validation

Run load tests that ramp well past saturation and hold there. The pass condition is that goodput plateaus instead of collapsing and latency for admitted requests stays bounded. Then inject slowness into a dependency and confirm that the limit drops on its own, without anyone retuning it.

Engineering cost

The upfront work is real. It covers admission middleware, consistent priority tagging, propagating deadlines through every hop, and client libraries that honor rejections and back off. In return:

• Nobody has to retune static limits after each deploy.
• Overload incidents are shorter and less frequent.
• Metrics show which layer rejected the traffic.
• Retry storms need less firefighting.

Build the limiter once, in a shared library or sidecar. If every team writes its own, you get inconsistent semantics and metrics nobody can compare.

How adaptive load shedding saves engineering time

Most of the time saved is time engineers would otherwise spend maintaining capacity numbers by hand and fighting overload during incidents. Concretely:

• No limit retuning after changes. With static RPS limits, every deploy, dependency change or shift in traffic mix raises the question of whether the configured number still holds. Someone has to re-run a load test or guess. An adaptive limiter re-derives its limit continuously from latency, so the routine work drops to checking that the limiter's bounds and smoothing still make sense.
• Less manual intervention during incidents. Without shedding, the standard response to overload is manual: lower a limit, drain traffic, restart saturated instances, or wait for a retry storm to subside. When the service bounds its own in-flight work and clients obey retry budgets, the system degrades and recovers without anyone touching it. On-call engineers can spend the incident finding the cause instead of keeping the service alive.
• Faster diagnosis. Rejections labelled with layer and reason, alongside in-flight, limit, wait-time and goodput metrics, answer the first incident questions directly: is this overload, where is it being shed, and which priority class is affected. Without those signals, engineers have to reconstruct the same picture from timeouts and error logs.
• Smaller blast radius. Cascading failures turn a single-service problem into a multi-team incident. Deadline propagation, cancellation and per-tenant sub-limits keep a slow dependency or noisy tenant from pulling in owners of unrelated services.
• Less alert noise. Counting sheds as their own status, instead of as errors, removes a class of pages that fire on correct behavior. That reduces the time spent triaging alerts that need no action.
• One implementation instead of many. A shared library or sidecar means one team designs, tests and documents admission behavior once. Every other team configures it instead of reimplementing it.

Illustrative estimate, with assumed inputs: assume an organisation has 8 services protected by static limits, ships each one twice a month, and spends an assumed 1 engineer-hour per release re-validating the limit. That is 16 engineer-hours a month spent on limit maintenance, before counting any incident time. Assume further that adaptive limits remove most of that re-validation and that the shared library needs some ongoing upkeep. The net saving then depends on how that upkeep compares with the hours removed. Replace these assumptions with your own release cadence and incident history before using the figure.

The saving appears only if the limiter is trusted. Teams that cannot see what it is doing tend to bypass it or wrap it in manual overrides, which puts the maintenance work back. The observability and load-testing work above is part of the cost of getting the time back, not optional extra.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice load shedding, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.