Bulkhead Pattern in Microservices: Sizing Pools to Contain Failures

What a bulkhead is and the failure it prevents

A bulkhead is a bounded, separately owned pool of a scarce resource, assigned to one dependency or one class of work. The resource can be threads, concurrency permits, database or HTTP connections, or a share of event-loop capacity. The name comes from ship hulls, which are divided into watertight compartments so that one breach floods one section instead of the whole vessel.

The failure it prevents rarely starts with a dependency going down. A dependency that fails fast returns an error in milliseconds, and the thread or connection goes straight back to the pool. The real damage comes from a dependency that is slow. Suppose a recommendations service starts taking 30 seconds to respond. Every request to it holds a worker thread or pooled connection for that whole time. Because the requests are not failing, nothing frees those resources. Within seconds the shared pool is full of calls waiting on one sick service. Requests for checkout or authentication, whose downstreams are healthy, then queue behind them and time out. One degraded dependency has taken down the whole service.

Isolation only works if the bulkhead sits on the resource that saturates first. Limiting threads per dependency does nothing if every dependency shares one connection pool that runs out sooner. Each bounded pool also needs a bounded queue in front of it. An unbounded queue turns "pool full" into unlimited waiting, and that brings back the same pile-up the bulkhead was meant to stop.

Any team whose service fans out to several downstreams of uneven reliability needs this pattern.

Copy WITHOUT BULKHEADS: one shared pool for every dependency
 +-----------+     +------------------------------+     +-----------------+
 | incoming  |---->| shared pool, all dependencies|---->| recommendations |
 | requests  |     | [R][R][R][R][R][R][R][R][R]  |     | SLOW: 30 s      |
 +-----------+     | [R][R][R]  held by slow recs |     +-----------------+
                   +------------------------------+
                     ^ waiting: [C][A][C][A][C]...   checkout/auth time out
                     | unbounded wait, no limit      even though healthy

 WITH BULKHEADS: bounded queue + bounded pool per dependency
                  +- recs pool ------+  +-----------------+
             +--->| q[4] -> [R][R][R]|->| recommendations |  full: fail fast
             |    +------------------+  | SLOW            |
 +---------+ |                          +-----------------+
 | incoming|-+    +- checkout pool --+  +-----------------+
 | requests|----->| q[4] -> [C][ ][ ]|->| checkout  OK    |
 +---------+ |    +------------------+  +-----------------+
             |    +- auth pool ------+  +-----------------+
             +--->| q[4] -> [A][ ][ ]|->| auth      OK    |
                  +------------------+  +-----------------+

 [R]/[C]/[A] = permit held by a call  [ ] = free  q[4] = bounded queue
 Bound the resource that saturates first (threads, connections, or both).

With one shared pool, slow recommendation calls hold every slot until healthy checkout and auth calls queue and time out. With a bounded pool and a bounded queue per dependency, the slow calls fill only their own compartment and extra requests fail fast.

Who should use bulkheads

Bulkheads are worth their cost when three conditions hold together:

• The service calls several downstreams, and they differ in reliability or importance. A gateway or backend-for-frontend that aggregates data from many services fits this description. So does an order service that calls payments, inventory, pricing and recommendations.
• Calls hold a scarce, shared resource while they wait. Examples include blocking clients on a fixed thread pool, a shared database connection pool, a per-host HTTP connection limit, or a single event loop with limited in-flight capacity.
• Some traffic matters more than other traffic. If losing recommendations is acceptable but losing checkout is not, you need a mechanism that keeps the optional work from consuming the capacity the critical work depends on.

Specific groups that benefit:

• Teams that own user-facing services with critical paths, such as checkout, login or payments, sitting next to optional features in the same process.
• Platform and multi-tenant teams, where one tenant's heavy or slow workload must not degrade others. Per-tenant pools put a ceiling on any single tenant.
• Teams integrating third-party or legacy systems whose latency they don't control and can't fix.
• Services with a history of cascading incidents, where postmortems repeatedly find that a slow dependency exhausted a shared pool.

Bulkheads matter less for a service with one or two downstreams of similar latency, for low-traffic internal tools, or for workloads where every request is equally critical and there is nothing to protect by partitioning. The section on partition boundaries covers when to skip them.

Implementation forms: thread pools, semaphores, connection pools and dedicated deployments

A bulkhead is a cap on how much of your capacity one dependency can hold. There are four common ways to enforce that cap, and they differ in how strong the isolation is and what it costs to run.

Thread pools give real isolation because the caller can give up waiting even when the downstream call never returns. The price is a handoff between threads on every request. In an event-loop runtime, adding threads works against the model, so a counting semaphore does the same job at almost no cost. A semaphore cannot interrupt a stuck call, though, so it depends on a timeout in the client underneath.

The code example below shows that a bulkhead needs no library. It has three steps:

1. Try to acquire a permit, waiting only a short, bounded time.
2. If no permit arrives in that time, reject immediately with a distinct error the caller can recognise.
3. If a permit is acquired, make the call and release the permit in a finally block, so that failures and timeouts also return it.

Copyimport asyncio

class BulkheadFull(Exception):
    """Distinct error so callers can tell 'shed by bulkhead' from 'dependency failed'."""

class Bulkhead:
    def __init__(self, max_concurrent: int, max_wait_s: float, call_timeout_s: float):
        # The whole bulkhead is this counter: at most N calls in flight to one dependency.
        self._permits = asyncio.Semaphore(max_concurrent)
        self._max_wait_s = max_wait_s          # short, bounded wait for a permit
        self._call_timeout_s = call_timeout_s  # a semaphore can't interrupt a stuck call, so time it out

    async def run(self, call, *args):
        # Step 1: try to get a permit, but never queue indefinitely.
        try:
            await asyncio.wait_for(self._permits.acquire(), timeout=self._max_wait_s)
        except asyncio.TimeoutError:
            # Step 2: reject fast instead of piling up waiters behind a slow dependency.
            raise BulkheadFull("inventory-service bulkhead full") from None

        # Step 3: make the call; finally returns the permit on success, error or timeout.
        try:
            return await asyncio.wait_for(call(*args), timeout=self._call_timeout_s)
        finally:
            self._permits.release()

inventory_bulkhead = Bulkhead(max_concurrent=20, max_wait_s=0.05, call_timeout_s=2.0)

async def get_stock(client, sku: str):
    try:
        return await inventory_bulkhead.run(client.fetch_stock, sku)
    except BulkheadFull:
        return None  # caller-chosen fallback, e.g. show 'availability unknown'

A bulkhead built from a counting semaphore in Python asyncio: a bounded wait for a permit, a distinct rejection error when the wait expires, and a per-call timeout with the permit released in a finally block.

Sizing pools with Little's Law

A bulkhead is a capacity decision before it is a code decision. Little's Law gives the starting point: average concurrency equals arrival rate multiplied by time in the system. For a pool in front of one dependency, the number of permits you need is roughly the request rate to that dependency multiplied by how long each call holds a permit.

Illustrative estimate: assume 200 req/s to an inventory service with an assumed p99 latency of 50 ms. 200 × 0.05 s = 10 concurrent calls. Adding about 50% headroom for bursts gives 15 permits.

Size from p99 rather than the mean. Requests arrive in bursts and latencies cluster, so a pool sized to the mean is at its limit during ordinary variance.

The pool size also limits the damage when latency degrades. If that dependency slows to an assumed 2 s, holding the same traffic would take 400 slots. With a cap of 15, at most 15 callers wait on it, and the rest of the service keeps running.

Both sizing mistakes are costly:

• Undersized pools reject healthy traffic. You get errors from your own limit, not from a failing dependency.
• Oversized pools let a slow dependency take most of your threads or connections, so they no longer isolate anything.

Static sizes drift as traffic grows and dependency latency changes. Revisit them periodically.

Adaptive concurrency limits avoid manual sizing. They adjust the limit from observed latency or error signals, similar to TCP congestion control. They have their own risks. They can oscillate when the signal is noisy, and they can converge slowly after a sudden change. Give them sensible minimum and maximum bounds.

When the bulkhead is full: rejection, queues and fallbacks

A full bulkhead has two options: reject the call immediately or let it wait briefly in a bounded queue. Immediate rejection keeps latency predictable and frees the caller's thread at once. A short queue absorbs bursts that last milliseconds, but each queued slot adds wait time to the caller's latency budget. Keep the queue small enough that the maximum wait stays well under the caller's timeout. Otherwise requests time out while still queued, and the work they eventually do is wasted.

Rejection should be a designed outcome with its own metric, log field and error type. It should not look the same as a dependency failure. If rejections can't be told apart from timeouts, nobody can tell whether the pool is undersized or the dependency is down.

Fallbacks run only during incidents, so they are often the least-tested code in the service. A fallback that reads from a cache can send a sudden flood of traffic to that cache at the worst possible moment. Exercise fallbacks deliberately, and bound them like any other dependency.

A bulkhead protects the caller, not the dependency. Rejected work still represents demand. If clients retry it immediately, they amplify the overload. Signal rejections so that upstream callers back off with jitter or shed the load, rather than resubmitting it.

How bulkheads work with timeouts, retries and circuit breakers

Timeouts, retries and circuit breakers each change how long work occupies a bulkhead slot. Their values must be set against the concurrency limit, not separately.

Timeouts bound how long a stuck call holds a slot. When a dependency hangs, in-flight work grows to arrival rate × timeout. If that product exceeds the limit, the pool fills and rejects healthy traffic before any call times out. The timeout therefore has to be shorter than limit ÷ arrival rate, which is the time a hung dependency takes to fill the pool.

Retries add attempts inside the same bulkhead. With a retry ratio r, in-flight work becomes rate × (1 + r) × timeout. Per-request retry counts allow r to approach the full retry count during an outage. A retry budget, capped as a fraction of recent successful calls, keeps r small and predictable.

Circuit breakers react only after enough failures accumulate in their window. The bulkhead caps the damage during that detection period. Once the breaker opens, it stops sending work to a dependency that is known to be failing, which frees the slots. Neither one replaces the other.

Illustrative estimate (assumed numbers, not a benchmark), for one dependency per instance:

• Assumed peak arrival rate: 100 req/s. Assumed p99 latency: 200 ms.
• Normal in-flight (Little's Law): 100 × 0.2 = 20. Concurrency limit: 30.
• Fill time if the dependency hangs: 30 ÷ 100 = 0.3 s, so the timeout is set to 250 ms.
• Retry budget: 10%. Worst-case in-flight: 100 × 1.1 × 0.25 = 27.5, which stays under 30.
• Breaker: opens at a 50% failure rate over 20 calls and stays open for 5 s before probing.

Copydependency: inventory-service

assumptions:
  peak_arrival_rate_rps: 100   # assumed peak calls/s from this instance
  p99_latency_ms: 200          # assumed healthy p99

bulkhead:
  # Normal in-flight = 100 req/s x 0.2 s = 20 (Little's Law).
  # The limit leaves headroom above that for bursts and retries.
  max_concurrent_calls: 30
  max_wait_ms: 0               # reject immediately when full instead of queueing

timeout:
  # A hung dependency fills the pool in 30 / 100 req/s = 300 ms.
  # The timeout must be shorter so calls start failing before the pool fills.
  per_attempt_ms: 250

retry:
  max_attempts: 2              # at most one retry per request
  # Budget caps retries at 10% of recent successful calls, so r <= 0.1
  # even when every request fails.
  budget_ratio: 0.10
  budget_window_s: 10
  # Worst-case in-flight = 100 x (1 + 0.1) x 0.25 s = 27.5, under the limit of 30.

circuit_breaker:
  # The bulkhead limits damage while the breaker gathers evidence.
  sliding_window_calls: 20
  failure_rate_threshold_pct: 50
  open_duration_s: 5           # stop sending work, which frees the slots
  half_open_probe_calls: 3     # trial calls before closing again
  # Count timeouts and bulkhead rejections as failures so a hang trips the breaker.
  record_failures: [timeout, connection_error, http_5xx]

A generic per-dependency resilience config in which the timeout, retry budget, concurrency limit and circuit breaker values are derived from one set of assumed traffic numbers, with the arithmetic in comments, so the timeout fires before the pool fills and retries stay within the limit.

Choosing partition boundaries and avoiding hidden shared resources

Partition along the lines where failures actually differ. Per dependency is the default: a slow payments API should not hold the threads that inventory lookups need. Within a service, split by endpoint criticality, so checkout gets its own pool and a recommendations spike cannot starve it. Per-tenant pools contain noisy neighbours. Separating reads from writes keeps a burst of cheap reads from blocking writes, which hold locks and are costly to drop.

Isolation is only as good as the narrowest shared layer. Two bulkheads that share an HTTP client connection pool, a DNS resolver, a host's file descriptors, a database, a network path or a downstream service behave as one bulkhead under load. For each partition, follow the call down the stack and find the point where it merges with the others.

The two layers complement each other. The application decides which work matters and how to degrade. The proxy enforces a ceiling that application code cannot accidentally bypass.

Skip bulkheads when you have few dependencies with uniform latency, or when traffic is low. Consider a 20-slot budget split into five pools of 4 slots each. A burst that one shared pool would absorb now triggers rejections, and the slots in idle pools sit unused. In that case the stranded capacity costs more than the isolation is worth.

Alternatives to bulkheads

Several other techniques address overload and slow dependencies. Most are complements rather than substitutes, and each leaves a gap that bulkheads fill.

In practice, a resilient service combines several of these. Timeouts bound each call, bulkheads bound concurrency per dependency, circuit breakers stop calls to dependencies known to be failing, and edge rate limiting or load shedding protects the service as a whole. If you can only add one mechanism, start with timeouts on every outbound call. Without them, no other technique works reliably. Add bulkheads next when you have multiple downstreams that share a pool.

How bulkheads save engineering time

The main payoff is operational. It shows up in incidents, investigations and on-call load rather than in feature work.

• Smaller incidents. When a slow dependency is contained to its own pool, the incident is "recommendations are degraded" rather than "the whole site is down". Fewer people get paged, fewer teams join the incident call, and the response is a scoped fallback rather than an emergency rollback or restart across the fleet.
• Faster diagnosis. In a shared pool, every endpoint times out at once, and engineers have to work out which dependency started it. With per-pool metrics, the saturated pool with rising rejections identifies the culprit directly. The distinct rejection error described earlier lets dashboards and logs separate "we shed this" from "the dependency failed" without anyone reading thread dumps.
• Less recovery work. Cascading failures often leave behind stuck threads, exhausted connection pools and retry storms that last after the original dependency recovers, which then need restarts. A bounded pool with bounded queues drains on its own once the dependency recovers.
• Fewer recurring postmortem actions. "Shared pool exhausted by slow dependency X" is a common postmortem finding. Fixing the class of problem once, with a reusable bulkhead wrapper and standard config, removes repeated one-off fixes for each new dependency.
• Safer integration of new dependencies. A team adding a call to an unfamiliar or third-party service can give it a small, bounded pool and a fallback. The worst case is then known before launch, which reduces how much load testing and review the integration needs before it ships.

Illustrative estimate (all inputs assumed): suppose a team has an assumed four cascading incidents a year caused by one slow dependency exhausting a shared pool. Assume each involves an assumed six engineers for an assumed three hours of response and diagnosis, plus an assumed eight hours of postmortem and follow-up work. That is 4 × (6 × 3 + 8) = 104 engineer-hours a year. If bulkheads turn those into contained degradations handled by one on-call engineer for an assumed one hour each, with a lighter assumed two-hour review, the cost drops to 4 × (1 + 2) = 12 engineer-hours. Set that against the setup and upkeep cost: an assumed 40 hours to build and roll out a shared wrapper, config and dashboards, plus an assumed 10 hours a year to review limits. Plug in your own incident history; the result depends entirely on how often shared-pool exhaustion actually happens to you.

The time savings depend on doing the work once and reusing it. One shared bulkhead wrapper, one config shape and one dashboard template across services cost far less to maintain than hand-rolled limits in each codebase.

Observability, fault injection and the cost of running bulkheads

Per-pool metrics

Each pool needs its own metrics. Aggregate service metrics hide exactly the imbalance a bulkhead creates. Export these, tagged by pool name:

• Active count: slots in use right now.
• Queue depth: callers waiting for a slot.
• Rejection count: calls refused because the pool and its queue were full.
• Wait time: time spent queued before getting a slot. This is the time a caller burns before the dependency is even contacted.
• Saturation ratio: active count divided by maximum slots.

Alert on sustained saturation, for example a ratio near 1.0 held over several minutes, together with a non-zero rejection rate. Short bursts are what the bulkhead is built to absorb, so paging on them trains people to ignore the alert.

Verifying isolation

Configuration shows what you intended, not what happens under load. To prove the pools are separate, inject latency into one dependency in a test environment while the others receive normal traffic. The slowed pool should saturate and reject. Latency and error rates for every other dependency should stay flat. If they rise, something is still shared. Common causes are an event loop, a client-side connection limit, or a log appender.

The cost

Bulkheads strand capacity, so utilization dashboards look worse. Every extra pool adds limits to configure and alerts to maintain. Defaults that nobody revisits get copied into new pools. The payoff is that an incident stays inside one feature. Investigations are also shorter because the saturated pool names the culprit, and cascading failures become rarer.

Use a small number of deliberately sized pools, and review each limit against measured latency rather than the values you picked at launch.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to bulkhead pattern microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.