Bulkhead Pattern in Microservices: Sizing Thread Pools and Limits

What a bulkhead is and the failure it contains

A bulkhead partitions a service's capacity so that resource isolation becomes failure containment. Each partition has its own pool of worker threads, permits, or pooled connections, and it is assigned to one dependency or one class of work. When a partition is exhausted, only the callers of that dependency wait or get rejected. Everything else keeps its share.

The failure it contains is quiet. Suppose a downstream stops failing fast and starts responding in eight seconds instead of eighty milliseconds. No exceptions are thrown. Each request to that dependency holds a worker thread or a connection for a hundred times longer than usual. Within seconds, every worker in a shared pool is blocked on that one call. Requests to healthy endpoints then queue behind them and time out. Your error dashboards stay green while latency for every endpoint climbs.

CopySHARED POOL (one slow dependency)        PARTITIONED POOLS
+-----------------------------+          payments  [####] full, rejecting
| ####################### #### |          catalog   [#...] serving
| all workers blocked on        |          profile   [##..] serving
| payments; catalog and profile |          search    [#...] serving
| requests wait in the queue    |
+-----------------------------+

A circuit breaker on its own does not prevent this. It trips only after its sliding window collects enough failed or slow calls to cross a threshold. While that window fills, the slow calls are already holding threads. Saturation can therefore happen before the breaker has the data to open. A bulkhead puts a hard limit on how much capacity those calls can take during that gap.

If your service calls several downstreams with different latency and criticality profiles, you need this pattern. That applies to backend engineers who own the call paths and to platform engineers who set the shared defaults.

Copy(A) SHARED WORKER POOL
 requests: /pay  /catalog  /profile
                 |
                 v
 +--------------------------------+       +-------------------------+
 | shared worker pool             |------>| payments   8s     SLOW  |
 | [P][P][P][P][P][P][P][P]       |       +-------------------------+
 | every worker blocked on pay    |   X-->  catalog  80ms  starved
 +--------------------------------+   X-->  profile  80ms  starved
   queue: catalog, profile, ...  -> timeouts, zero errors thrown

(B) PARTITIONED POOLS (BULKHEADS)
 requests: /pay  /catalog  /profile
                 |
                 v
 +--------------------------------+
 | route by dependency            |
 +--------------------------------+
      |               |               |
      v               v               v
 +-----------+   +-----------+   +-----------+
 | payments  |   | catalog   |   | profile   |
 | [P][P][P] |   | [C][ ][ ] |   | [R][ ][ ] |
 | FULL:     |   | serving   |   | serving   |
 | rejecting |   |           |   |           |
 +-----------+   +-----------+   +-----------+
      |               |               |
      v               v               v
 payments 8s     catalog 80ms    profile 80ms
 (SLOW)          (healthy)       (healthy)

In a shared pool one slow dependency blocks every worker and starves healthy endpoints without raising errors, while partitioned pools confine the saturation to the slow dependency's own partition.

How it works: thread-pool isolation versus semaphore limits

There are two ways to build a partition, and they bound different things.

Thread-pool isolation hands each call to a pool dedicated to one dependency. The request thread submits the task and either waits on the result with a deadline or moves on. Because the slow call runs on a different thread, a hung dependency can exhaust only its own pool, and request-handling threads stay free to serve other routes. The pool can also have a bounded queue in front of it, so short bursts wait briefly instead of being rejected outright. The costs are real. Every call pays a handoff and a context switch, and every pool reserves memory for its threads' stacks. Anything held in thread-locals, such as trace IDs, logging context and the security principal, stays on the original thread unless you wrap tasks to copy it across.

A semaphore limit runs the call on the caller's own thread and counts permits. Acquiring a permit costs almost nothing, and context carries over because the call never changes threads. However, it bounds in-flight calls, not threads. If a call blocks with no timeout, the caller's thread blocks with it. The semaphore stops new callers from entering, but the threads already inside are stuck, and those threads belong to your shared request pool.

A semaphore is only as safe as the timeout on the call it guards.

Sizing each partition with Little's Law

A bulkhead limit is a capacity decision. Little's Law gives the starting number: required concurrency ≈ arrival rate × latency.

Illustrative estimate (assumed inputs): a dependency receives 200 req/s with a p99 latency of 50 ms. 200 × 0.05 s = 10 concurrent slots. Bursts push arrivals above the average, so add headroom. A limit of 20 to 25 slots absorbs a doubling without rejecting healthy traffic.

Now assume the dependency degrades to 2 s latency at the same arrival rate. 200 × 2 s = 400 slots. Sizing for that number would remove the bulkhead's purpose, because 400 threads or connections held open on a failing dependency is the resource exhaustion the bulkhead exists to prevent. Derive the limit from healthy-state demand plus headroom. When latency rises, the bulkhead should fill and reject the excess quickly. A deep queue only moves the wait somewhere else. Callers still hold their own threads and sockets, queued requests time out before they run, and the backlog extends recovery after the dependency heals.

The limit is also per instance, but the dependency sees all instances together. Twenty autoscaled instances × 25 slots means 500 concurrent calls, which can exceed the downstream's real capacity or a database's connection cap. Agree on the aggregate budget with the dependency's owner, and cap autoscaling or per-instance limits to fit within it.

Fixed limits drift as traffic and latency change. Adaptive concurrency limiting adjusts the limit from observed latency. The trade-off is that the limit changes during incidents, which makes behavior harder to predict and reason about.

How to implement it: resilience4j and connection pools

In resilience4j, the semaphore-based Bulkhead limits concurrency on the caller's own thread. maxConcurrentCalls sets the number of permits, and maxWaitDuration sets how long a caller blocks for a permit before the call is rejected. Keep the wait at or near zero on request paths. A long wait just moves the queue into your request threads. The ThreadPoolBulkhead runs calls on a dedicated executor sized by coreThreadPoolSize, maxThreadPoolSize and queueCapacity. A slow dependency then consumes only its own threads, at the cost of a thread handoff and context propagation.

As an illustrative estimate, assume 40 requests per second at 0.25 s p99 latency. That gives 10 concurrent calls, so set maxConcurrentCalls to about 12 and maxWaitDuration to 0. For the thread-pool variant, set core and max to 12 and use a small queue, such as 4. Don't rely on defaults, because they are not sized for your traffic. Check the documentation for your version for exact property names and behavior.

Below the application, use HTTP clients with per-host or per-route connection limits so one slow downstream can't take the whole connection pool. Give databases separate pools: one for OLTP and another for reporting or batch work. If one shared pool serves every tenant or endpoint, the slowest consumer holds connections and starves the others. That shared pool is the single point of failure the bulkhead was meant to remove.

Every bulkheaded call needs a timeout. Without one, a hung call holds its permit or thread indefinitely, and the partition fills and stays full.

Copy# Spring Boot application.yml for the resilience4j Spring Boot starter.
# Values come from the illustrative estimate: 40 req/s x 0.25 s p99 = ~10 concurrent calls.
# Property names vary between versions; check the docs for the version you use.
resilience4j:
  bulkhead:
    instances:
      inventoryService:
        # Semaphore bulkhead: runs on the caller's thread, ~10 needed plus headroom
        maxConcurrentCalls: 12
        # Reject immediately instead of queueing inside request threads
        maxWaitDuration: 0ms
  thread-pool-bulkhead:
    instances:
      pricingService:
        # Dedicated executor so a slow pricing dependency only uses its own threads
        coreThreadPoolSize: 12
        maxThreadPoolSize: 12
        # Small queue absorbs brief bursts without hiding sustained overload
        queueCapacity: 4
  timelimiter:
    instances:
      pricingService:
        # Every bulkheaded call needs a timeout so hung calls release their thread
        timeoutDuration: 500ms
        cancelRunningFuture: true
# The semaphore bulkhead has no time limiter of its own here. Bound
# inventoryService calls with the HTTP client's connect and read timeouts
# so a hung call cannot hold its permit indefinitely.

A resilience4j configuration that sizes a semaphore bulkhead and a thread-pool bulkhead from the illustrative 40 req/s at 0.25 s p99 estimate and pairs the thread-pool bulkhead with a time limiter so every bulkheaded call has a timeout.

Composing bulkheads with timeouts, retries, circuit breakers and load shedding

A timeout only protects the partition if it actually ends the work. Abandoning a future while the underlying socket read keeps blocking still leaves the thread or permit occupied. Set timeouts on the client I/O itself (connect and read timeouts, or a deadline propagated to the driver) so the permit is released when the clock runs out.

Retries and queues

Place retries outside the bulkhead so each attempt must acquire a fresh permit. A retry loop inside the bulkhead holds one permit across every attempt, which multiplies occupancy.

Illustrative estimate (assumed numbers, not a benchmark): permits needed ≈ arrival rate × attempts × per-attempt latency. At 50 req/s, 3 attempts and a 200 ms timeout per attempt, that is 50 × 3 × 0.2 = 30 permits. Without retries it is 50 × 1 × 0.2 = 10.

Waiting for a permit has a similar cost. Time spent queued counts against the caller's deadline, so a caller may still be waiting after its own timeout has already passed. That turns a fast failure into a slow one. A zero or very short maximum wait is usually the better default.

Breakers and shedding

The bulkhead caps damage immediately, during the window before the circuit breaker has collected enough failures to open. Once the breaker opens, it stops sending traffic to the unhealthy dependency entirely. Decide deliberately whether bulkhead rejections count as breaker failures. Load shedding at the service edge protects the process as a whole, which no single partition can do.

The rejected path

Rejections need a defined response, such as a cached value, a reduced feature set, or a clear 503 with Retry-After. These fallbacks run under stress, so they must be correct and tested. A bulkhead contains the blast radius. It does not fix the slow dependency.

Copyimport asyncio

class Rejected(Exception):
    pass

bulkhead = asyncio.Semaphore(10)  # permits reserved for the inventory dependency

async def guarded(fetch, timeout_s=0.2):
    # Fast rejection, no queue: time spent waiting for a permit counts
    # against the caller's deadline and turns a fast failure into a slow one.
    if bulkhead.locked():
        raise Rejected("inventory bulkhead full")
    async with bulkhead:
        # Timeout is innermost, around the I/O. wait_for cancels the coroutine,
        # so an async-native client stops reading and the permit is released.
        return await asyncio.wait_for(fetch(), timeout_s)

async def with_retry(fetch, attempts=3):
    # Retry sits OUTSIDE the bulkhead. Each attempt acquires a fresh permit,
    # and no permit is held during the backoff sleep.
    for i in range(attempts):
        try:
            return await guarded(fetch)
        except Rejected:
            raise  # partition is saturated, so retrying only adds load
        except asyncio.TimeoutError:
            if i == attempts - 1:
                raise
            await asyncio.sleep(0.05 * 2 ** i)

async def get_stock(fetch, cache):
    # Outermost layer: the fallback. A circuit breaker, if used, would wrap
    # with_retry here so an open breaker skips the bulkhead entirely.
    try:
        return await with_retry(fetch)
    except (Rejected, asyncio.TimeoutError):
        # Degraded answer. This path runs under stress, so it needs its own tests.
        return cache.get("stock", {"status": "unknown", "stale": True})

Shows an asyncio bulkhead layered from the inside out: an I/O timeout inside the permit, fast rejection instead of queueing, retries outside the bulkhead, and a cached fallback for rejected or timed-out calls.

Partitioning strategies and when to avoid bulkheads

Every partition is a fixed allocation. A pool of 20 reserved for an idle dependency cannot serve a busy one, so stranded capacity sits next to rejected requests. The same cap also limits peak throughput to that dependency even when the rest of the service is quiet. If you combine all three schemes (tenant × endpoint × dependency), you get hundreds of limits that nobody can tune.

Avoid or defer bulkheads in three cases:

• The service has a single downstream, so there is nothing to isolate it from.
• Concurrency is low enough that one shared limit is sufficient.
• The team lacks per-bulkhead metrics. Without them, sizes are guesses.

Other mechanisms can replace or complement bulkheads:

• Timeouts with circuit breakers often contain slow dependencies on their own.
• Global load shedding or admission control protects the whole process.
• Adaptive concurrency limits adjust automatically to observed latency.
• Separate deployments or cells give full isolation, at infrastructure cost.
• Queues decouple asynchronous work from request paths.

On reactive stacks, event loops, or virtual threads, thread exhaustion is unlikely. The scarce resource moves to downstream connections, memory, and the dependency's own capacity, so you still need a concurrency limit. It now guards those resources rather than threads.

Observability, testing and the cost to engineering time

Metrics per bulkhead

Every partition should emit four signals, tagged with the bulkhead name. Available permits or idle threads show headroom. Queue depth shows pressure building before calls are refused. Rejections count calls turned away. Wait time measures how long a call waited for a permit or thread. Wait time matters because a bulkhead with a queue can add latency long before it rejects anything.

Alerts that separate success from misconfiguration

A bulkhead is doing its job when rejections rise in one partition, that dependency's latency is also elevated, and the other partitions stay healthy. Treat this as a dependency incident, not a bulkhead problem.

It is misconfigured in two cases:

• Rejections during normal bursts while the dependency's latency is normal. The partition is too small for real concurrency, so you are shedding traffic you could have served.
• A partition whose permits are never close to exhausted over weeks. It probably reserves capacity you need elsewhere, or it never tested the failure it was meant to contain.

Alert on rejections combined with dependency latency, not on rejections alone.

Verify containment with fault injection

In staging or a controlled production experiment, add latency to one dependency until its partition saturates. Then confirm that unrelated endpoints keep their baseline latency and success rate. If they degrade, something is still shared, such as a common executor, connection pool, or request-handling thread.

Cost to engineering time

The up-front work is sizing, configuration, propagating context into isolated threads, and testing fallbacks. Limits also need ongoing tuning as traffic changes. The payoff comes during incidents. A slow dependency degrades one feature instead of exhausting the whole service, so you investigate one partition instead of chasing a broad outage across many symptoms.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to bulkhead pattern microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.