What is the bulkhead pattern in microservices, and what failure does it prevent?

A bulkhead is a bounded concurrency limit assigned to one dependency, with an explicit reject path for calls that exceed it. The name comes from ship hulls, where sealed compartments keep one breach from flooding the whole vessel. In a service, the compartment is the number of in-flight calls a single downstream is allowed to hold at once.

The failure it prevents builds in steps. Suppose a service sends every outbound call through one shared executor:

  1. The inventory service slows from 50 ms to 5 seconds per call.
  2. Its calls don't fail. They hang, and each one holds a request thread and an HTTP connection-pool slot for the full duration.
  3. New requests keep arriving at the normal rate, so the number of stuck threads grows until the shared pool is exhausted.
  4. Calls to the healthy pricing and user services now queue behind inventory calls or get rejected. The service's own health check can't get a thread, so the orchestrator marks the instance unhealthy and restarts it. That spreads the load onto other instances, which then fail the same way.

With a separate limit per downstream, inventory fills its own compartment and further inventory calls are rejected immediately. Pricing, user lookups and health checks keep their capacity.

Two common setups only look like protection. A shared pool sized to a large number delays exhaustion but does not prevent it. A bulkhead with an unbounded queue never rejects anything, so slow calls still pile up as memory use and latency. Both carry the cost of a bulkhead without the isolation.

The pattern fits any service that calls several downstreams with different latency profiles, especially when one of them is slow, remote or less reliable than the rest.

A) ONE SHARED EXECUTOR
                       +------------------------------------+
  requests ----------->| shared pool  [I][I][I][I][I][I][I] |
  health check --X---->| full: inventory calls hold slots   |
                       +------------------------------------+
                          |             |             |
                          v             v             v
                    +-----------+ +-----------+ +-----------+
                    | inventory | | pricing   | | user      |
                    | 5 s/call  | | starved   | | starved   |
                    +-----------+ +-----------+ +-----------+

B) PER-DOWNSTREAM BULKHEADS
  requests ------+-----------------+-----------------+
                 |                 |                 |
                 v                 v                 v
          +-------------+   +-------------+   +-------------+
          | inv pool    |   | pricing pool|   | user pool   |
          | [I][I][I]   |   | [P][ ][ ]   |   | [U][ ][ ]   |
          | full        |   | has room    |   | has room    |
          +-------------+   +-------------+   +-------------+
            |        |             |                 |
 reject <---+        v             v                 v
 (fast)        +-----------+ +-----------+     +-----------+
               | inventory | | pricing   |     | user      |
               | 5 s/call  | | healthy   |     | healthy   |
               +-----------+ +-----------+     +-----------+
  health check ------> own capacity, still answered

With one shared executor, slow inventory calls ([I]) take every slot and starve pricing, user lookups and the health check; with a bounded pool per downstream, only further inventory calls are rejected and the other paths keep their capacity.

Who should use the bulkhead pattern?

Bulkheads pay off for services whose own availability depends on several downstreams that can fail independently. Typical candidates:

  • Aggregating services, such as backend-for-frontend layers, API composition services and gateways that fan out one inbound request to several dependencies. One slow dependency should cost you one section of the response, not the whole response.
  • Services on blocking, thread-per-request stacks, where every in-flight downstream call holds a platform thread and a connection. This is where a shared pool gets exhausted fastest.
  • Services calling third-party or cross-region APIs, whose latency you do not control and which tend to degrade by getting slow rather than by failing cleanly.
  • Shared platform services with many callers, where one tenant or one class of traffic, such as batch jobs, should not consume capacity that interactive traffic needs.

The people who own the decision are the service's developers, who wire the limits and fallbacks into the call path, and whoever operates the service in production, who reviews limits against traffic and alerts on saturation. Bulkheads only work if the team is willing to treat a rejection as a designed outcome with a defined response, not as a bug.

Teams with a single downstream, or with uniformly fast calls already bounded by timeouts and a server-side request concurrency limit, get little from adding bulkheads. The granularity section below covers those cases in more detail.

How does the bulkhead pattern work?

Every bulkhead follows the same lifecycle, whatever library or runtime implements it:

  1. Assign a named limit to each dependency. Each downstream, or each critical endpoint, gets its own fixed number of concurrent calls. The limits are independent, so one can be exhausted while the others are idle.
  2. Admit or reject at the call site. Before calling the dependency, the caller asks its bulkhead for capacity. If capacity is available, the call proceeds. If not, the caller either waits a very short, bounded time or is rejected immediately.
  3. Run the call under a timeout. An admitted call holds its slot until it completes, fails or times out. The timeout ensures that a hung downstream cannot hold slots indefinitely.
  4. Release the slot on every exit path. Success, exception and timeout must all return capacity to the bulkhead, which is why implementations release in a finally block or completion callback.
  5. Handle the rejection. A rejected call goes straight to a fallback: a fast error, a cached value or a partial result.

The key property is that the limit is fixed while latency is allowed to vary. When a downstream slows, it cannot claim more concurrency than its limit. Its slowness turns into rejections for calls to that dependency alone, instead of resource exhaustion that spreads to the rest of the service. The next section covers the two ways to enforce the limit.

How does a bulkhead work: semaphore vs thread-pool isolation?

There are two ways to enforce the limit, and they differ in which thread does the work.

A semaphore bulkhead keeps a counter of in-flight calls. Before calling, the caller tries to acquire a permit. If it gets one, it makes the call on its own thread and releases the permit in a finally. If no permit is free within a configured wait, the call is rejected. This adds almost no overhead and leaves thread-local state where it already is. However, the caller is still blocked for the full duration of the downstream call. Resilience4j's Bulkhead follows this model: it decorates a Supplier and runs it inline.

A thread-pool bulkhead submits the call to a dedicated, bounded executor with its own queue and returns a future right away. The caller can then wait on that future with a deadline and move on. Resilience4j's ThreadPoolBulkhead follows this model: it decorates a supplier into a CompletionStage.

Aspect Semaphore Thread pool
Executing thread Caller's Dedicated executor
Overhead A counter Thread handoff and queue
Caller can give up early No Yes, via future timeout
Thread-local context Preserved Lost unless propagated

The thread-pool model has two costs. First, MDC entries, tracing spans, security context and transactions are bound to the caller's thread, so none of them reach the worker unless you copy them across explicitly. Transactions generally cannot be carried over at all.

Second, when the caller's timeout fires, only the caller is released. The worker remains blocked in I/O, so its pool slot is not freed until the socket or client timeout ends the call. If client timeouts are missing or set too long, the pool can still fill up with stuck calls.

import java.util.Map;
import java.util.concurrent.*;
import java.util.function.Supplier;
import org.slf4j.MDC;

class Bulkheads {
  // Semaphore: cap in-flight calls; the work runs on the caller's own thread.
  private final Semaphore permits = new Semaphore(10);

  <T> T viaSemaphore(Supplier<T> call) throws InterruptedException {
    if (!permits.tryAcquire(50, TimeUnit.MILLISECONDS)) {
      throw new RejectedExecutionException("bulkhead full"); // fail fast
    }
    try {
      return call.get(); // caller stays blocked for the full downstream call
    } finally {
      permits.release();
    }
  }

  // Thread pool: fixed workers plus a bounded queue; overflow is rejected.
  private final ExecutorService pool = new ThreadPoolExecutor(10, 10, 0L,
      TimeUnit.MILLISECONDS, new ArrayBlockingQueue<>(20),
      new ThreadPoolExecutor.AbortPolicy());

  <T> T viaPool(Supplier<T> call) throws Exception {
    Map<String, String> mdc = MDC.getCopyOfContextMap(); // thread-bound: copy it
    Future<T> f = pool.submit(() -> {
      if (mdc != null) MDC.setContextMap(mdc);
      try { return call.get(); } finally { MDC.clear(); }
    });
    try {
      return f.get(2, TimeUnit.SECONDS); // caller gives up at its deadline
    } catch (TimeoutException e) {
      f.cancel(true); // blocking socket I/O typically ignores the interrupt, so
      throw e;        // the worker frees only when the client's own timeout fires
    }
  }
}

A version-agnostic sketch using plain java.util.concurrent: the semaphore bulkhead runs the call inline under a permit, while the thread-pool bulkhead hands it to a bounded executor, copies MDC context across explicitly, and lets the caller time out even though the worker may stay blocked.

How do you size a bulkhead limit for each downstream?

Start with Little's Law: average concurrency equals arrival rate multiplied by time in the system. Apply it to each dependency separately, because each has its own traffic share and latency profile.

Scenario (illustrative estimate) Arrival rate Latency Concurrent calls needed
Healthy downstream 200 req/s 50 ms (p99) ~10, plus headroom (say 15–20)
Degraded downstream 200 req/s 2 s 400

The healthy case suggests a limit of around 15–20. The degraded case shows why you must fix the limit and let latency vary. If concurrency is allowed to grow with latency, a slow dependency claims 400 slots, and those slots come out of capacity that other downstreams need. The bulkhead exists to cap that number. When the cap is reached, excess calls are rejected, and the rest of the service keeps running.

Both extremes cause problems. A limit set too low rejects calls during ordinary traffic peaks. Callers then see errors that look like an outage even though the downstream is healthy. A limit set too high never fills before the shared resource is exhausted, so it gives no isolation.

Use load tests to confirm the numbers. Run them at realistic peak rates, and also inject downstream latency, such as p99 slowdowns and stalls that last until the timeout. Measuring only the healthy path tells you nothing about how the limit behaves under failure.

Limits are usually enforced per instance. A limit of 20 across 10 replicas permits 200 concurrent calls to the downstream, and autoscaling raises that figure further. Check the multiplied total against what the downstream can actually serve.

What should happen when the bulkhead is full?

A full bulkhead should reject quickly. A thread-pool bulkhead with an unbounded queue only looks isolated: excess work piles up behind the busy threads, each queued request inherits the downstream's full latency plus its wait time, and the heap grows until the service fails for reasons unrelated to that downstream. Use a zero or small bounded queue, sized to absorb a brief burst rather than sustained overload.

Semaphore bulkheads have the equivalent setting in their maximum wait duration, which is how long a caller blocks waiting for a permit. Keep it at zero or a few milliseconds. A long wait ties up the caller's own request thread, so the overload spreads to the caller, which is the failure the bulkhead was meant to prevent.

Decide what a rejection returns before you need it:

Rejection response When it fits Status to return
Fast failure No safe substitute exists 503 with Retry-After
Cached or degraded response Stale data is acceptable 200, marked as stale
Partial result The page aggregates several downstreams 200, with the missing parts flagged
Per-tenant limit hit One caller exceeds its own share 429 with Retry-After

The same caller must not retry a rejection immediately. The bulkhead is full because capacity is exhausted, and an instant retry adds load at the moment the bulkhead is shedding it. Honour the retry hint or back off.

A rejection without a fallback still hurts users. It protects the rest of the service, but the slowness of one downstream now appears to users as errors.

import java.util.Map;
import java.util.concurrent.*;

public abstract class DownstreamBulkheads {
  record Result(int status, String body, Integer retryAfterSeconds) {}

  // Thread-pool bulkhead for inventory: a small bounded queue absorbs a brief burst.
  // AbortPolicy rejects at once when threads and queue are full; it never blocks or grows.
  private final ThreadPoolExecutor inventoryPool = new ThreadPoolExecutor(
      10, 10, 0L, TimeUnit.MILLISECONDS,
      new ArrayBlockingQueue<>(5), new ThreadPoolExecutor.AbortPolicy());

  // Semaphore bulkhead for pricing: 20 concurrent calls on the caller's own thread.
  private final Semaphore pricingPermits = new Semaphore(20);
  private final Map<String, String> stockCache = new ConcurrentHashMap<>();

  Result getStock(String sku) throws InterruptedException, ExecutionException {
    Future<String> call;
    try {
      call = inventoryPool.submit(() -> callInventory(sku));
    } catch (RejectedExecutionException full) {
      // Rejected: serve stale data if we have it, otherwise fail fast with a retry hint.
      String cached = stockCache.get(sku);
      return cached != null ? new Result(200, "{\"stale\":true,\"stock\":" + cached + "}", null)
                            : new Result(503, "inventory busy", 2); // send as Retry-After: 2
    }
    try { return new Result(200, call.get(800, TimeUnit.MILLISECONDS), null); }
    catch (TimeoutException slow) { call.cancel(true); return new Result(503, "inventory slow", 2); }
  }

  Result getPrice(String sku) throws InterruptedException {
    // Max wait of 5 ms: a long wait would park the caller's request thread and spread the overload.
    if (!pricingPermits.tryAcquire(5, TimeUnit.MILLISECONDS)) return new Result(503, "pricing busy", 1);
    try { return new Result(200, callPricing(sku), null); }
    finally { pricingPermits.release(); }
  }

  abstract String callInventory(String sku) throws Exception;
  abstract String callPricing(String sku);
}

A plain-JDK sketch of a thread-pool bulkhead with a small bounded queue and immediate rejection, plus a semaphore bulkhead with a 5 ms maximum wait; a rejection returns a stale cached response when one exists and otherwise a 503 with a Retry-After hint.

How should bulkheads be combined with timeouts, circuit breakers and retries?

That is why the bulkhead cannot work alone. Each layer bounds a different thing. The bulkhead limits how many calls to a dependency run at once. The timeout limits how long any single call can hold a permit. The circuit breaker stops making calls once a dependency is clearly failing. Without a timeout, a bulkhead fills with hung calls and stays full. Without a breaker, a dead dependency still uses up every permit until each call times out.

A common ordering from outermost to innermost is retry → circuit breaker → timeout → bulkhead → call. With this ordering:

  • Each retry attempt must acquire a new permit.
  • An open breaker rejects attempts before they consume a permit.
  • The timeout releases a permit when a call stalls.

If retry sits inside the bulkhead, one request holds its permit for the whole retry sequence. Retries also multiply load on the downstream wherever you place them, so give them a small attempt count and a budget. If every failure turns into three attempts, those retries take over the bulkhead from fresh traffic. Every attempt, including its backoff, must finish before the caller's own deadline. Otherwise the caller gives up while your threads keep working on a request nobody is waiting for.

Align the limits below the bulkhead too. If the bulkhead allows 50 concurrent calls but the HTTP client connection pool for that host allows 20, 30 threads sit waiting for a connection. The same applies to the database pool for a data dependency. Set the bulkhead at or below the smallest pool behind it.

A bulkhead protects only the caller. It does nothing for a database, network link, or client pool shared with other services.

// Resilience4j: decorators apply inside-out, so the first one listed wraps the call most tightly.
// Effective order, outermost to innermost: retry -> circuit breaker -> timeout -> bulkhead -> call
ThreadPoolBulkhead bulkhead = ThreadPoolBulkhead.of("inventory",
    ThreadPoolBulkheadConfig.custom()
        .coreThreadPoolSize(20)
        .maxThreadPoolSize(20)   // keep <= the HTTP client's max connections for this host (assumed 20)
        .queueCapacity(10)       // small queue: reject early rather than pile up waiting work
        .build());

TimeLimiter timeout = TimeLimiter.of(TimeLimiterConfig.custom()
    .timeoutDuration(Duration.ofMillis(800))  // per attempt, well under the caller's deadline
    .cancelRunningFuture(true)
    .build());

CircuitBreaker breaker = CircuitBreaker.of("inventory", CircuitBreakerConfig.custom()
    .failureRateThreshold(50)
    .slidingWindowSize(20)
    .waitDurationInOpenState(Duration.ofSeconds(10))
    .build());

// Budget (illustrative): 2 attempts x 800 ms + 100 ms backoff = 1.7 s, inside a 2 s caller deadline.
Retry retry = Retry.of("inventory", RetryConfig.custom()
    .maxAttempts(2)              // small count: each attempt takes a new bulkhead permit
    .waitDuration(Duration.ofMillis(100))
    .build());

ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2);

CompletableFuture<String> result = Decorators
    .ofSupplier(() -> inventoryClient.fetchStock("sku-123"))
    .withThreadPoolBulkhead(bulkhead)        // innermost: bounds concurrency
    .withTimeLimiter(timeout, scheduler)     // bounds how long the caller waits per attempt
    .withCircuitBreaker(breaker)             // open breaker rejects before a permit is taken
    .withRetry(retry, scheduler)             // outermost: every retry re-acquires a permit
    .get()
    .toCompletableFuture();

// The time limiter stops the wait, but cancelling the future may not stop the worker thread.
// Also set connect/read timeouts on inventoryClient so the permit is actually freed.

A Resilience4j decorator chain that applies retry, circuit breaker, timeout and a thread-pool bulkhead from outermost to innermost, with the bulkhead sized to the HTTP connection pool and the retry budget kept inside the caller's deadline.

How granular should bulkheads be: per downstream, endpoint or tenant?

Start with one bulkhead per downstream service and split further only when you can name the failure the split prevents. A downstream is the natural unit because it usually fails as a whole: when it slows, its connection pool, network path and host all slow together. Split by endpoint when one operation on that downstream is both slow and less important, such as a report export that should not consume permits needed by a checkout lookup.

Granularity Blast radius contained Main cost Use when
Per downstream One slow dependency Few pools, easy to size Default for most services
Per critical endpoint One slow operation More limits to tune per dependency Latency differs sharply between endpoints
Per tenant or priority class One noisy caller Fragmented capacity Shared service with uneven or contractual load

Every pool adds cost. Thread-pool bulkheads reserve memory for each thread's stack, add context switching, and add configuration that someone must review whenever traffic changes. A service with dozens of poorly sized pools can perform worse than one with three well-chosen pools, because each pool is either too small and rejects requests under normal bursts, or too large and protects nothing.

Per-tenant isolation improves fairness but fragments capacity. A quiet tenant's pool holds idle permits while a busy tenant's pool rejects requests, even though total capacity is available. Priority classes, such as one pool for interactive traffic and one for batch traffic, usually give most of the fairness with less waste.

Skip bulkheads when a service has a single dependency, because there is nothing to isolate it from. Also skip them when calls are uniformly fast and bounded by timeouts. In those cases, a limit on the server's request concurrency does the same job.

How do you monitor bulkheads, and what are the alternatives?

Export five signals per bulkhead, tagged with the dependency name: in-flight count, available permits, queue depth (thread-pool bulkheads only), rejection rate, and call latency for calls that got a permit. Each answers a different question. Permits near zero with flat latency means the limit is too tight for normal traffic, so raise it. Permits near zero with rising latency means the downstream is slowing and the bulkhead is doing its job, so leave the limit alone. A queue that is often non-empty means callers are waiting rather than failing fast.

Alert on sustained saturation, such as available permits at zero for several minutes, rather than on single rejections. Brief rejections during a traffic spike are the intended behaviour. Paging on them trains people to ignore the alert.

Bulkhead metrics also shorten incidents. When one dependency's pool saturates while the others stay healthy, the dashboard points at the cause directly. Responders can skip tracing thread dumps across the whole service.

Approach What it limits Trade-off
Static bulkhead Concurrency per dependency Limits go stale as latency changes
Adaptive concurrency limiting Concurrency, adjusted from observed latency More tuning and harder to reason about
Rate limiting / load shedding Request rate or admission at the edge Does not isolate one slow dependency

On async or virtual-thread runtimes, a slow call no longer pins a scarce platform thread, so thread-pool isolation matters less. You still need a per-dependency concurrency limit, usually a semaphore. Without one, you can overload the downstream and exhaust the shared connection pool.

How can bulkheads save engineering time?

Most of the time bulkheads save comes from turning wide, confusing incidents into narrow, obvious ones.

  • Fewer cascading incidents to investigate. Without isolation, one slow dependency can exhaust the shared pool, fail health checks and trigger restart loops across replicas, as described at the start of this article. Those incidents are expensive because the visible symptoms, such as errors on every endpoint and instances restarting, point away from the cause. With bulkheads, the same downstream slowdown shows up as rejections on one dependency, and the rest of the service keeps working. A partial outage with a known cause takes less effort to resolve than a full outage that first has to be diagnosed.
  • Faster diagnosis. Per-dependency metrics answer "which downstream is the problem?" directly from a dashboard. Responders do not need to take thread dumps, correlate stack traces or bisect recent deploys to find which calls are holding the threads.
  • Less firefighting during the incident. Because the blast radius is contained, the usual emergency steps, such as scaling out, restarting instances or hot-patching timeouts, are often unnecessary. Scaling out is also less likely to backfire: without per-instance limits, new replicas can simply add more stuck calls against the slow downstream.
  • Tuning by configuration, not code. Once calls go through named bulkheads, adjusting a limit, a queue size or a maximum wait is a configuration change reviewed against metrics. It does not require reworking the call path each time a dependency's traffic or latency changes.
  • Failure behaviour designed once. Deciding up front what a rejection returns (fast failure, stale data, partial result) replaces ad hoc decisions made under pressure during an outage. It also makes the behaviour testable: load tests with injected downstream latency show what users will see before production does.
  • Shared, standard building blocks. Using an existing library's bulkhead, timeout, circuit breaker and retry decorators, rather than hand-rolled concurrency control per client, means less bespoke code to write, review and debug, and one consistent set of metrics across services.

The time is not free. Someone has to size the limits, write fallbacks, run latency-injection tests and revisit limits as traffic changes. The trade is worth it when the service has several dependencies that can degrade independently. In that case the recurring cost of maintaining a few well-chosen limits is usually smaller than the cost of recurring service-wide incidents that start in a single downstream.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to bulkhead pattern microservices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.