What is the bulkhead pattern in microservices, and what failure does it prevent?
The bulkhead pattern gives each downstream dependency its own bounded concurrency budget: a fixed number of calls that may be in flight to that dependency at once. When the budget is spent, new calls to that dependency are rejected or queued briefly, instead of borrowing capacity that the rest of the service needs.
The failure it prevents
Take a synchronous, thread-per-request service with one shared pool of servlet worker threads. Endpoints /orders and /profile call a pricing service. /health and /catalog do not. Illustrative estimate, with assumed inputs: 200 worker threads and 100 requests per second to pricing-backed endpoints.
- Pricing slows from 50 ms to 10 s per call, but does not fail fast.
- Each request that calls pricing now holds a worker thread for about 10 seconds instead of 50 ms.
- At 100 requests per second, all 200 threads are blocked on pricing within about two seconds.
- Requests to
/catalogarrive, find no free thread, queue, and then time out. - The load balancer sees health checks fail and removes healthy instances, which concentrates traffic on the remaining ones.
The service never called pricing from /catalog, yet /catalog went down. A per-dependency bulkhead caps pricing at a small share of the threads, so the rest stay available for other work.
Who benefits most
Teams running blocking, thread-per-request services that fan out to several downstreams of uneven reliability benefit most. The payoff shows up during incidents. A slow dependency degrades one feature instead of causing a full outage. Triage also gets narrower: the bulkhead that is rejecting calls points straight at the dependency causing the problem.
WITHOUT BULKHEAD: one shared pool (illustrative: 200 threads)
/orders /profile /catalog /health
v v v v
+----------------------------------------------+
| Shared worker pool: 200/200 threads busy |
| every thread blocked on pricing (~10 s) |
+----------------------------------------------+
| | X no free thread:
v v queue, time out,
+------------------------+ health checks fail
| Pricing: 50 ms -> 10 s |
+------------------------+
WITH BULKHEAD: per-dependency budget (illustrative cap: 20)
/orders /profile /catalog /health
v v v v
+----------------------------------------------+
| Worker pool: 200 threads |
| +-------------------+ +-------------------+ |
| | Pricing bulkhead | | ~180 threads free | |
| | max 20 in flight | | for /catalog, | |
| | full -> reject | | /health, etc. | |
| +-------------------+ +-------------------+ |
+----------------------------------------------+
| |
v v
+------------------------+ responses served; only
| Pricing: 50 ms -> 10 s | pricing-backed endpoints
+------------------------+ degrade
Illustrative estimate with assumed inputs: without a bulkhead, a slow pricing service ties up every shared worker thread and takes down /catalog and /health; with a per-dependency cap, only pricing calls are rejected and the remaining threads keep serving other endpoints.
How does bulkhead isolation work: semaphore vs thread pool isolation?
There are two ways to build that boundary, and they differ in which thread does the waiting.
Semaphore isolation puts a counter in front of the call. The caller's own thread acquires a permit, makes the call, and releases the permit. When no permits are free, the call is rejected or waits briefly. It costs almost nothing. With a blocking client, though, the request thread stays parked inside the call until it returns. The semaphore limits how many request threads one dependency can hold, but each of those threads stays occupied for the full duration of the call.
Thread pool isolation sends the call to a dedicated executor with a bounded queue. The caller receives a future and can stop waiting on its own deadline, independently of the downstream call. The worker thread is still stuck until the call returns, but the stall stays inside that dependency's pool. The cost is context switches, more threads, and explicit propagation of thread-local context such as MDC fields, trace IDs, and security context.
Reactive and virtual-thread services change this trade-off. A parked virtual thread or a pending non-blocking call costs little, so a dedicated pool buys almost nothing. The resource worth protecting is the dependency's capacity, which means a concurrency cap is usually the right control.
| Aspect | Semaphore | Thread pool |
|---|---|---|
| Overhead | Counter check | Thread handoff, queue, extra threads |
| Timeout behaviour | Caller waits as long as the client blocks | Caller can abandon the future on its own deadline |
| Thread the call runs on | Caller's thread | Pool worker thread |
| Context propagation | Automatic (same thread) | Must be copied explicitly |
| Best-fit clients | Non-blocking, async, virtual threads | Blocking clients on platform threads |
How do you size a bulkhead with Little's Law?
Little's Law states that the average number of requests in flight equals the arrival rate multiplied by the time each request spends in the system. For a bulkhead, that gives the concurrency limit you need: calls per second to the dependency times how long each call holds a permit.
Illustrative estimate. Assume one instance sends 200 req/s to a pricing service with a p99 latency of 50 ms. That is 200 × 0.05 = 10 concurrent calls. Add roughly 50% headroom for bursts and you get a limit of about 15.
If latency degrades while traffic stays the same, the demand on the bulkhead grows with it:
| Latency | Concurrency needed (200 req/s) | Fits a limit of 15? |
|---|---|---|
| 50 ms | 10 | Yes |
| 200 ms | 40 | No, rejects about 60% |
| 1 s | 200 | No, rejects about 90% |
Size the limit for the worst latency you would still accept serving, not for the worst case you can imagine. A limit of 200 would absorb a 1 s stall, but it would also let that dependency hold 200 threads, which is the failure the bulkhead exists to stop. Rejections at 200 ms are the bulkhead doing its job.
Queue capacity needs the same deliberate choice. A large queue hides saturation: callers wait, latency climbs, and nothing is rejected until timeouts start firing. A small or zero queue fails fast, so the fallback runs while it is still useful.
Limits apply per instance. Fifteen permits across 40 replicas means the dependency can see up to 600 concurrent calls. Autoscaling raises that number without anyone changing the configuration, so check it against the dependency's real capacity.
Traffic and latency drift over time, so a static limit becomes wrong. Review limits against observed metrics on a regular schedule.
How do you implement a resilience4j bulkhead per downstream dependency?
Create one named instance per downstream dependency, such as payments, inventory and search. The name scopes the limit and labels its metrics, which lets you see which dependency is saturating. Get instances from BulkheadRegistry or ThreadPoolBulkheadRegistry so every caller that uses the same name shares the same pool and limit. If you construct a bulkhead inline at each call site, each site gets its own counter and the limit is never enforced.
Semaphore bulkhead
Bulkhead caps in-flight calls with maxConcurrentCalls. The call runs on the caller's thread. When all permits are taken, the caller blocks for up to maxWaitDuration and is then rejected. For request-serving threads, a short wait or zero is usually right, because waiting holds the request thread you are trying to protect. In Spring Boot, set these under resilience4j.bulkhead.instances.<name>.
Thread pool bulkhead
ThreadPoolBulkhead hands work to a dedicated executor that coreThreadPoolSize, maxThreadPoolSize and queueCapacity define, and it returns a CompletionStage. A bounded queue absorbs short bursts. Once the queue is full, calls are rejected immediately. Configure it under resilience4j.thread-pool-bulkhead.instances.<name>. Code-based config is useful in tests, but keep production limits in properties so they can be tuned without a rebuild.
Context propagation
MDC, tracing spans and security context are typically stored in thread-locals. They stay on the calling thread, so logs lose correlation IDs, spans detach from their parents, and authorization checks see no principal. To fix this, wrap the task at submission. Capture the context on the caller's thread, restore it inside the worker before the task runs, and clear it in a finally block so a pooled thread doesn't leak one request's identity into the next request. Resilience4j's thread pool configuration accepts context propagators for this purpose. Your tracing library may also provide executor or task wrappers that do the same thing.
import io.github.resilience4j.bulkhead.ThreadPoolBulkhead;
import io.github.resilience4j.bulkhead.ThreadPoolBulkheadRegistry;
import java.util.Map;
import java.util.concurrent.CompletionStage;
import java.util.function.Supplier;
import org.slf4j.MDC;
public class PaymentsClient {
// One shared, named instance. Limits come from
// resilience4j.thread-pool-bulkhead.instances.payments.*
private final ThreadPoolBulkhead bulkhead;
public PaymentsClient(ThreadPoolBulkheadRegistry registry) {
this.bulkhead = registry.bulkhead("payments");
}
public CompletionStage<String> charge(String orderId) {
// Without the wrapper, logs written on the worker thread have no correlation ID.
return bulkhead.executeSupplier(withMdc(() -> callPaymentsApi(orderId)));
}
// Capture on the caller's thread and restore on the worker. Apply the same
// pattern to the tracing span and the security context.
static <T> Supplier<T> withMdc(Supplier<T> task) {
Map<String, String> captured = MDC.getCopyOfContextMap(); // runs on the caller's thread
return () -> {
if (captured != null) {
MDC.setContextMap(captured); // runs on the pool thread
}
try {
return task.get();
} finally {
MDC.clear(); // stop the context leaking into the next task on this pooled thread
}
};
}
private String callPaymentsApi(String orderId) { return "charged:" + orderId; }
}
A named ThreadPoolBulkhead from the registry, with each submitted task wrapped so the caller's MDC is captured, restored on the worker thread and cleared afterward.
How should bulkheads be combined with timeouts, circuit breakers, and retries?
Order matters because each decorator wraps the one inside it. A reasonable composition, outermost to innermost: retry → circuit breaker → time limiter → bulkhead → client call.
Keep retries outside the bulkhead. If the retry sits inside, one request holds its slot through every attempt and every backoff.
Illustrative estimate (assumed numbers, not a benchmark): slot hold time = attempts × per-attempt timeout + total backoff. With 3 attempts × 2 s + 2 backoffs × 0.5 s, a slot is held for 7 s instead of 2 s. That is 3.5× the occupancy for the same traffic, so the bulkhead fills while the dependency is degraded, which is exactly when you need slots free.
Size timeouts from the caller's deadline. If your caller gives up at 3 s, a 5 s dependency timeout wastes a slot on work nobody will read. Budget per-attempt timeout × attempts plus backoff below the inbound deadline.
Use the circuit breaker and bulkhead for different jobs. The circuit breaker stops sending work to a dependency it already knows is failing. The bulkhead caps how much work is in flight while the circuit breaker is still collecting evidence.
Configure client-level timeouts as well. A time limiter abandons the wait, but it does not cancel the remote work or unblock a thread stuck in a socket read. Set connect and read timeouts on the client itself.
Handle rejections explicitly. Map a bulkhead-full error to a fallback, a cached or degraded response, or a fast 503 or 429 with documented retry semantics. Exclude it from retry, for example with Resilience4j's ignoreExceptions on BulkheadFullException. Retrying a rejection into the same full bulkhead only adds load.
// Outermost to innermost: retry -> circuit breaker -> time limiter -> bulkhead -> client.
// In Decorators, the first with* call is the innermost layer.
HttpClient http = HttpClient.newBuilder()
.connectTimeout(Duration.ofMillis(300)) // client-level timeouts still required
.build();
HttpRequest req = HttpRequest.newBuilder(URI.create(""))
.timeout(Duration.ofMillis(900)) // stops waiting on a hung response
.build();
ThreadPoolBulkhead bulkhead = ThreadPoolBulkhead.of("pricing", ThreadPoolBulkheadConfig.custom()
.coreThreadPoolSize(10).maxThreadPoolSize(10).queueCapacity(5).build());
TimeLimiter limiter = TimeLimiter.of(TimeLimiterConfig.custom()
.timeoutDuration(Duration.ofSeconds(1)).cancelRunningFuture(true).build());
CircuitBreaker breaker = CircuitBreaker.ofDefaults("pricing");
// Budget under a 3 s inbound deadline: 2 attempts x 1 s + 0.5 s backoff = 2.5 s
Retry retry = Retry.of("pricing", RetryConfig.custom()
.maxAttempts(2).waitDuration(Duration.ofMillis(500))
// never retry a rejection back into the same full bulkhead or open breaker
.ignoreExceptions(BulkheadFullException.class, CallNotPermittedException.class)
.build());
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2);
Supplier<String> call = () -> {
try { return http.send(req, HttpResponse.BodyHandlers.ofString()).body(); }
catch (Exception e) { throw new RuntimeException(e); }
};
CompletionStage<String> quote = Decorators.ofSupplier(call)
.withThreadPoolBulkhead(bulkhead) // innermost: caps in-flight calls
.withTimeLimiter(limiter, scheduler) // per-attempt wait limit
.withCircuitBreaker(breaker) // stop calling a known-bad dependency
.withRetry(retry, scheduler) // outermost: each attempt takes a fresh slot
// rejections become a degraded response instead of an error or a retry
.withFallback(List.of(BulkheadFullException.class, CallNotPermittedException.class),
t -> "{\"price\":null,\"degraded\":true}")
.get();
A Resilience4j composition that keeps retries outside the bulkhead, fits the per-attempt timeouts and backoff inside a 3 s caller deadline, still sets connect and response timeouts on the HTTP client, and maps bulkhead and circuit-breaker rejections to a degraded fallback instead of retrying them.
How do you monitor and test failure isolation?
Instrument every bulkhead before you change a single size. Each one needs five signals, tagged by dependency name:
- Available permits (semaphore) or active threads (thread pool), which show how close you run to the limit.
- Queue depth for thread-pool bulkheads. A queue that is always non-empty means added latency that callers cannot see.
- Rejected call count, the direct sign that isolation is working, or that the bulkhead is undersized.
- Call latency to the dependency, measured inside the bulkhead so queue wait does not hide it.
- Fallback rate, measured separately from errors.
Resilience4j publishes bulkhead metrics and events through its common metrics integrations, such as Micrometer, so these signals can reach your existing dashboards without custom code. Alert on rejections that last several minutes rather than on a single spike, because short bursts are expected. Alert on fallback rate as well. A fallback that returns cached or default data keeps the error rate flat, so a dependency can be down for hours while your dashboards look healthy.
Proving isolation with a load test
Run steady mixed traffic against the service, then inject latency into one dependency using a fault-injecting proxy or a stub that sleeps. The test passes if both of these hold:
- Endpoints that never call the slow dependency keep their baseline latency and error rate.
- The slow dependency's bulkhead fills and starts rejecting calls instead of queuing without limit.
If the first check fails, threads or connections are still shared somewhere. Common culprits are the servlet pool, an HTTP client connection pool, or a common executor. Finding that in a test costs about an hour. Finding it in production means a cascading outage, an incident review, and a rushed fix while every endpoint is down at once.
When should you avoid the bulkhead pattern, and what are the alternatives?
Bulkheads cost resources even when nothing is failing. Each thread-pool bulkhead adds memory for stacks and queues, extra context switches, and a handoff between threads on every call. A service with forty dependencies should not get forty pools. Isolate the risky or high-volume dependencies and let stable, low-traffic calls share a pool.
Splitting also reduces usable capacity. A shared pool of 100 threads absorbs uneven load. Four bulkheads of 25 cannot lend idle slots to each other, so one busy dependency gets rejected while the other bulkheads have free slots. In reactive or virtual-thread services, a call that is waiting does not hold a platform thread. Dedicated platform pools there usually add overhead without adding protection, so a semaphore-style concurrency limit is the better fit.
Bulkheads also cost engineering time. Each one needs configuration, context propagation across the thread handoff (tracing, MDC, security context), its own metrics, and periodic limit reviews as traffic changes. A stale limit is worse than no limit.
| Alternative | Fits when | Gap |
|---|---|---|
| Plain client timeouts | Few dependencies, low concurrency | Slow calls still hold threads up to the timeout |
| Circuit breaker alone | Failures are fast and clear | Does not cap concurrency before the breaker trips |
| Adaptive concurrency limits | Latency changes over time and static sizing goes stale | Harder to reason about and tune |
| Rate limiting | You need to protect a downstream quota | Limits request rate, not threads held by in-flight calls |
| Edge load shedding | Overload starts with inbound traffic | Cannot see which dependency is slow |
| Separate instances per workload | Workloads differ sharply in criticality | Higher infrastructure and deployment cost |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to bulkhead pattern microservices, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.