What is a Resilience4j circuit breaker and how does its state machine work?
A Resilience4j circuit breaker wraps calls to a remote dependency and tracks whether they succeed. When too many calls fail or run slowly, it stops making them for a while. A Spring service without one keeps sending requests to a dependency that is already struggling. Each request ties up a thread and a connection until it times out, and the extra load makes the dependency's recovery harder. The breaker replaces that slow, expensive failure with a fast, cheap one.
The breaker has three normal states:
- CLOSED: Calls pass through, and the breaker records each outcome in a sliding window. Once the window holds enough calls to be meaningful, the breaker checks the failure rate and the slow-call rate. If either exceeds its threshold, it moves to OPEN. In this state the caller sees the dependency's real responses and exceptions.
- OPEN: Every call is rejected immediately with
CallNotPermittedException, and the dependency is never contacted. The breaker stays OPEN for a configured wait duration and then moves to HALF_OPEN. This is the state where your fallback does its work. - HALF_OPEN: A limited number of trial calls go through. Any calls beyond that limit are rejected with
CallNotPermittedException. When the trials finish, the breaker evaluates them against the same thresholds. If they pass, it returns to CLOSED. If they fail, it goes back to OPEN and the wait starts again.
State is held in memory and is separate for every service instance. Each replica builds its own failure window, runs its own trial calls, and decides on its own when to close. At any moment, some replicas may consider a dependency healthy while others are rejecting calls to it. Account for this when you read dashboards and when you reason about how much trial traffic a recovering dependency will receive.
failure or slow-call rate >= threshold
+--------------------+ (enough calls in window) +--------------------+
| CLOSED |-------------------------->| OPEN |
| calls pass through | | calls rejected at |
| outcomes recorded | | once with |
| in sliding window | | CallNotPermitted- |
| caller sees real | | Exception; no call |
| results and errors | | to the dependency |
+--------------------+ +--------------------+
^ | ^
| trials within wait | | trials
| thresholds duration | | fail
| elapses v |
| +------------------------------+ |
+-------------------| HALF_OPEN |--------+
| limited trial calls allowed; |
| extra calls rejected with |
| CallNotPermittedException |
+------------------------------+
State is in memory per instance; replicas decide independently:
+----------------------+ +------------+
| Replica A: CLOSED |---- real calls ---->| |
| Replica B: OPEN |--X no calls made | Dependency |
| Replica C: HALF_OPEN |---- trial calls --->| |
+----------------------+ +------------+
Resilience4j circuit breaker state transitions and per-replica state: CLOSED records outcomes, OPEN fails fast with CallNotPermittedException, HALF_OPEN admits limited trial calls, and each instance can be in a different state for the same dependency.
Who needs a circuit breaker, and when should you avoid one?
The breaker pays off for backend and platform engineers whose services make synchronous calls to remote dependencies that can fail or slow down, such as HTTP APIs, payment gateways, or a shared database behind another team's service. It helps most when a dependency's failures tend to persist for seconds or minutes, and when failing fast frees the caller to serve a degraded response instead of queueing work that will fail anyway.
It is not enough on its own. The breaker only counts calls that complete. Without a bounded call duration, request threads can all block on a hung dependency before enough calls finish to trip it. In the CLOSED state, it also places no limit on how many calls run at once. Pair it with mechanisms that bound time and concurrency:
| Mechanism | What it bounds | Gap it leaves |
|---|---|---|
| Client timeout | Duration of one call | No memory of repeated failures |
| TimeLimiter | Duration of async calls | Does not apply to plain blocking calls |
| Bulkhead | Concurrent calls | Keeps calling a dead dependency |
| Retry | Transient failures | Multiplies load during outages |
| RateLimiter | Call rate | Ignores whether calls succeed |
| Queue or async decoupling | Caller's coupling to latency | Adds delivery and ordering complexity |
If a call can be moved onto a queue, decoupling may remove the need for a breaker altogether.
The breaker also has a development cost. You take on more configuration, fallback paths that need their own tests, and metrics that someone must watch. Weigh that against the time saved when an outage stays contained to one dependency and the breaker's state transitions point directly at the cause.
How do you choose a sliding window and failure thresholds?
Once you have decided to use a breaker, the window determines what evidence it acts on. A COUNT_BASED window holds the last N calls, so slidingWindowSize is a number of calls. A TIME_BASED window holds calls from the last N seconds. In both cases, minimumNumberOfCalls is how many calls must be recorded before the breaker calculates a failure rate.
On a low-traffic endpoint, a small minimum is risky. With a minimum of 5 and a failureRateThreshold of 50, three transient errors among the first five calls produce a 60% failure rate and open the breaker, which is a false open.
Illustrative estimate. Assume a full outage begins after a period of only successful calls, with a threshold of 50% and a minimum of 20 calls.
| Window | Assumed traffic | Approximate time to open |
|---|---|---|
| COUNT_BASED, 100 calls | 50 req/s | ~1 s (50 failures) |
| COUNT_BASED, 100 calls | 0.5 req/s | ~100 s |
| TIME_BASED, 60 s | 50 req/s | ~30 s |
| TIME_BASED, 60 s | 0.5 req/s | ~30 s |
A count window reacts in proportion to traffic volume. A time window reacts in proportion to wall-clock time.
Some dependencies degrade by getting slow rather than by throwing errors. To catch this, set slowCallDurationThreshold below your latency budget and set slowCallRateThreshold so that a sustained run of slow calls opens the breaker.
Configure which exceptions count:
recordExceptions: Use this for connection errors and timeouts. If you set it, any exception not on the list counts as a success.ignoreExceptions: Use this for 4xx client errors and business validation failures. These reflect the caller's input, not the dependency's health, so the breaker should count them as neither failure nor success.
The tuning trade-off is sensitivity against stability. A small window, a low minimum, and a low threshold make the breaker open quickly, but it will also flap open and closed during brief noise. Larger values make it stable, but it lets a real outage run longer before it opens.
# application.yml: Resilience4j circuit breaker for one downstream dependency
resilience4j:
circuitbreaker:
instances:
inventoryService:
# Evaluate the last 60 seconds of calls, not the last N calls,
# so reaction time does not depend on request volume.
slidingWindowType: TIME_BASED
slidingWindowSize: 60
# Require enough evidence before computing a rate. A low value
# lets two or three unlucky calls open the breaker on a quiet endpoint.
minimumNumberOfCalls: 20
failureRateThreshold: 50
# Treat calls slower than the latency budget as unhealthy, so a
# dependency that slows down without throwing still trips the breaker.
slowCallDurationThreshold: 2s
slowCallRateThreshold: 80
# Recovery behaviour after the breaker opens.
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 5
# Only these exceptions, and their subclasses, count as failures.
# Any other exception not ignored below counts as a success.
recordExceptions:
- java.net.ConnectException
- java.net.SocketTimeoutException
- java.util.concurrent.TimeoutException
- org.springframework.web.client.HttpServerErrorException
# These reflect the caller's input, not the dependency's health.
# They count as neither failure nor success.
ignoreExceptions:
- org.springframework.web.client.HttpClientErrorException
- com.example.inventory.ValidationException
A Spring Boot YAML configuration that uses a time-based window with a minimum call count, adds slow-call thresholds, and records connection errors, timeouts and 5xx responses as failures while ignoring 4xx client errors and validation exceptions.
How should you tune recovery from the OPEN state?
Once the breaker opens, three settings decide how recovery plays out. waitDurationInOpenState sets how long calls are rejected before the breaker lets trial traffic through. permittedNumberOfCallsInHalfOpenState sets how many trial calls are allowed in the HALF_OPEN state. The breaker returns to CLOSED or OPEN based on the failure rate of those trial calls. automaticTransitionFromOpenToHalfOpenEnabled decides what triggers the move to HALF_OPEN. With it off, the transition happens on the first call after the wait expires, so an idle path stays OPEN indefinitely. With it on, a timer moves the breaker to HALF_OPEN even when no traffic arrives.
A short wait restores service faster, but it probes a dependency that may still be restarting or draining a backlog. A longer wait gives the dependency room to recover, at the cost of rejecting calls it could already handle.
Every replica runs its own breaker, so the trial load multiplies. Illustrative estimate: with 10 permitted half-open calls and 30 replicas, a recovering dependency can receive 300 near-simultaneous requests the moment the wait expires. If the replicas opened at about the same time, they will probe at about the same time. Size the trial count for the whole fleet, not for one instance.
Breaker granularity
One breaker per downstream service is easy to configure and monitor. However, if only one slow endpoint fails, it can open the breaker and block that service's healthy endpoints too. A breaker per endpoint, or per tenant, contains the fault where it starts. The cost is more instances to configure, more metric series to track, and more state to reason about during an incident.
# application.yml - Resilience4j circuit breakers, one instance per endpoint
resilience4j:
circuitbreaker:
configs:
default:
slidingWindowType: COUNT_BASED
slidingWindowSize: 50
minimumNumberOfCalls: 20
failureRateThreshold: 50
# Reject calls for 30s so the dependency can restart or drain its backlog
waitDurationInOpenState: 30s
# Trial calls per replica. With 30 replicas this is up to 90 probes
# fleet-wide, not 3. Size the value for the whole fleet.
permittedNumberOfCallsInHalfOpenState: 3
# A timer moves OPEN -> HALF_OPEN even when no calls arrive,
# so idle or low-traffic paths do not stay OPEN indefinitely
automaticTransitionFromOpenToHalfOpenEnabled: true
instances:
# Separate breakers per endpoint of the same downstream service,
# so a failing search endpoint does not block order lookups
inventorySearch:
baseConfig: default
# Expensive endpoint that is slow to recover: wait longer, probe less
waitDurationInOpenState: 60s
permittedNumberOfCallsInHalfOpenState: 2
inventoryGetItem:
baseConfig: default
inventoryReserve:
baseConfig: default
# Writes are critical: fewer trial calls during recovery
permittedNumberOfCallsInHalfOpenState: 2
A Spring Boot application.yml with a shared default config that tunes the wait duration, half-open trial calls and the automatic OPEN-to-HALF_OPEN transition, plus separate breaker instances for each endpoint of one downstream service so a fault on one endpoint does not block the others.
How do you configure a Resilience4j circuit breaker in Spring Boot?
In Spring Boot, the configuration lives in application.yml under resilience4j.circuitbreaker. Declare a shared base under configs (commonly default) holding your window type, size, and thresholds. Then define each named instance under instances and point it at that base with baseConfig, overriding only what differs. This keeps per-endpoint breakers cheap to add and makes deviations from the baseline obvious in review.
To apply an instance, annotate the calling method with @CircuitBreaker(name = "inventory", fallbackMethod = "inventoryFallback"). The annotation works through Spring AOP, so the aspect support must be on the classpath. The fallback must return the same type as the protected method and take the same parameters, plus one trailing exception parameter. Resilience4j matches the fallback by that signature, and the exception type you declare decides which failures it handles. Declaring CallNotPermittedException handles only rejections from an OPEN breaker. Declaring Throwable handles everything.
Three mistakes leave a method unprotected without any obvious error:
- Self-invocation. Calling the annotated method from another method in the same bean skips the Spring proxy, so the breaker never runs. Move the protected call into a separate bean.
- Fallback signature mismatch. The fallback is resolved by reflection, so a wrong parameter list or return type may surface only when a failure first needs the fallback, which is usually during an incident.
- Instance-name typos. A name with no matching entry typically gets an instance built from default settings. You get a breaker, just not the one you tuned.
Property names and defaults change between releases. Verify them, including the values shown here, against the documentation for the Resilience4j and Spring Boot versions you actually run.
// InventoryClient.java: protected calls live in their own bean so callers go through the Spring proxy
@Service
public class InventoryClient {
private final InventoryApi api; // placeholder for your HTTP client
public InventoryClient(InventoryApi api) { this.api = api; }
// name must match resilience4j.circuitbreaker.instances.inventory exactly
@CircuitBreaker(name = "inventory", fallbackMethod = "inventoryFallback")
public StockLevel getStock(String sku) {
return api.fetchStock(sku);
}
// Same return type and parameters, plus a trailing exception.
// Handles only rejections while the breaker is OPEN.
private StockLevel inventoryFallback(String sku, CallNotPermittedException ex) {
return StockLevel.unknown(sku);
}
// Catch-all for other failures (timeouts, 5xx). The most specific matching type is chosen.
private StockLevel inventoryFallback(String sku, Throwable ex) {
return StockLevel.unknown(sku);
}
}
// OrderService.java: calls through the injected bean, not this.getStock(...)
@Service
public class OrderService {
private final InventoryClient inventory;
public OrderService(InventoryClient inventory) { this.inventory = inventory; }
public boolean canFulfil(String sku, int qty) {
// Goes through the proxy, so the breaker runs. A self-call inside InventoryClient would bypass it.
return inventory.getStock(sku).available() >= qty;
}
}
This example puts the @CircuitBreaker method in a separate bean so calls go through the Spring proxy, and pairs it with fallbacks that match its signature plus an exception parameter: one for OPEN-state rejections and one catch-all.
What makes a fallback honest instead of misleading?
A fallback decides what the caller believes happened. If it returns an empty list or a zero balance when the downstream service fails, the caller treats that as a real answer. Your error-rate dashboards stay green because nothing threw, and users act on wrong data. A visible error is usually better than a plausible lie.
| Strategy | Honest when | Misleading when |
|---|---|---|
| Cached or stale data | The response carries its age or a staleness flag | It is presented as current |
| Degraded default | The default is truly neutral, such as hiding a recommendations panel | It stands in for a fact, such as price, stock, or permissions |
| Queued retry | The caller gets "accepted, pending" and the operation is idempotent | It reports success before the work happens |
| Fail fast with clear error | Always; the caller can decide | Rarely, unless the error is vague |
Inside the fallback, separate the two causes. CallNotPermittedException means the breaker is OPEN and no call was made. Log it at low severity and don't page anyone, because the original failures were already recorded. Any other exception is a real downstream failure: log it with context and let it count toward alerting. With the Spring annotations, you can declare separate fallback methods whose last parameter is a different exception type. The most specific match is chosen, so each path stays small.
Fallback code runs only during incidents, so it decays quietly. A cache that was never populated or a serializer that was never called will fail exactly when you need it. Write tests that force the breaker OPEN and assert what the fallback returns.
How should the circuit breaker be combined with TimeLimiter and Retry?
A circuit breaker only records an outcome when a call finishes. A request that hangs on a stalled socket is never counted as a failure or as a slow call, so the breaker keeps sending traffic while threads pile up. Every protected call needs a deadline. For asynchronous methods, a TimeLimiter cancels the future and the breaker records a timeout. For synchronous calls, set connect and read timeouts on the HTTP or database client. A slow-call threshold only works if calls actually end.
Ordering with Retry
Where you place Retry determines what the breaker sees.
- Retry inside the breaker. The breaker records one outcome per logical call, while the dependency receives every attempt. Failures look less frequent than they are, the breaker opens late, and the retries become a retry storm against a service that is already struggling.
- Retry outside the breaker. Each attempt passes through the breaker and is recorded. Once the breaker opens, later attempts fail immediately with
CallNotPermittedException. Configure Retry to skip that exception. Retrying a rejection only adds latency.
Illustrative estimate (assumed numbers, not a benchmark): downstream load = R × A₁ × A₂ × …, where R is the incoming request rate and Aₙ is the maximum attempts at each layer that retries. At an assumed 200 req/s with 3 attempts in your service and 3 more in a gateway, the dependency can receive up to 200 × 3 × 3 = 1,800 req/s.
In the Spring Boot integration, the annotation aspects apply Retry as the outermost layer by default, with the CircuitBreaker inside it and the TimeLimiter closer to the actual call. The aspect order is configurable. Confirm the default in your version's documentation rather than assuming it.
How do you verify and monitor circuit breaker behavior?
Verification has two parts: watching the breaker in production and proving its behavior in tests before it ships.
Expose state and metrics
With the Resilience4j Spring Boot starter and Actuator on the classpath, each breaker instance can register a health indicator through registerHealthIndicator in its instance config. The starter also provides Actuator endpoints that list breakers and their recent events. Micrometer publishes gauges and counters for each instance, including the current state, calls grouped by outcome, and calls that were not permitted. Check the metric names your version emits before you build dashboards on them.
Alert on two signals. The first is a state transition to OPEN, which tells you a dependency is degraded. The second is the rate of not-permitted (rejected) calls, which shows how much user traffic is being served by the fallback. A breaker that moves between OPEN and HALF_OPEN for a long time without closing points to a wait duration or permitted-call count that needs tuning, not to a passing incident.
Test the state machine
Put the dependency behind a stubbed HTTP server and use a test profile with a small window and a short waitDurationInOpenState. Then:
- Return errors or delayed responses until the breaker opens. Read its state from the
CircuitBreakerRegistryand assert that it is OPEN. - Assert that the next calls return the fallback without reaching the stub.
- Wait out the open duration, restore healthy responses, and assert that the breaker passes through HALF_OPEN and returns to CLOSED.
These tests catch mistakes that produce no error on their own:
- Self-invocation: a method calls another method on the same class, so the call bypasses the proxy and the annotation does nothing.
- Misspelled instance name: the breaker silently uses the default configuration instead of the one you wrote.
- Fallback signature mismatch: the fallback does not match the protected method. Depending on the version, this fails at startup or at the moment the fallback is first needed.
If these problems are found during an incident, someone has to reconstruct why the protection never engaged while the outage is still in progress. A failing test catches the same problem in CI, where it costs a few minutes to fix.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to resilience4j circuit breaker, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.