What is a circuit breaker and how do its states work?
A circuit breaker wraps calls to a remote dependency and stops making them once that dependency is clearly failing. Without one, every request to a slow or broken service waits for its timeout while holding a thread, a pooled connection, or an event-loop callback slot. Those resources run out well before the timeout pressure ends, and the caller fails alongside the dependency. A breaker trades a few fast, deliberate rejections for keeping the caller healthy.
How does it work?
The breaker is a small state machine that sits in front of the call. Each call goes through the breaker, which records the outcome as success, failure, or (in Resilience4j) slow in a sliding window of recent calls. After each recorded outcome, the breaker recomputes the failure rate over that window and compares it with the configured threshold. While the circuit is open, the breaker does not invoke the wrapped function at all. Resilience4j throws CallNotPermittedException. Opossum rejects the returned promise and runs the registered fallback if one exists. The state and counters live in memory inside each process, so every instance of a service decides on its own.
The breaker has three states:
- Closed: Calls pass through and outcomes are recorded. It trips to open when the error rate crosses a threshold. In Resilience4j that threshold is
failureRateThreshold. In Opossum it iserrorThresholdPercentage. The breaker only evaluates the rate after a minimum sample:minimumNumberOfCallsin Resilience4j,volumeThresholdin Opossum. Without that floor, two failures out of three calls would open the circuit. - Open: Calls are rejected immediately, or routed to a fallback, until a wait period ends. That period is
waitDurationInOpenStatein Resilience4j andresetTimeoutin Opossum. - Half-open: A limited number of trial calls probe the dependency. Resilience4j allows
permittedNumberOfCallsInHalfOpenStatetrials and closes the circuit if their failure rate stays below the threshold. Opossum sends a single trial call. Success closes the circuit, and failure reopens it.
failureRateThreshold + minimumNumberOfCalls
errorThresholdPercentage + volumeThreshold
+--------+ ------------------------------------> +------+
| CLOSED | | OPEN |
+--------+ <---+ +--> +------+
| trials succeed | |
| trials fail | | waitDurationInOpenState
| | | resetTimeout
| +-----------+ | |
+----- | HALF-OPEN | --------+ |
+-----------+ <---------------+
permittedNumberOfCallsInHalfOpenState (Opossum: one trial call)
Each tuning parameter controls exactly one transition. Treat them as separate decisions rather than one combined "sensitivity" setting.
R4j: failureRateThreshold, after minimumNumberOfCalls
Opossum: errorThresholdPercentage, after volumeThreshold
+--------+ ---------------------> +--------+
| CLOSED | | OPEN |
+--------+ +--------+
^ ^ |
| | | R4j: waitDurationInOpenState
| trial(s) fail | | Opossum: resetTimeout
| | v
| +-------------+
+---------------------------| HALF-OPEN |
trials pass +-------------+
R4j: failure rate of permittedNumberOfCallsInHalfOpenState
trial calls stays below failureRateThreshold
Opossum: the single trial call succeeds
Circuit breaker state transitions, each labeled with the Resilience4j (R4j) and Opossum parameters that control it.
Who should tune circuit breakers, and when is a breaker the right tool?
Tuning belongs to the team that owns the caller's latency budget and pager, which is usually the service making the call. Platform engineers typically set shared defaults. Involve the dependency's owners because they know how it fails, but the breaker exists to protect the caller.
When should you use a circuit breaker?
Breakers pay off on synchronous calls to remote dependencies that can fail or slow down. A stalled downstream holds threads, sockets or pending promises until the caller degrades too. Fan-out services gain the most, because one slow backend out of ten can stall every request. Shared connection pools are the other classic case, where one bad dependency can exhaust connections the healthy paths need.
A breaker is also a good fit when you have a useful degraded response to serve while the dependency is down, such as a labelled cached value, a partial page, or an explicit 503. It also helps when the dependency benefits from being left alone to recover. Failing fast takes load off a struggling service, while timeouts and retries add load to it.
When should you avoid a circuit breaker?
Skip breakers in these cases:
- In-process calls. They have no independent failure to isolate.
- Very low-traffic endpoints. The minimum call count may never be reached within the window, so the breaker either never trips or trips on noise.
- Idempotent batch jobs. A queue with redelivery handles these better.
- Calls where a timeout plus one bounded retry already keeps the caller healthy. A breaker only adds state you have to tune.
Also avoid a breaker when no team will own its configuration and alerts. An untuned breaker that trips on noise, or never trips at all, is worse than none, because it gives false confidence.
What are the alternatives?
The mechanisms below solve neighbouring problems. Most production setups combine several of them rather than choosing just one. A timeout should always be present, because a breaker can only count failures it actually observes. For low-traffic dependencies, a timeout with a single bounded retry is often enough. For asynchronous work, a durable queue with redelivery and backoff replaces the breaker. For isolating resource pools, a bulkhead (Resilience4j ships one as a separate module) limits the damage without tracking failure rates.
| Mechanism | Handles | Does not handle |
|---|---|---|
| Timeout | Bounds the wait on a single call | Repeated calls to a dead dependency |
| Retry with backoff | Brief, transient failures | Sustained outages, where it multiplies load |
| Bulkhead | Isolates the resource pool for each dependency | Reducing the number of calls to a failing dependency |
| Rate limiting | Caps outbound or inbound request rate | Reacting to errors or latency |
| Load shedding | Lets a saturated server drop excess work | Protecting the caller |
| Mesh outlier detection | Ejects individual bad hosts | Whole-service outages and per-operation failures |
How do you choose a failure rate threshold and minimum call count?
Start from the dependency's normal error rate, measured over days rather than minutes. If a payment API normally fails 0.5–1% of calls, a threshold of 5% would trip on routine noise during a bad hour. A threshold of 25–50% means "this dependency is broken, not flaky." Set an upper limit by working backward from the caller. Estimate how long requests can pile up on timeouts before thread pools, connection pools, or the event loop saturate, and choose a threshold low enough to trip before then.
The minimum call count prevents a statistically meaningless sample from opening the breaker. Two failures out of three calls is 67%, but it tells you very little. Resilience4j's minimumNumberOfCalls and Opossum's volumeThreshold both suppress tripping until enough calls are recorded in the window. Size the value so that a short burst of failures cannot reach the threshold alone.
The trade-off is that a high minimum can hide a total outage. If a dependency receives four calls per window and the minimum is twenty, the breaker never opens, even at 100% failure. For low-traffic dependencies, use a longer window rather than a lower minimum.
Count only errors that indicate the dependency is unhealthy. Validation errors and most 4xx responses are caller mistakes, so excluding them keeps bad input from opening the breaker.
| Concern | Resilience4j | Opossum |
|---|---|---|
| Trip threshold | failureRateThreshold |
errorThresholdPercentage |
| Minimum sample | minimumNumberOfCalls |
volumeThreshold |
| Failure classification | recordExceptions / ignoreExceptions |
errorFilter (return true to not count) |
// ---- Resilience4j (Java) ----
CircuitBreakerConfig paymentCb = CircuitBreakerConfig.custom()
// Baseline error rate is ~0.5-1%; 50% means "broken", not "flaky"
.failureRateThreshold(50)
// 60-second time-based window, matching Opossum's rolling window below
.slidingWindowType(CircuitBreakerConfig.SlidingWindowType.TIME_BASED)
.slidingWindowSize(60)
// A burst of a few failures cannot open the breaker on its own
.minimumNumberOfCalls(20)
// Only infrastructure-level errors indicate an unhealthy dependency
.recordExceptions(IOException.class, TimeoutException.class)
// Caller mistakes (bad input, 4xx) never count toward the failure rate
.ignoreExceptions(ValidationException.class, ClientErrorException.class)
.build();
// ---- Opossum (Node.js) ----
const CircuitBreaker = require('opossum');
const paymentBreaker = new CircuitBreaker(callPaymentApi, {
timeout: 3000, // slow calls count as failures
errorThresholdPercentage: 50, // same trip point as Resilience4j
rollingCountTimeout: 60000, // 60s rolling window...
rollingCountBuckets: 6, // ...split into 10s buckets
volumeThreshold: 20, // minimum requests in the window before tripping
resetTimeout: 30000, // wait before trying half-open
// Return true to NOT count the error as a failure
errorFilter: (err) => {
if (err.name === 'ValidationError') return true;
const status = err.statusCode;
// 408 and 429 signal timeout or overload, so keep counting them
if (status === 408 || status === 429) return false;
return status >= 400 && status < 500;
},
});
Equivalent Resilience4j and Opossum configurations for a payment dependency. Both use a 50% failure threshold, a 20-call minimum over a 60-second window, and failure classification that ignores validation errors and client-side 4xx responses.
How do sliding windows and slow-call settings differ between Resilience4j and Opossum?
Resilience4j offers two window types. A count-based window evaluates the last N calls, so it reacts at a speed set by traffic volume. A time-based window aggregates the last N seconds, so it reacts at a fixed pace regardless of throughput. Opossum only uses time: rollingCountTimeout sets the window length in milliseconds and rollingCountBuckets splits it into slices that expire one at a time.
Window size trades reaction speed against flapping. A short window opens the breaker within seconds of a real outage, but it can also trip on a five-second GC pause or deploy blip that would have resolved on its own. A longer window absorbs those brief spikes but keeps sending traffic to a dead dependency for longer.
| Concern | Resilience4j | Opossum |
|---|---|---|
| Window type | slidingWindowType (COUNT_BASED / TIME_BASED) |
Time-based only |
| Window length | slidingWindowSize (calls or seconds) |
rollingCountTimeout (ms) |
| Window granularity | Internal | rollingCountBuckets |
| Slow-call detection | slowCallDurationThreshold + slowCallRateThreshold |
timeout (rejects the call, counts as failure) |
The slow-call models differ. Resilience4j lets a slow call finish, then counts it toward a separate slow-call rate. Opossum's timeout rejects the promise and records a failure.
Set either duration from observed latency rather than intuition. A threshold slightly above p99 flags real degradation without penalizing normal tail latency. For example, if p99 is 650 ms, a 30-second window with an 800 ms threshold is a reasonable start. In Resilience4j, use TIME_BASED, size 30, and a 50% slow-call rate. In Opossum, use rollingCountTimeout 30000, 10 buckets, and timeout 800.
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig.SlidingWindowType;
import java.time.Duration;
public class InventoryBreaker {
// Assumed input: observed p99 for the inventory service is ~650 ms.
static CircuitBreaker build() {
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
// Aggregate the last 30 seconds so reaction speed does not depend on traffic volume.
// Opossum equivalent: rollingCountTimeout: 30000, rollingCountBuckets: 10
.slidingWindowType(SlidingWindowType.TIME_BASED)
.slidingWindowSize(30)
// Do not evaluate rates until the window holds enough calls to be meaningful.
.minimumNumberOfCalls(20)
// Open on errors once half the calls in the window fail.
.failureRateThreshold(50)
// A call is "slow" just above p99, so normal tail latency is not penalized.
// The call still completes; it is only counted toward the slow-call rate.
// Opossum equivalent: timeout: 800 (rejects the promise, counts as a failure)
.slowCallDurationThreshold(Duration.ofMillis(800))
.slowCallRateThreshold(50)
// How long to stay open before letting trial calls through.
// Opossum equivalent: resetTimeout: 30000
.waitDurationInOpenState(Duration.ofSeconds(30))
.build();
return CircuitBreaker.of("inventory-service", config);
}
}
A Resilience4j time-based breaker using a 30-second window and an 800 ms slow-call threshold just above an assumed 650 ms p99, with comments noting the matching Opossum options.
How long should a breaker stay open, and how many half-open trial calls are enough?
Set the open duration from how long the dependency usually takes to recover, not from a round number. If a database failover typically completes in 20–40 seconds, a 5-second waitDurationInOpenState (or Opossum resetTimeout) sends probes into a dependency that cannot answer yet, and each failed probe restarts the wait. A 5-minute wait leaves you serving fallbacks for minutes after the dependency has recovered. A reasonable starting point is the low end of the observed recovery time, adjusted later from incident data.
Half-open sizing works differently in the two libraries. Resilience4j admits permittedNumberOfCallsInHalfOpenState trial calls and decides using the failure rate across them, so 5–10 calls avoids closing on one lucky success. Opossum lets a single request through after resetTimeout, and that one result decides whether the circuit closes or reopens.
Breaker state is per instance. Every replica opens, waits, and probes independently, and replicas that tripped during the same incident tend to probe at about the same time. With 40 replicas and 10 permitted calls each, a recovering service receives 400 probe calls at roughly the same moment. With Opossum it receives 40. Size the trial count with the replica count in mind.
Retries interact with the breaker. When retries sit outside the breaker, each attempt counts as a separate call, so the breaker trips sooner than the threshold suggests. Retrying a rejection from an open breaker only adds load and latency. Order the decorators as retry → breaker → timeout. Put the timeout innermost so slow calls are recorded as failures, and configure retries to skip Resilience4j's CallNotPermittedException (or Opossum's open-circuit rejection).
// Order: retry -> breaker -> timeout (timeout innermost, retry outermost)
CircuitBreaker breaker = CircuitBreaker.of("inventory", CircuitBreakerConfig.custom()
.failureRateThreshold(50)
.slidingWindowSize(20)
.minimumNumberOfCalls(10)
// Low end of observed recovery time (failover takes ~20-40s), not a round number
.waitDurationInOpenState(Duration.ofSeconds(20))
// Several trials so one lucky success can't close it; x replica count = probe burst
.permittedNumberOfCallsInHalfOpenState(5)
.build());
Retry retry = Retry.of("inventory", RetryConfig.custom()
.maxAttempts(3)
.waitDuration(Duration.ofMillis(500))
// Retrying an open-circuit rejection only adds latency; fail fast instead
.ignoreExceptions(CallNotPermittedException.class)
.build());
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder(URI.create(""))
// Innermost timeout: a slow call throws, so the breaker records it as a failure
.timeout(Duration.ofSeconds(2))
.GET()
.build();
Supplier<String> call = () -> {
try {
return client.send(request, HttpResponse.BodyHandlers.ofString()).body();
} catch (IOException e) { // includes HttpTimeoutException
throw new UncheckedIOException(e);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IllegalStateException(e);
}
};
// Wrap inside-out: breaker around the timed call, retry around the breaker
Supplier<String> guarded = Retry.decorateSupplier(retry,
CircuitBreaker.decorateSupplier(breaker, call));
A Resilience4j setup that sizes the open wait and half-open trial count from recovery time, puts the timeout innermost and the retry outermost, and skips retries when the breaker rejects a call with CallNotPermittedException.
Worked example: converting traffic and error data into breaker settings
Illustrative estimate. Assume each instance sends 50 requests per second to an inventory service. The baseline error rate is 1%, p99 latency is 400 ms, incidents typically recover in about 30 s, and there are 10 replicas.
- Failure threshold: 50%. That is fifty times the baseline, so normal noise never trips it, while a real outage crosses it within seconds.
- Window: 10 s, time-based. At 50 rps this holds about 500 calls, enough to smooth out bursts while still reacting quickly.
- Minimum calls: 20. With a 1% baseline, 10 failures among 20 calls is practically impossible unless something is actually broken.
- Timeout / slow call: 1 s. This is 2.5× p99, which leaves headroom for GC pauses but stops threads from piling up.
- Open wait: 15 s. Ten replicas probe independently, so the dependency sees roughly one probe burst every 1.5 s while it recovers. A shorter wait would add load during recovery.
| Setting | Resilience4j | Opossum |
|---|---|---|
| Failure threshold | failureRateThreshold 50 |
errorThresholdPercentage 50 |
| Minimum volume | minimumNumberOfCalls 20 |
volumeThreshold 20 |
| Window | time-based, slidingWindowSize 10 (seconds) |
rollingCountTimeout 10000, rollingCountBuckets 10 |
| Slow call / timeout | slowCallDurationThreshold 1 s |
timeout 1000 |
| Open wait | waitDurationInOpenState 15 s |
resetTimeout 15000 |
| Trial calls | permittedNumberOfCallsInHalfOpenState 5 |
single trial request |
At one tenth of the traffic (5 rps), the 10 s window holds only about 50 calls, so a short blip can trip the breaker. Widen the window to 30 s and keep 20 as the minimum. Reaching 20 calls now takes about 4 s instead of 0.4 s, which is the cost of avoiding false trips.
Granularity. One breaker per dependency is the simplest option. Its weakness is that a single broken endpoint can open the circuit for healthy ones. Breakers per endpoint isolate those failures, but they split the traffic: each breaker sees fewer calls, so its minimum-call setting delays detection. They also multiply config entries, dashboards and metric series. Split only the endpoints whose failure modes or latency profiles actually differ.
How do you design fallbacks and monitor breakers without hiding outages?
A fallback should degrade honestly. Good options are a cached or partial response explicitly marked as degraded, such as a response header or a degraded field the caller can check, or a fast, explicit error like a 503 with a retry hint. The dangerous fallback silently returns stale prices, an empty list or a default value. Callers treat that as truth, so the outage turns into data corruption that surfaces days later. If you cannot label a fallback as degraded, fail fast instead.
Make state observable
Error-rate alerts lag and blur together. Breaker state transitions tell you exactly which dependency was isolated and when. Here is how to capture them in each library:
- Resilience4j: subscribe to the circuit breaker's event publisher, especially
onStateTransition, and export its metrics through the Micrometer integration. These include state, failure rate, slow-call rate and not-permitted calls. - Opossum: listen for the
open,halfOpenandcloseevents, plusfallback,rejectandtimeout. Forward them to your metrics pipeline with a label for the dependency name.
Alert when a breaker enters open. Alert separately when it stays open longer than expected or flaps between states.
Cost and upkeep
Per-dependency tuning, dashboards and failure testing take real engineering time up front. That time pays back during incidents, because responders see which dependency was cut off instead of digging through cascading timeouts.
Thresholds also drift as traffic grows and dependencies change. Review them against production metrics on a regular schedule, and rerun fault-injection tests to confirm breakers still open and recover as intended.
How can circuit breakers save engineering time?
The savings come from incidents that stay small and are quicker to diagnose, not from the breaker itself:
- Contained blast radius. When a dependency fails, an open breaker keeps the caller's threads, connections and event loop free. The incident stays "inventory is down, product pages show degraded stock data" instead of "the whole storefront is down." Fewer services are affected, so fewer teams are paged and fewer components need checking afterwards.
- Faster triage. A state-transition event names the dependency and timestamps when it was isolated. Responders start from that signal instead of working backwards from timeouts spread across many services and log streams.
- Automatic recovery. The half-open probe closes the circuit once the dependency recovers. Nobody has to restart callers, flip a feature flag, or drain stuck requests by hand after the downstream fix lands.
- Less hand-rolled resilience code. Resilience4j and Opossum provide the windowing, state machine, events and metrics. Teams configure them instead of writing and maintaining ad hoc error counters and retry guards in each client.
- Reviewable, shared configuration. Breaker settings live in code or config next to the call site, so they go through normal review. Platform-level defaults mean most teams only tune the few dependencies whose traffic or failure profile is unusual.
The time saved depends on how often your dependencies fail and how far failures currently cascade, so judge the benefit from your own incident history rather than expecting a fixed payoff. A breaker that is never reviewed, alerts on noise, or hides outages behind silent fallbacks costs time instead of saving it.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to circuit breaker tuning, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.