Node.js Circuit Breaker with Opossum: Tuning Thresholds and Fallbacks

How the circuit breaker pattern maps onto Opossum's states

A circuit breaker sits between your code and a dependency it calls synchronously, such as another service or a third-party API. When that dependency starts failing or hanging, the breaker stops sending it requests for a while and fails fast instead. Without one, each request holds a socket, memory, and often a slot in a connection pool while it waits for a timeout. Under load, those waiting requests pile up until the caller is also unhealthy. Once that happens, the original outage has spread.

Opossum models this with three states:

• Closed. Calls pass through to the wrapped function. Opossum records successes, failures, and timeouts in a rolling window of time-based buckets. A call that exceeds timeout counts as a failure even if the dependency would eventually have answered. If the failure percentage in the window crosses errorThresholdPercentage, the circuit opens. It only opens once the window holds at least volumeThreshold requests, so a handful of calls cannot trip it.
• Open. Calls are rejected immediately without touching the dependency, or are routed to a fallback if you registered one. This lasts for resetTimeout milliseconds.
• Half-open. After resetTimeout elapses, Opossum lets a trial call through. If it succeeds, the circuit closes and normal traffic resumes. If it fails, the circuit reopens and the reset timer starts again.

The beneficiaries are backend and platform engineers whose request paths depend on synchronous calls they don't control. Keep the scope clear: the breaker protects the caller. It may reduce pressure on a struggling dependency, but that is a side effect, not a guarantee. Other callers without breakers will keep hammering it.

Copy +----------------+        +-------------------+        +----------------+
 | Caller         |  call  | Opossum breaker   | closed | Dependency     |
 | (your service) |------->| (state machine    |------->| (service or    |
 |                |<-------|  shown below)     |<-------|  3rd-party API)|
 +----------------+ result +-------------------+ reply  +----------------+
                    (or fast failure / fallback while open)

  +----------+  failures + timeouts in rolling window  +----------+
  |  CLOSED  |  >= errorThresholdPercentage, with at   |   OPEN   |
  | calls go |  least volumeThreshold requests         | reject / |
  | through  |---------------------------------------->| fallback |
  +----------+                                         +----------+
       ^                                                 |     ^
       |                              after resetTimeout |     |
       |                              elapses            v     |
       |                                         +-----------+ |
       |      trial call succeeds                | HALF-OPEN | |
       +-----------------------------------------| one trial |-+
                                                 |   call    | trial call
                                                 +-----------+ fails; timer
                                                               restarts

Opossum sits between the caller and the dependency: closed passes calls through, a failure rate above errorThresholdPercentage (once volumeThreshold is met) opens the circuit, resetTimeout moves it to half-open, and the trial call either closes it or reopens it.

Wrapping a dependency call with new CircuitBreaker and breaker.fire

The integration point is small. Take the async function that calls the dependency, such as getInventory(sku) built on fetch or your HTTP client, and pass it to new CircuitBreaker(getInventory, options). Callers then use breaker.fire(sku) instead of getInventory(sku). The arguments pass through unchanged, and fire returns a promise. That promise resolves with the function's result, rejects with its error, or rejects immediately when the breaker is open. Any direct call that bypasses fire also bypasses the breaker's statistics, so route every call through it.

The wrapped function must enforce its own deadline. When Opossum's timeout fires, it rejects the promise it returned to you, but it does not cancel the work underneath. The socket stays open and the request keeps running, and slow calls pile up behind a breaker that thinks it has moved on. Create an AbortSignal, either with AbortSignal.timeout(ms) or with an AbortController you abort yourself, and pass it to the HTTP call. Set the abort deadline at or slightly below the breaker's timeout so the request is actually torn down.

Create one breaker per downstream dependency, or per dependency and endpoint, and keep them in a small registry keyed by name, such as a Map. A single global breaker lets one failing service cut off every healthy one.

Granularity is a judgment call:

• Too coarse: If inventory-service has one breaker, a broken /reservations endpoint also blocks a healthy /stock.
• Too fine: If every endpoint-and-parameter combination gets its own breaker, low-traffic keys may never reach volumeThreshold. Those breakers never trip, so they provide no protection.

Start with one breaker per dependency. Split out an endpoint only when it has its own failure modes and enough traffic to produce a meaningful error rate.

Copyconst CircuitBreaker = require('opossum');

const BREAKER_TIMEOUT_MS = 3000;
// Abort a little before the breaker gives up, so the socket is really torn down
// instead of lingering after Opossum has already rejected the caller's promise.
const ABORT_MS = 2800;

async function getInventory(sku) {
  const url = `)}`;
  const res = await fetch(url, { signal: AbortSignal.timeout(ABORT_MS) });
  if (!res.ok) throw new Error(`inventory-service responded ${res.status}`);
  return res.json();
}

const defaults = {
  timeout: BREAKER_TIMEOUT_MS,
  errorThresholdPercentage: 50,
  resetTimeout: 10000,
  volumeThreshold: 10, // finer-grained keys need enough traffic to ever reach this
};

// One breaker per dependency (or dependency + endpoint), never one global breaker.
const breakers = new Map();

function breakerFor(name, fn, options = {}) {
  if (!breakers.has(name)) {
    breakers.set(name, new CircuitBreaker(fn, { ...defaults, ...options, name }));
  }
  return breakers.get(name);
}

// Callers use fire() so every call counts toward this breaker's statistics.
async function lookupStock(sku) {
  return breakerFor('inventory-service', getInventory).fire(sku);
}

module.exports = { lookupStock, breakerFor };

A per-dependency Opossum breaker registry where the wrapped fetch call aborts itself via AbortSignal.timeout just under the breaker's timeout, and callers go through breaker.fire().

Deriving timeout, errorThresholdPercentage, rolling window and volumeThreshold from real data

Timeout from tail latency

Set timeout from the dependency's measured latency distribution, not from a round number. Take p99 from your own metrics over a representative period, add headroom for normal variance, and keep the result below the caller's own deadline. If p99 is 400 ms, a 600 ms timeout tolerates slow-but-healthy calls and still fails fast when the dependency hangs.

Error rate, window and volume

errorThresholdPercentage is evaluated over a rolling window of rollingCountTimeout milliseconds, split into rollingCountBuckets buckets. Old buckets drop out as time passes, so a longer window remembers earlier successes and reacts more slowly. volumeThreshold is the minimum number of requests in the window before the breaker can trip. It keeps one failure at startup or overnight from reading as a 100% error rate.

Illustrative estimate (assumed numbers, not a benchmark): 20 req/s, a window full of successes, then the dependency starts failing 60% of calls, with a 50% threshold. Time to trip ≈ W × T / E = window × threshold ÷ new error rate.

Aggressive settings turn a two-second blip into a full feature outage. Conservative settings keep callers waiting on a dead dependency while threads and sockets pile up, and the failure cascades.

Use errorFilter to return true for expected 4xx and 404 responses, so those do not count as failures. Only 5xx responses, timeouts and connection errors should count toward tripping the breaker.

Copyconst CircuitBreaker = require('opossum');

// Action: call the dependency and attach the HTTP status to failures.
async function getProfile(userId) {
  const res = await fetch(`);
  if (!res.ok) {
    const err = new Error(`profile-service responded ${res.status}`);
    err.statusCode = res.status;
    throw err;
  }
  return res.json();
}

const breaker = new CircuitBreaker(getProfile, {
  // Measured p99 is 400 ms. Add headroom and stay below the caller's deadline.
  timeout: 600,
  // Open when at least 50% of calls in the window fail.
  errorThresholdPercentage: 50,
  // 10 s window in ten 1 s buckets. Old buckets age out every second.
  rollingCountTimeout: 10000,
  rollingCountBuckets: 10,
  // Need at least 20 calls in the window before tripping, so one overnight
  // failure at 0.2 req/s cannot open the circuit.
  volumeThreshold: 20,
  // After opening, wait 30 s before letting a trial request through (half-open).
  resetTimeout: 30000,
  // Return true to NOT count the error. Expected 4xx (including 404) are
  // client or data issues, not a sick dependency. 5xx, opossum timeouts and
  // connection errors (no statusCode) still count as failures.
  errorFilter: (err) =>
    typeof err.statusCode === 'number' && err.statusCode >= 400 && err.statusCode < 500,
});

// Degrade the feature instead of failing the whole page while open.
breaker.fallback((userId) => ({ id: userId, displayName: null, degraded: true }));

breaker.on('open', () => console.warn('profile-service breaker OPEN'));
breaker.on('halfOpen', () => console.info('profile-service breaker HALF-OPEN'));
breaker.on('close', () => console.info('profile-service breaker CLOSED'));

module.exports = (userId) => breaker.fire(userId);

An Opossum breaker with timeout set from measured p99 plus headroom, a 10-second rolling window in 1-second buckets, and a volumeThreshold that stops low-traffic trips. Its errorFilter ignores 4xx responses so only 5xx responses, timeouts and connection errors count toward opening the circuit.

Designing fallbacks as product decisions

Opossum invokes the function registered with breaker.fallback(fn) whenever a call fails, times out, or is rejected because the circuit is open. The function receives the same arguments passed to fire. Its return value becomes the caller's result, so choosing what it returns is a product decision, not just a technical one. Two rules apply regardless of strategy. The fallback must be cheap, because it runs on every rejected call during an outage. It must also never call the failing dependency, or it recreates the load the breaker exists to shed.

Queued writes need the most care. Replays can arrive out of order or twice, so the downstream handler must be idempotent. You also have to decide what happens when the queue itself fills up.

Stale data has a quieter risk. Success rates and latency look healthy while users see hour-old prices. Subscribe to the breaker's fallback event and alert on its rate separately from errors, so a masked outage still pages someone.

Copyconst CircuitBreaker = require('opossum');

// Last successful price per product, kept in process memory.
const lastGood = new Map(); // productId -> { price, fetchedAt }
const MAX_STALE_MS = 15 * 60 * 1000; // product decision: older prices are worse than none

async function fetchPrice(productId) {
  const res = await fetch(`);
  if (!res.ok) throw new Error(`pricing returned ${res.status}`);
  const body = await res.json();
  lastGood.set(productId, { price: body.price, fetchedAt: Date.now() });
  return { productId, price: body.price, stale: false };
}

const breaker = new CircuitBreaker(fetchPrice, {
  timeout: 2000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
});

// Receives the same args as fire(). Reads memory only and never calls pricing.
breaker.fallback((productId) => {
  const cached = lastGood.get(productId);
  if (cached && Date.now() - cached.fetchedAt < MAX_STALE_MS) {
    // Flag staleness so the UI can label it instead of looking normal.
    return { productId, price: cached.price, stale: true, asOf: cached.fetchedAt };
  }
  // No acceptable cached value: fail fast with a clear message.
  throw new Error('Pricing is temporarily unavailable. Please try again shortly.');
});

// Count fallback activations separately from errors so masked outages still alert.
let fallbackCount = 0;
breaker.on('fallback', () => { fallbackCount += 1; });
setInterval(() => {
  console.log(JSON.stringify({ metric: 'pricing.fallback.count', value: fallbackCount }));
  fallbackCount = 0; // ship to your metrics pipeline and alert on its rate
}, 60000);

module.exports = (productId) => breaker.fire(productId);

An Opossum fallback that serves recently cached prices marked as stale and fails fast once the cache is too old. The fallback never calls the pricing service, and a separate counter on the 'fallback' event lets you alert on fallback rate independently of errors.

Choosing resetTimeout and handling the half-open state

When resetTimeout elapses, Opossum moves the breaker from open to half-open and lets the next call through as a trial. Calls that arrive while that trial is in flight are rejected as if the circuit were still open, so they get the fallback instead of piling onto a dependency that may still be down. If the trial succeeds, the breaker closes. If it fails or times out, the breaker reopens and the reset timer starts again.

That reopen cycle is how a short resetTimeout causes flapping. Each failed trial costs one real request a full timeout. Dashboards show the breaker toggling between half-open and open, and the dependency receives a steady trickle of probes while it is trying to recover.

Illustrative estimate (assumed numbers, not a benchmark): failed trials ≈ recovery time ÷ (resetTimeout + timeout). With an assumed 60 s recovery, a 5 s resetTimeout and a 3 s timeout, that is 60 ÷ 8 ≈ 7 failed trials. Raising resetTimeout to 30 s cuts that to about 1.

To size resetTimeout, look at how long the dependency took to recover in past incidents, such as restarts, failovers and deploys. Set it near the typical recovery time rather than the best case.

Retries need one home. Retries inside the wrapped function inflate latency toward timeout and hide failures from the breaker's error count. Retries outside the breaker turn each rejected call into several, and one of those retries can land on the half-open trial. Choose one layer, cap the number of attempts, add jittered backoff, and treat breaker rejections as non-retryable.

Observing and testing breaker behavior

Opossum emits an event for every transition and outcome, and these events are the raw material for tuning. Attach listeners for open, halfOpen and close and write each one as a structured log line that includes the breaker name and the pod identity. A state change is rare, so every one deserves a record. fallback, timeout, reject and failure fire far more often, so increment counters for them instead of logging each one. On a fixed interval, export breaker.stats as gauges. The ratio of timeouts to failures shows whether timeout is too tight. A high reject count paired with a fast recovery suggests resetTimeout is too long.

Breaker state lives in process memory. Every pod trips, probes and recovers on its own, so ten replicas can be in ten different states at once. Label metrics by instance and build dashboards that show the spread across instances. Alert on the share of instances that are open, not on a single average, because an average hides one stuck pod.

Deterministic tests

Don't depend on real network faults in tests. Inject the dependency and replace it with a stub that rejects or returns a promise that never settles. Configure the breaker with a short timeout, a low volumeThreshold and a short resetTimeout. Then fire enough calls to cross the threshold and assert that breaker.opened is true and that callers get the fallback output. Wait out the reset period, let the stub succeed, and assert that the breaker passes through half-open and closes.

These tests encode your thresholds as expected behavior. If someone loosens errorThresholdPercentage or removes a fallback, CI fails on the pull request instead of the change surfacing during an incident.

When to avoid it, alternatives, and impact on cost and engineering time

A breaker earns its keep only when enough traffic flows through it to produce a meaningful error rate. For a call made a few times an hour, volumeThreshold is rarely reached, so a timeout plus an alert tells you the same thing with less machinery. Fire-and-forget work such as emails, webhooks, and audit writes belongs on a queue. A queue keeps failed work and retries it later, while a breaker just drops calls or sends them to a fallback. Sharing breaker state across a fleet is tempting, but every instance already sees the same failing dependency and trips within its own window. Coordinating state adds a store that can itself fail, and the gain is usually a few seconds.

Illustrative estimate (assumed numbers, not a benchmark). Break-even incidents per year = setup hours ÷ hours saved per incident. Assume 24 hours for tuning, fallback design, tests, and alerts, and 6 engineer-hours saved per contained incident through less cascading failure and a clearly named failing dependency. That gives 24 ÷ 6 = 4 incidents a year to pay back the work.

A breaker does not fix the root cause. It contains the failure and points at the failing dependency, so someone still has to fix that dependency.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nodejs circuit breaker opossum, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.