Circuit Breaker Pattern, Retries and Timeouts as One Failure Budget

What partial failure looks like and why slow is worse than down

In a distributed system, partial failure is what normally happens, not an edge case. One replica's disk fills up, a database fails over, a downstream service stalls in a garbage-collection pause, a load balancer drops some connections. The rest of the system keeps running and keeps sending traffic to the part that is struggling.

A dependency that is fully down is usually the easier case. Connections get refused, errors come back within milliseconds, and the caller frees its resources right away. A slow dependency is worse. Each request to it holds a thread or async task, a pooled connection, a socket and its buffers for as long as the caller is willing to wait. If calls normally take 50 ms and now take 30 seconds, every in-flight call holds those resources roughly 600 times longer. Pools run out, queues grow, memory climbs, and the caller stops answering its own clients. One slow service can then take healthy services down with it.

Three patterns limit this. Treat them as one resilience budget applied in a fixed order: timeouts first, to cap how long any call can hold resources; retries with backoff second, to recover from short failures within that cap; and the circuit breaker pattern last, as a backstop that stops calls to a dependency that keeps failing.

These patterns are a cost of choosing a microservice architecture. Splitting a system into services turns in-process function calls into network calls, and network calls can hang. Any team whose services call other services over the network needs these patterns. They matter most on high-traffic synchronous request paths and when a call goes through several services in a row, because one slow hop holds up every caller above it.

Copy +---------+
 | Clients |
 +---------+
      |
      v
 +-----------------+     +-----------------+     +-----------------+
 |   Service A     |---->|   Service B     |---->|   Service C     |
 |  threads: FULL  |     |  conn pool: FULL|     |  SLOW: 30 s     |
 |  queue growing  |     |  memory rising  |     |  (normal 50 ms) |
 +-----------------+     +-----------------+     +-----------------+
          ^                       ^                       |
          +------- waits ---------+------- waits ---------+

  One slow hop holds threads, connections and memory at every caller
  above it: 50 ms -> 30 s means each call holds resources ~600x longer.

 Inside each caller, every outbound call passes through one budget:

 +-----------+   +--------------------+   +-------------------+
 | 1 TIMEOUT |-->| 2 RETRY + BACKOFF  |-->| 3 CIRCUIT BREAKER |--> dependency
 | caps how  |   | recovers short     |   | backstop: stops   |
 | long a    |   | failures inside    |   | calls to a dep    |
 | call holds|   | the timeout cap    |   | that keeps failing|
 +-----------+   +--------------------+   +-------------------+

A slow downstream service exhausts resources at every caller above it, so each outbound call goes through a timeout, then retries with backoff, then a circuit breaker, in that order.

Timeouts first: connect, read and propagated deadlines

Without an explicit timeout, a caller waits as long as its client library allows, and some defaults are very long or unbounded. While it waits, it holds a thread, a pooled connection and memory for a request the user may already have abandoned.

Connect vs. read timeouts

A connect timeout limits how long the TCP (and TLS) handshake can take. A failure here usually means the host is unreachable or its accept queue is full, so keep it short. A read or request timeout limits how long the caller waits for a response once connected, so it has to cover the real work the dependency does.

Derive values from observed latency

Set timeouts from the dependency's latency distribution under normal load, not from library defaults.

Illustrative estimate (assumed numbers, not a benchmark): timeout = p99 latency × headroom → 180 ms × 1.5 = 270 ms. requests held while dependency hangs = request rate × timeout (Little's law) → 400 req/s × 0.27 s ≈ 108. With a 30 s timeout, the same rate would hold 12,000 requests.

Propagate the deadline

Set one deadline at the edge and pass the remaining time to each hop. Each service sets its own timeout to min(local timeout, remaining budget − safety margin). If the budget is already spent, it fails immediately instead of starting work nobody will receive. gRPC propagates deadlines natively. Over HTTP, use a header your services agree on.

The trade-off

If timeouts are too tight, normal spikes like GC pauses and cold caches become false failures and trigger retries that add load. If they are too loose, a slow dependency drains your pools. Dependency latency also drifts over time, so re-derive values regularly and alert when p99 gets close to the configured timeout.

Copyimport time
import requests

CONNECT_TIMEOUT_S = 0.1   # TCP/TLS handshake only: an unreachable host should fail fast
READ_TIMEOUT_S = 0.27     # illustrative: p99 180 ms x 1.5 headroom, re-derive regularly
SAFETY_MARGIN_S = 0.02    # time reserved to handle the reply and respond upstream
BUDGET_HEADER = "X-Request-Budget-Ms"  # header name all services agree on


class DeadlineExceeded(Exception):
    pass


def deadline_from_request(headers, default_budget_s=1.0):
    # The edge sets the budget. Inner services turn the remaining milliseconds
    # into a local monotonic deadline, so clock skew between hosts doesn't matter.
    budget_ms = headers.get(BUDGET_HEADER)
    budget_s = int(budget_ms) / 1000 if budget_ms else default_budget_s
    return time.monotonic() + budget_s


def call_dependency(url, deadline):
    remaining = deadline - time.monotonic() - SAFETY_MARGIN_S
    if remaining <= 0:
        # Fail now instead of starting work that nobody will receive.
        raise DeadlineExceeded("budget spent before call")
    connect_t = min(CONNECT_TIMEOUT_S, remaining)
    read_t = min(READ_TIMEOUT_S, remaining)
    # requests takes a (connect, read) tuple. The read timeout applies to each
    # socket read, not to the whole response, so a server that trickles bytes
    # can still exceed it. Keep the deadline check above for that reason.
    return requests.get(
        url,
        timeout=(connect_t, read_t),
        headers={BUDGET_HEADER: str(int(read_t * 1000))},  # pass remaining budget on
    )

Separate connect and read timeouts in Python requests, capped by a propagated remaining-time budget so an inner call never outlives its caller.

Retries with backoff: what to retry, how often and at which layer

A retry is only safe when repeating the call cannot change the outcome. In practice, that means two kinds of call. The first is an idempotent read. The second is a write that carries an idempotency key, which the server uses to deduplicate. The error must also be plausibly transient: a connection reset, a timeout, throttling (429) or an unavailable response (503). Validation and authorization failures will fail the same way on every attempt, so retrying them only adds load.

Use exponential backoff with jitter between attempts. The delay ceiling doubles each time, for example 100 ms, then 200 ms, then 400 ms. The actual wait is a random value below that ceiling, so clients that failed together do not retry together. Set a hard cap on attempts, and never let a backoff wait run past the propagated deadline.

Illustrative estimate. Assume three layers: a gateway, a service and a data client. Each makes one call plus 3 retries, so each layer multiplies attempts by 4. One user request can then become 4 × 4 × 4 = 64 calls to the bottom dependency, and this happens when that dependency is already struggling. To avoid this, retry at one layer only, usually the direct client of the dependency, and have the other layers pass errors upward. Add a retry budget that caps retries at a fixed share of recent traffic, so retrying stops once failures become widespread.

Copyimport random
import time

RETRYABLE_STATUS = {429, 503}  # throttled or unavailable; 400/401/403 never retried

class RetryBudget:
    """Each request deposits `ratio` tokens; each retry costs one token.
    When failures are widespread, the tokens run out and retrying stops."""
    def __init__(self, ratio=0.1, max_tokens=10.0):
        self.ratio, self.max_tokens, self.tokens = ratio, max_tokens, max_tokens

    def on_request(self):
        self.tokens = min(self.max_tokens, self.tokens + self.ratio)

    def try_spend(self):
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

def call_with_retries(send, deadline, budget, max_attempts=4, base=0.1, cap=2.0):
    """`send` must be an idempotent read or a write carrying an idempotency key.
    Use this only in the direct client of the dependency, not in every layer."""
    budget.on_request()
    for attempt in range(max_attempts):  # hard cap: 1 call + 3 retries
        try:
            status = send(timeout=max(0.0, deadline - time.monotonic()))
        except (ConnectionResetError, TimeoutError):
            status = None  # transient transport failure
        if status is not None and status not in RETRYABLE_STATUS:
            return status  # success, or a failure that will repeat on every attempt
        if attempt == max_attempts - 1 or not budget.try_spend():
            break
        ceiling = min(cap, base * 2 ** attempt)  # 0.1 s, 0.2 s, 0.4 s ...
        delay = random.uniform(0, ceiling)  # full jitter spreads out clients that failed together
        if time.monotonic() + delay >= deadline:
            break  # never wait past the propagated deadline
        time.sleep(delay)
    raise RuntimeError("gave up: attempts exhausted, retry budget spent or deadline reached")

A single-layer retry wrapper. It retries only transient errors, uses capped exponential backoff with full jitter, stops at the propagated deadline, and draws on a retry budget that halts retries once failures become widespread.

How the circuit breaker pattern works

A breaker wraps calls to one dependency and keeps a record of recent outcomes. While closed, calls go through and each result is logged in a sliding window, based on either a count of calls or a time period. When the window crosses a threshold, the breaker goes open and rejects calls immediately for a cool-down period. That gives the dependency time to recover without being hit by your traffic. When the cool-down ends, the breaker goes half-open and lets a small, fixed number of trial calls through. If they succeed, it closes. If any fail, it opens again and the cool-down restarts.

The minimum call count prevents bad trips on small samples. Two failures out of three calls is 67%, but that tells you nothing. If you set the minimum too high, though, a low-traffic dependency never reaches it and the breaker never trips.

Count timeouts, connection errors and 5xx responses as failures. Also count successful calls that run past a latency threshold, because a slow dependency uses up your threads without ever returning an error. Do not count 4xx responses: they mean the caller sent a bad request, not that the dependency is unhealthy.

Where the breaker's state lives is another trade-off. A breaker in each instance needs no coordination, but every instance has to learn about the failure separately. A shared breaker reacts as one, but adds a dependency of its own. Granularity is a similar choice. One breaker per dependency is the default. If one slow endpoint or one heavy tenant could trip a breaker that blocks every other caller too, split it into breakers per endpoint or per tenant.

Copyimport time
from collections import deque

class CircuitBreaker:
    """Minimal per-instance breaker for one dependency. Not thread-safe; illustrative only."""
    def __init__(self, window=20, min_calls=10, failure_rate=0.5,
                 slow_call_s=2.0, cooldown_s=30.0, trial_calls=3):
        self.window = deque(maxlen=window)  # count-based sliding window: True = failure
        self.min_calls, self.failure_rate, self.slow_call_s = min_calls, failure_rate, slow_call_s
        self.cooldown_s, self.trial_calls = cooldown_s, trial_calls
        self.state, self.opened_at, self.trials, self.successes = "closed", 0.0, 0, 0

    def allow(self):
        if self.state == "open":
            if time.monotonic() - self.opened_at < self.cooldown_s:
                return False  # fail fast so the dependency can recover
            self.state, self.trials, self.successes = "half_open", 0, 0
        if self.state == "half_open":
            if self.trials >= self.trial_calls:
                return False  # only a fixed number of trial calls get through
            self.trials += 1
        return True

    def record(self, elapsed_s, status=None, error=None):
        client_error = status is not None and 400 <= status < 500  # caller's fault, not ill health
        failed = not client_error and (error is not None or (status or 0) >= 500
                                       or elapsed_s > self.slow_call_s)  # slow success = failure
        if self.state == "half_open":
            if failed:
                self.state, self.opened_at = "open", time.monotonic()  # restart cool-down
            else:
                self.successes += 1
                if self.successes >= self.trial_calls:
                    self.state = "closed"
                    self.window.clear()  # start fresh; old failures no longer apply
            return
        if client_error:
            return  # 4xx stays out of the window entirely
        self.window.append(failed)
        if len(self.window) >= self.min_calls and sum(self.window) / len(self.window) >= self.failure_rate:
            self.state, self.opened_at = "open", time.monotonic()

A minimal per-instance circuit breaker for one dependency. It uses a count-based failure-rate window with a minimum call count, treats slow calls, 5xx responses and errors as failures while ignoring 4xx, waits out a cool-down when open, and allows a fixed number of trial calls in half-open.

When the breaker is open: fail fast, degrade or fall back

An open breaker only decides that the call will not happen. What the caller gets instead is a product decision, and it belongs to whoever owns the user experience, not to whoever configured the breaker. You have three options.

Fail fast with a clear error. Return immediately with an explicit status, such as 503 with a Retry-After header, and an error code the client can act on. This is the honest default. It keeps the failure visible in dashboards and lets upstream callers apply their own policy.

Serve cached or degraded data. Show the last known recommendations, hide the reviews widget, or render the page without personalization. This works when a slightly old answer is still a correct answer for the user.

Return a static default. Use an empty list, a generic banner, or a feature flag set to off. It is cheap and predictable, but the default must be safe to act on.

Fallbacks have a cost. They hide real failures: if every response returns 200, alerting built on error rates stays quiet while a dependency is down for hours. Stale data can also be worse than an error. A cached price can sell below cost, cached inventory can oversell stock, and a cached authorization decision can grant access that was revoked minutes ago. For these cases, fail fast.

Implementing the patterns together: bulkheads, wrapper order and avoiding duplicate layers

Bulkheads first

Give each dependency its own connection pool or concurrency limit. Without that isolation, a single slow dependency can hold every worker thread, and calls to healthy services queue behind it. A bulkhead caps how much capacity one failure can consume. When the cap is reached, the call is rejected immediately, and that rejection is a fast, cheap signal the breaker can count.

Wrapper order decides what the breaker sees

A common composition, from outside to inside, is: overall deadline → retry → circuit breaker → bulkhead → per-attempt timeout.

• Retry outside the breaker: the breaker records every attempt. Once it opens, the remaining retries fail instantly instead of reaching the struggling service.
• Breaker outside the retry: the breaker records only the final outcome. Each logical call can send several attempts downstream before the breaker registers a single failure.

Choose the order deliberately. In most libraries, the order in which you register the wrappers is the order in which they apply.

Library or mesh

Avoid duplicate layers

Retries multiply across layers. As an illustrative estimate, three attempts each at the client, the mesh and the downstream service add up to 27 requests for one user action. Assign retries to one layer and turn them off in the others.

Related controls and when to skip them

Load shedding and rate limiting protect the server side and complement client-side breakers. The full stack is overkill for in-process calls, which have no network to fail. It is also unnecessary for offline batch jobs, where a simple retry with a checkpoint usually covers the failure modes.

Observability, fault injection and the impact on cost and engineering time

What to emit

Each pattern needs its own signal. If all you have is an aggregate error rate, you cannot tell whether a dependency is slow, refusing connections or being shielded by an open breaker. Emit these per dependency:

• Timeouts, tagged by type (connect, read, deadline exceeded). A rise in connect timeouts points to network or capacity problems. A rise in read timeouts points to a slow dependency.
• Retry attempts, counted separately from original requests. Comparing the two shows how much extra load you send downstream while it is struggling.
• Breaker state transitions (closed→open, open→half-open, half-open→closed), as events with timestamps. These are what you correlate with deploys and incidents.
• Rejected calls, from both the breaker and the bulkheads. These are requests you chose not to send, and they should drive alerts and fallback dashboards.

Test with faults, not configuration

A timeout value in a config file only tells you what someone intended. To know what actually happens, inject faults in staging or in controlled production experiments: add latency to a dependency, drop connections in the middle of a request, and return error responses. Then check that deadlines propagate, that retries stop at the budget, and that the breaker opens and recovers when it should.

Cost and payoff

The patterns are not free. Someone has to design the failure modes, build idempotency-key storage for retried writes, tune thresholds against real traffic, and review those settings as dependencies change. A single shared policy library with tested defaults pays this cost once instead of once per team. During an incident, responders can rely on consistent behaviour and the same metric names across services. That makes a failing dependency quicker to diagnose and stops one team's untuned client from spreading the outage further.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to circuit breaker pattern, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.