Cascading Readiness Failures: Keep Dependency Checks Out of Probes

How cascading readiness failures unfold

A cascading readiness failure starts when a dependency shared by every replica, such as a primary database, a cache cluster or a downstream API, slows down or stops responding. If each replica's readiness probe calls that dependency, every probe fails together, because the replicas share the same fault. Kubernetes marks each failing pod as not ready and removes it from the Service's endpoints. Once the last pod is removed, the Service has no endpoints. Requests that could have been served from memory, served with stale data or rejected cleanly now fail at the network layer.

The damage spreads to callers. If an upstream service's readiness check calls this one, its probes fail next and its pods are removed too. A slow cache in one corner of the call graph can empty Services several hops away. Backend and platform engineers who own services with shared dependencies are the people exposed. Their own code may be healthy while their service drops out of rotation.

Illustrative estimate

These numbers are assumed, not measured: probe period 5 s, timeout 2 s, failure threshold 3, success threshold 1. The dependency stalls for 30 s.

• Each pod becomes unready after three consecutive failed probes. Depending on where in the probe cycle the stall begins, that takes about 12–17 s.
• Every replica crosses that threshold within the same few seconds, so the Service is empty by about 17 s.
• Recovery needs one successful probe after the stall ends. That adds up to 5 s, plus the time for endpoint updates to propagate.

In this example, a 30 s stall leaves the Service with no endpoints for roughly 15–20 s. Callers that check this Service in their own readiness probes are then removed from rotation as well.

Copy  Clients
     |
     v
+-----------------------------+
| Service A   endpoints: 0    |  (4) A's readiness check calls B,
|  [A1 x]  [A2 x]             |      fails, and A is drained too
+-------------+---------------+
              | requests + A's readiness check
              v
+-----------------------------+
| Service B   endpoints: 0    |  (3) last B pod removed from
|  [B1 x]  [B2 x]  [B3 x]     |      endpoints; Service B is empty
+---+--------+--------+-------+
    |        |        |   readiness probe from each B pod
    v        v        v   (2) every probe fails together
+-----------------------------+
| Shared DB / cache           |  (1) dependency stalls for 30 s
|  STALLED                    |
+-----------------------------+

  x = pod marked not ready

 Illustrative estimate (assumed: period 5 s, timeout 2 s, threshold 3)
 t=0 s ------- t=12-17 s -------- t=30 s ------- t=30-35 s+
 stall         all B pods         stall ends     B ready after one
 begins        unready, B empty                  probe + propagation

A 30 s stall in a shared dependency fails every replica's readiness probe at once, empties Service B and then drains Service A, whose readiness check calls B (timeline is an illustrative estimate from assumed probe settings, not a measurement).

Who should use this approach

This guidance applies to anyone who writes or reviews probe configuration for services on Kubernetes:

• Backend engineers whose services call a database, cache, queue or another internal API. If every replica talks to the same dependency, a readiness check against it can empty the Service.
• Platform and SRE teams who maintain shared Helm charts, deployment templates or golden-path service scaffolds. A dependency check baked into a template spreads to every service built from it, so fixing it there has the widest effect.
• Teams that run deep call graphs, where one service's readiness depends on another's. These are the setups where a fault several hops away can drain Services that have nothing wrong with them.
• Owners of services with mixed criticality, such as a product page that can render without recommendations. Request-level degradation pays off most when some features can keep working while others cannot.

It matters less for a single-replica batch job that is not behind a Service, or for a workload that is useless without one dependency and has no callers that check its readiness. Even there, keeping dependency checks out of liveness still avoids restart storms.

What readiness, liveness and startup probes actually mean

That chain reaction runs on how Kubernetes interprets each probe. The three probe types answer different questions, and the kubelet reacts to each failure differently.

Putting a dependency check in liveness is strictly worse than putting it in readiness. When a database goes away, every replica fails liveness at about the same time and restarts. The restart does not fix the database. It does throw away connection pools, caches and JIT-warmed code. Repeated failures push containers into growing restart back-off. When the database returns, every replica reconnects at once and hits it with a thundering herd just as it is trying to recover.

Readiness has effects beyond traffic routing. A rolling update counts a new pod as available only once it is ready, so a dependency outage can stall a deploy that has nothing to do with that dependency. It can even block the rollout of the fix. PodDisruptionBudgets count ready pods as healthy. If your pods are unready because of someone else's outage, the eviction API can refuse evictions. Node drains then hang, and cluster maintenance stalls behind an incident your service did not cause.

How the approach works

The design splits one overloaded question, "is everything healthy?", into separate signals, and each one goes to the component that can act on it:

1. Probes report only on the pod itself. Startup says initialization finished. Liveness says the process is not wedged. Readiness says this pod can take a request right now. None of them makes a network call to a shared dependency, so a shared fault cannot flip every replica at once.
2. Dependency health is reported, not enforced. A background task in each pod records the last-known state of every dependency and exposes it on a status endpoint and as metrics. Dashboards and alerts read it, and Kubernetes never does.
3. Failures are handled where the context exists. Each outbound call has a timeout, bounded retries and a circuit breaker. When a dependency fails, the handler for that route decides whether to serve cached data, return a partial response or fail fast with a 503.
4. Configuration errors are caught once, at startup. Credentials and endpoints are validated before the pod first reports ready, so a bad rollout stalls on its new pods instead of taking traffic.

The result is that a dependency outage shows up as errors on the features that need that dependency and as an alert that names it. It does not show up as an empty Service, a stalled deploy or a restart loop. The sections below cover each piece in turn.

What readiness should check: a decision rule

Readiness should answer one question: can this pod accept a request and do useful work with it right now? That covers local conditions only. Initialization has finished. Caches are warm and config is loaded. The process is not draining for shutdown. The server can accept and handle a request. Whether the database or a downstream API is reachable is excluded by default, because that answer is the same for every replica.

The exception only holds when the failure is truly unshared. The difficulty is that a pod sees symptoms, not causes. A connection pool that looks wedged on one pod may be wedged on all of them because the database is saturated. A failing mount may be one bad volume or a storage-backend outage. If you guess "local" when the cause is shared, every replica flips unready at once and you have rebuilt the cascade. Only include a check when the failure has a local signature you can verify, such as a sidecar's own health endpoint or a mount path that can no longer be read.

The opposite risk is real too. A rollout with wrong credentials or a bad endpoint will pass a purely local check and receive traffic. Catch this at startup, not continuously. Validate config and authenticate once before reporting ready, and exit non-zero on failure. The rollout stalls on the new pods while the old ones keep serving.

Implementing separate liveness, readiness and dependency-status endpoints

Expose three endpoints with separate contracts. /livez returns 200 if the process can answer at all. It takes no shared locks and does no I/O. /readyz reads in-memory flags: startup finished, not draining, and local queue depth under its limit. /status/dependencies reports the last-known state of databases, caches and downstream services, refreshed by a background task. Dashboards and alerts read it. Probes never do.

In the container spec, give the startup probe a wide window, for example periodSeconds: 5 with failureThreshold: 30, so slow warm-up never triggers a liveness restart. Make liveness slow to act: timeoutSeconds: 2, periodSeconds: 10, failureThreshold: 3, which means roughly 30 seconds of consistent failure before a restart. Restarts are expensive, so require sustained evidence first. Make readiness quicker: timeoutSeconds: 1, periodSeconds: 5, failureThreshold: 2. Removing a pod from endpoints is cheap and reversible.

On SIGTERM, set a draining flag so /readyz returns 503, but keep serving. Wait longer than readiness detection time plus endpoint propagation, then stop accepting connections, finish in-flight work and exit. terminationGracePeriodSeconds must cover this whole sequence.

Probe handlers must be non-blocking with no synchronous network calls. Serve them from a dedicated listener or thread so a saturated worker pool or event loop cannot time out liveness and cause restarts under load.

That isolation has a cost. Probes stay green while the pod is drowning. Counter this by putting a cheap saturation signal into readiness, such as in-flight requests over a limit. Read it from counters rather than by exercising the busy pool.

Copyspec:
  # Covers the drain: ~10s readiness detection + propagation margin,
  # then in-flight work finishes. Must exceed the app's total shutdown time.
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      image: registry.example.com/api:1.4.2
      ports:
        - name: http
          containerPort: 8080   # application traffic
        - name: probes
          containerPort: 8081   # /livez and /readyz only
      # Wide warm-up window: up to 5s x 30 = 150s before liveness takes over.
      startupProbe:
        httpGet: { path: /livez, port: probes }
        periodSeconds: 5
        timeoutSeconds: 2
        failureThreshold: 30
      # Slow to act: ~30s of sustained failure before a costly restart.
      # /livez does no I/O and takes no shared locks.
      livenessProbe:
        httpGet: { path: /livez, port: probes }
        periodSeconds: 10
        timeoutSeconds: 2
        failureThreshold: 3
      # Quick to act: ~10s to leave endpoints, which is cheap and reversible.
      # /readyz reads in-memory flags only: startup done, not draining,
      # queue depth and in-flight count under their limits.
      readinessProbe:
        httpGet: { path: /readyz, port: probes }
        periodSeconds: 5
        timeoutSeconds: 1
        failureThreshold: 2
      # No probe points at /status/dependencies; dashboards and alerts
      # scrape it on port 8080 instead.

A Kubernetes pod spec that gives startup, liveness and readiness probes deliberately different timing on an isolated probe port, with a termination grace period sized to cover readiness-driven draining.

Graceful degradation at the request level

Dependency failures are better handled in the request path, where the code knows which feature is affected.

• Timeouts. Every outbound call gets a timeout shorter than the caller's own deadline.
• Bounded retries. Retries are capped at a small number of attempts, with exponential backoff and jitter so replicas don't retry in lockstep.
• Circuit breakers. A breaker fails fast once errors cross a threshold, so callers stop queueing behind a dead dependency.

When a call fails or the breaker is open, the handler decides what to return. It can serve cached or partial data, or send an explicit 503 with Retry-After. It should not hang.

Classify each dependency per feature, not per service:

The same dependency is hard for one route and soft for another. A single boolean health status can't express "checkout is down, browsing works," so any readiness answer is wrong for some routes.

A common objection is that a Ready pod accepts requests it can only fail. That's true, but removing the pod from rotation helps only if another replica or zone still has a working path to the dependency. When the dependency is shared, every replica fails the same check and the Service is left with no endpoints. Then the unaffected features go down too. Failing one feature fast with a clear 503 costs far less.

The sketch below wraps a single call in a timeout and falls back to cached data on a timeout or an open breaker.

Copy// Pricing is a soft dependency for the product page. On failure, serve the
// last known price marked stale instead of failing the whole page.
type Price = { sku: string; amount: number; currency: string };

// Base address of the pricing service, supplied by configuration.
const PRICING_BASE = process.env.PRICING_BASE ?? "";

const priceCache = new Map<string, Price>(); // last good value per SKU
const breaker = { failures: 0, openUntil: 0 }; // minimal circuit breaker

async function getPrice(sku: string): Promise<{ price: Price; stale: boolean } | null> {
  // Breaker open: skip the call so requests don't queue behind a dead service.
  if (Date.now() < breaker.openUntil) return fromCache(sku);
  try {
    // Timeout well under the page's own request deadline.
    const res = await fetch(`${PRICING_BASE}/prices/${encodeURIComponent(sku)}`, {
      signal: AbortSignal.timeout(300),
    });
    if (!res.ok) throw new Error(`pricing returned ${res.status}`);
    const price = (await res.json()) as Price;
    priceCache.set(sku, price);
    breaker.failures = 0; // healthy response resets the breaker
    return { price, stale: false };
  } catch {
    // Timeout, network error or bad status: count it. After 5 consecutive
    // failures, open the breaker for 10s and fail fast to the cache.
    if (++breaker.failures >= 5) breaker.openUntil = Date.now() + 10_000;
    return fromCache(sku);
  }
}

function fromCache(sku: string): { price: Price; stale: boolean } | null {
  const price = priceCache.get(sku);
  // null tells the page to render without a price rather than erroring.
  return price ? { price, stale: true } : null;
}

A feature-level fallback for a soft dependency: the pricing call has a 300 ms timeout and a minimal circuit breaker, and it falls back to the last cached price, marked stale, so the product page keeps rendering while pricing is down.

Alternatives and how they compare

Removing dependency checks from probes is not the only response to cascading readiness failures. The common alternatives each fix part of the problem:

• Keep the dependency check in readiness but loosen it. A higher failureThreshold or longer periodSeconds rides out short blips. It does not change the failure mode: a stall that outlasts the threshold still fails every replica together, and it also delays removal of pods that are genuinely broken.
• Check dependencies only at startup. Validating a connection before first reporting ready catches bad config and credentials, and this article recommends it. On its own it says nothing about outages after startup, so it complements request-level handling rather than replacing it.
• Fail-open readiness. Some teams have readiness check a dependency but report ready anyway when the check fails on many pods, or after it has failed for a while. Kubernetes Services have no built-in "too many unready, route to all" mode, so this logic lives in application code, needs cross-pod coordination or guesswork, and is hard to test. It reaches roughly the same end state as leaving the check out, with more moving parts.
• Outlier detection in the traffic layer. Client-side load balancers and service meshes can eject individual endpoints that return errors, based on observed responses rather than probes. That handles the genuinely local failures readiness cannot see reliably. It works alongside the approach here and does not replace per-route degradation.
• Split workloads by dependency. Running checkout and browsing as separate Deployments lets each one fail on its own hard dependencies. That reduces blast radius at the cost of more deployments to operate. Even then, a readiness check on a shared dependency still empties the affected Deployment's Service, so the probe rules still apply.

The approach in this article is the baseline the alternatives build on: probes describe the pod, and dependency failures are handled and reported per feature.

When to avoid this approach and what it costs

Fallback paths are extra code, and they are among the least exercised code in a service. A cache fallback that runs only during an outage can break silently. The cache schema drifts, a serializer changes, or the TTL logic stops matching what the product expects. You often find out during the incident it was meant to soften. Budget for tests that force each fallback to run, not only for writing it.

Some domains should not degrade at all. A payment authorization or an inventory reservation answered from stale data can double-charge a customer or oversell stock. There, a fast, explicit 503 is the correct behavior, and the request-level design should fail closed.

Resilience primitives need tuning. Thresholds that are too tight make a circuit breaker flap between open and closed, which is worse than either state. Retries multiply load on a dependency that is already struggling. Three attempts per call can roughly triple traffic at the worst moment, so cap them and add jittered backoff. Breakers are usually per instance, but hundreds of them can still trip and probe in lockstep when they share the same timeouts. Randomize those timeouts as well.

The biggest organizational cost is lost visibility. If your team treated a wave of NotReady pods as its outage alarm, that signal is gone. Replace it with explicit per-dependency error-rate and latency metrics, and alert on those before you remove the checks.

Against these costs, weigh what you get back. A partial failure no longer becomes a full outage. Deploys and node drains are no longer blocked by a database outside your control. Triage starts from a metric that names the failing dependency, instead of from a fleet of pods that look broken but are not.

How it saves engineering time

Most of the time saved comes from incidents that stay small and from investigations that start at the right place:

• Shorter triage. When the alert says "pricing timeouts from product page," the on-call engineer starts at the cause. With dependency checks in readiness, the first signal is usually a set of NotReady pods across several services, and someone has to work back through the call graph to find the shared fault.
• Smaller incidents to coordinate. A cascade pulls in every team whose Service emptied, even though their code was fine. Keeping failures scoped to the features that need the dependency keeps those teams out of the incident channel.
• Unblocked deploys during outages. New pods become ready on local conditions, so a rollout, including the rollout of a fix, is not held up waiting for an unrelated dependency to recover.
• Uninterrupted cluster maintenance. PodDisruptionBudgets see healthy pods, so node drains and upgrades proceed instead of hanging until someone notices and intervenes.
• No restart-loop cleanup. With dependencies out of liveness, there are no back-off cycles to wait out and no thundering-herd reconnect to manage when the dependency returns.

The design also moves effort earlier, into fallback tests and fault drills run on a schedule, rather than leaving it to unplanned work under pressure.

Testing the design and alerting on the right signals

Prove the design before an incident does. In a non-production cluster, inject a fault into one shared dependency: add latency with a proxy in front of it, block its port with a network policy, or scale it to zero. Run each fault long enough to cover several probe periods, then check three outcomes:

• Endpoint counts stay stable. Watch the Service's ready endpoints throughout. If pods drop out, a dependency check is still hiding in a readiness probe.
• Errors stay scoped. Only features that use the broken dependency should show higher error rates or degraded responses. Unrelated routes should not change.
• Recovery needs no restarts. Restore the dependency and confirm traffic heals without manual intervention. If a pod has to be restarted, it is probably holding a broken connection pool or a cached failure state.

Alert on what users feel

Alerts should fire on dependency-status metrics and per-feature error rates, not on probe failures. A readiness failure says a pod left rotation, not why. "Payments API unreachable from checkout" points the on-call engineer at the cause right away.

Probe review checklist

• Does readiness call a database, cache, queue or downstream service? If so, move that check to the dependency-status endpoint.
• Can liveness fail for any reason other than the process being wedged?
• Are probe timeouts shorter than the dependency's client timeouts?
• Does every external dependency expose a status metric?

Each item removes a failure mode that otherwise turns up mid-incident or mid-rollout, when it costs the most time to diagnose.

Copy#!/usr/bin/env bash
set -euo pipefail
NS=staging
DEP=payments-api   # shared dependency to break
SVC=checkout       # service that must stay in rotation
DURATION=120       # covers several 10s probe periods

ready_endpoints() {
  kubectl get endpointslices -n "$NS" -l kubernetes.io/service-name="$SVC" -o json \
    | jq '[.items[].endpoints[]? | select(.conditions.ready == true)] | length'
}
restarts() {
  kubectl get pods -n "$NS" -l app="$SVC" -o json \
    | jq '[.items[].status.containerStatuses[]?.restartCount] | add // 0'
}

ORIG=$(kubectl get deployment "$DEP" -n "$NS" -o jsonpath='{.spec.replicas}')
BEFORE_EP=$(ready_endpoints); BEFORE_RS=$(restarts)
echo "baseline: $BEFORE_EP ready endpoints, $BEFORE_RS restarts"

kubectl scale deployment "$DEP" -n "$NS" --replicas=0

for ((t=0; t<DURATION; t+=10)); do
  NOW=$(ready_endpoints)
  echo "t=${t}s ready=$NOW"
  [[ "$NOW" -lt "$BEFORE_EP" ]] && echo "FAIL: readiness still depends on $DEP"
  sleep 10
done

kubectl scale deployment "$DEP" -n "$NS" --replicas="$ORIG"
kubectl rollout status deployment "$DEP" -n "$NS" --timeout=180s

AFTER_RS=$(restarts)
[[ "$AFTER_RS" -eq "$BEFORE_RS" ]] && echo "PASS: no restarts" \
  || echo "FAIL: restarts went $BEFORE_RS -> $AFTER_RS; check pools/cached failures"

A bash fault-injection drill that scales a shared dependency to zero in a staging cluster, then checks that the dependent Service keeps its ready endpoints and recovers without container restarts.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cascading readiness failures, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.