Microservice Health Check Aggregation: A Dependency-Aware Design

Liveness, readiness and deep health checks: three questions, three consumers

A service exposes health at three levels, and each one answers a different question for a different consumer. When they are blurred together, the result is either useless signals or automated systems acting on information they shouldn't.

A liveness failure has one remedy, which is a restart. A restart only helps when the fault is inside the process, so liveness must never depend on anything outside it. Readiness is also local, but it can be temporary. An instance that is still loading configuration or shutting down should be skipped, not killed.

An aggregated dependency health endpoint is the third kind. It reports the status of every dependency the service needs, which makes it a diagnostic tool and not a control signal. If an orchestrator acts on it, one failing database can take every instance out of service at the same moment.

Teams that should build one are backend and platform teams running services with several downstream dependencies. These are the services where "it's up but broken" is a common incident and where on-call engineers need to see quickly which dependency is at fault.

Why dependency checks in probes cause restart storms

Consider a service running twenty replicas against one shared PostgreSQL primary. The primary hits a lock contention spike and query latency climbs from 5 ms to 3 seconds. If each replica's liveness probe runs SELECT 1 with a 2-second timeout, all twenty fail at roughly the same moment, because they share the same dependency and usually the same probe interval. The orchestrator has no way to know the failure is shared. It sees twenty unhealthy containers and restarts them. If the failing check is readiness instead, it removes all twenty from the load balancer. Either way, a slow database has become a complete outage of a service that could still have served cached reads, queued writes or returned useful errors.

Recovery then makes things worse. Twenty fresh processes start at once, each opening a full connection pool and warming caches against a database that is already struggling. That surge can push the dependency back over the edge, the probes fail again, and the cycle repeats.

There is an honest exception. A readiness check that tests something only this instance owns is useful. Examples include a connection pool that has leaked every connection or a local disk that has filled up. Pulling that one replica from rotation routes traffic to healthy peers, which is exactly what readiness is for.

The deciding question is whether the failure is local to one instance or shared across the fleet. Local failures belong in readiness. Shared failures belong in deep health checks that report status without triggering orchestrator action. There is no universal rule, because the same dependency can fail in either way, so each check needs a deliberate classification.

Classifying dependencies and designing the response contract

Three criticality classes cover most services:

The rollup is the worst status produced by any non-informational component. A single boolean cannot express "working, with reduced features," and that is the state operators most need to see.

Each component entry in the response should carry these fields:

• name
• status
• criticality
• latency_ms, the duration of the last check
• last_checked, an absolute timestamp
• age_ms, how old the cached result is
• error, a short reason such as connection refused or timeout after 500ms

age_ms lets a consumer tell a fresh failure apart from a stale success. Keep error terse so it cannot leak connection strings or stack traces.

For HTTP status codes, return 200 for both healthy and degraded. Return 503 only when a hard dependency is down. Load balancers and uptime monitors usually read the status code and ignore the body. If a degraded service returns 503, those consumers pull capacity that could still serve most requests. One slow cache can then empty an entire pool. Always returning 200 creates the opposite problem: code-only monitors never alert, and the body becomes the only signal that a hard dependency has failed.

Criticality labels drift as code changes. A "soft" dependency becomes hard once a new code path stops tolerating its absence. Assign each label an owning team and review it whenever the call graph changes, for example as a checklist item on pull requests that add or remove client calls.

Copyfrom dataclasses import dataclass, asdict
from typing import Optional

RANK = {"healthy": 0, "degraded": 1, "unhealthy": 2}

@dataclass
class Component:
    name: str
    status: str               # probe result: "up" or "down"
    criticality: str          # "hard" | "soft" | "informational"
    latency_ms: int           # duration of the last check
    last_checked: str         # absolute ISO 8601 UTC timestamp
    age_ms: int               # age of the cached result
    error: Optional[str] = None  # terse reason only, never DSNs or stack traces

def effect(c: Component) -> str:
    # Informational components are reported but never change the rollup.
    if c.status == "up" or c.criticality == "informational":
        return "healthy"
    return "unhealthy" if c.criticality == "hard" else "degraded"

def rollup(components: list[Component]) -> tuple[int, dict]:
    # Overall status is the worst effect of any component.
    overall = max((effect(c) for c in components), key=RANK.__getitem__, default="healthy")
    # 503 only when a hard dependency is down; degraded still serves traffic.
    code = 503 if overall == "unhealthy" else 200
    return code, {"status": overall, "components": [asdict(c) for c in components]}

code, body = rollup([
    Component("orders-db", "up", "hard", 4, "2026-10-24T09:15:02Z", 1200),
    Component("cache", "down", "soft", 500, "2026-10-24T09:15:02Z", 1200,
              "timeout after 500ms"),
    Component("analytics-exporter", "down", "informational", 12,
              "2026-10-24T09:14:50Z", 13400, "connection refused"),
])
print(code, body["status"])  # 200 degraded: load balancer keeps this instance in the pool

A minimal Python rollup that maps hard, soft and informational component failures to an overall healthy, degraded or unhealthy status, builds the per-component response body, and returns 503 only when a hard dependency is down.

How to implement cached, concurrent, time-bounded checks

Run checks off the request path

Run every check on a background timer, not when the endpoint is called. Each tick launches all checks concurrently, so a tick takes as long as the slowest check rather than the sum of all of them. Give each check its own timeout well below the caller's. If the aggregator gives up at 2 s, a 500 ms per-check budget leaves room for network and serialization. Record a timeout as a failure with an explicit reason, such as "timed out after 500 ms", so an operator can tell a slow dependency from one that refuses connections. Write results to a cache that the handler only reads. A burst of probes then never becomes a burst of dependency traffic.

Make checks meaningful but cheap

A TCP connect proves a port is open, not that the database can answer queries. A write or an expensive query adds load and side effects. A fresh connection on every run tests a path that real traffic doesn't use, and it can exhaust connection limits. Prefer a trivial query on the existing pool, or a metadata call such as describing a queue or confirming that a bucket exists.

Damp flapping

Change the reported state only after N consecutive results agree. This hysteresis buys stability at the cost of slower detection.

Illustrative estimate (assumed numbers, not a benchmark): worst-case detection delay ≈ N × interval + timeout. With N = 3, a 10 s interval and a 0.5 s timeout, the delay is 3 × 10 + 0.5 = 30.5 s. Plug in your own values to check whether that delay is acceptable.

Copyimport asyncio, time

CHECK_TIMEOUT = 0.5  # per-check budget, well below the aggregator's 2 s
INTERVAL = 10        # seconds between ticks
N = 3                # consecutive agreeing results before the state flips
cache = {}           # name -> state; the only thing the handler reads

async def db_check(pool):
    # Cheap and meaningful: a trivial query on the existing pool, no new connection.
    async with pool.acquire() as conn:
        await conn.fetchval("SELECT 1")

async def run_one(name, check):
    try:
        await asyncio.wait_for(check(), timeout=CHECK_TIMEOUT)
        return name, True, "ok"
    except asyncio.TimeoutError:  # a timeout is a failure with an explicit reason
        return name, False, f"timed out after {int(CHECK_TIMEOUT * 1000)} ms"
    except Exception as exc:      # e.g. connection refused
        return name, False, f"{type(exc).__name__}: {exc}"

def record(name, ok, reason):
    e = cache.setdefault(name, {"healthy": None, "last": ok, "streak": 0})
    e["streak"] = e["streak"] + 1 if ok == e["last"] else 1
    e["last"] = ok
    # Hysteresis: take the first result, then flip only after N agreeing results.
    if e["healthy"] is None or (ok != e["healthy"] and e["streak"] >= N):
        e["healthy"] = ok
    e.update(reason=reason, checked_at=time.time())

async def checker_loop(checks):
    while True:
        # Concurrent: a tick lasts as long as the slowest check, not the sum.
        results = await asyncio.gather(*(run_one(n, c) for n, c in checks.items()))
        for name, ok, reason in results:
            record(name, ok, reason)
        await asyncio.sleep(INTERVAL)

def health_handler():
    # Request path: read the cache only. A burst of probes adds no dependency traffic.
    return {n: {"healthy": e["healthy"], "reason": e["reason"]} for n, e in cache.items()}

A minimal asyncio sketch in which a background loop runs all checks concurrently, each under its own timeout. Timeouts are recorded as failures with a reason, the reported state flips only after N consecutive agreeing results, and the endpoint handler only reads the cache.

Avoiding recursive fan-out with a system-level aggregator

If A's deep check calls B's deep check, which calls C's, one request to A turns into a walk of the whole subgraph below it. Fan-out multiplies the cost at every level. As an illustrative estimate, with three dependencies per service and three levels, a single probe triggers 3 + 9 + 27 = 39 downstream calls, and every dashboard refresh or load balancer probe repeats that walk. Latency stacks too. A's timeout has to exceed the sum of the timeouts beneath it, or A reports a failure that is really just slowness further down. A cycle in the graph (A → B → A) makes the checks call each other until a timeout breaks the loop.

The rule is that each service reports only its direct dependencies, using a cheap signal such as a connection ping or the dependency's readiness endpoint, never the dependency's own deep check.

The system-level aggregator

A separate aggregator builds the system-wide view. It keeps a dependency map that is either declared (config in version control) or discovered (from tracing or service mesh data). It polls each service's deep endpoint on a fixed interval, so the load is one call per service per interval no matter how many people are viewing the status page. It then rolls status up by criticality.

Because the aggregator sees the whole graph, it can point to the likely root cause. When twelve services report the same database as failing, it flags the database and marks the twelve as impacted instead of showing twelve red tiles. A service it cannot reach is shown as unknown, not as healthy or down. The aggregator is itself a dependency, so it needs its own external monitoring, including an alert when its last successful poll goes stale.

Copy +------------------+         +-------------------------------------------+
 | External monitor |-------->| Aggregator                                |
 | alerts if last   | watches | - dependency map (declared/discovered)    |
 | poll is stale    |         | - polls each deep endpoint per interval   |
 +------------------+         | - one call per service per interval       |
                              | - rolls up status by criticality          |
                              | - flags the likely root cause             |
                              +-------------------------------------------+
                                    |               |               |
                        poll deep   |               |               | timeout
                                    v               v               v
                              +------------+  +------------+  +------------+
                              | Service A  |  | Service B  |  | Service C  |
                              | impacted   |  | impacted   |  | UNKNOWN    |
                              +------------+  +------------+  +------------+
                                 |      |       ^   |
                                 |      +-------+   |
                            ping |   readiness only | ping
                                 v                  v
                              +-----------------------+
                              | Database              |
                              | ROOT CAUSE (flagged)  |
                              +-----------------------+

 Each service checks only its direct dependencies with a cheap signal;
 none calls another service's deep check, so probes never fan out.

The aggregator polls each service's deep endpoint once per interval and flags the shared database as the root cause instead of marking every dependent service red. It shows unreachable Service C as unknown and is itself watched by an external monitor that alerts on stale polls.

When to avoid it, security exposure and alternatives

A deep health endpoint pays off when many services share many dependencies and on-call engineers need to localise faults quickly. In a system with one or two dependencies, the failing component is usually obvious from the error logs, so the endpoint becomes code you maintain without getting much back. If your team already alerts on request-level SLOs, the endpoint adds diagnosis, not detection.

Even a well-built endpoint has a hard limit: a green check does not prove users are being served. A synthetic check can pass while real requests fail on malformed data in specific records, expired or misconfigured auth for some tenants, or a partial network partition that only affects some replicas. Treat the endpoint as one signal among several.

Deep responses can also leak hostnames, versions, raw error messages and topology, which is useful reconnaissance for an attacker. Serve the detailed view on an internal port or behind authentication, and expose only a coarse up/degraded/down status publicly. Rate-limit the endpoint even when it is cached, because the cache protects your dependencies, not the endpoint itself. For error detail, an error category such as "timeout" usually gives on-call engineers enough to act on. A full exception string mostly helps whoever is probing you.

Impact on cost and engineering time

For load, assume each instance refreshes its cached result once per interval, so caller traffic to the endpoint doesn't multiply dependency traffic. The formula is:

checks/s per dependency = instances ÷ interval (s)

Worst-case staleness = interval + check timeout

Illustrative estimate (assumed numbers, not a benchmark): 40 instances, 5 dependencies, 2 s check timeout.

In this illustrative estimate, with the assumed 40 instances and 2 s timeout, shortening the interval from 30 s to 2 s cuts worst-case staleness from 32 s to 4 s, about 28 s less detection lag. The cost is roughly 15× more probe traffic (20 instead of ~1.3 checks/s per dependency). These figures follow from the formulas and the assumed inputs, not from measurement, so substitute your own instance count, interval and timeout. The staleness column matters because a dependency can fail just after a check completes. The cached "healthy" result then stays in place until the next check finishes or times out. Choose the interval based on how quickly someone can act on the signal, not on how fresh the result looks.

On the development side, you pay for writing and testing a check per dependency and keeping critical/non-critical labels accurate as the architecture changes. You also need an owner for each label and must run the aggregator as a service with its own alerts. Stale labels are the most common failure: a dependency that became critical but is still marked optional quietly hides real outages.

The savings show up during incidents. On-call engineers can see which dependency failed first, so they don't chase timeouts across five downstream services. Keeping probes free of dependency checks also removes restart storms that you caused yourself, which are often harder to diagnose than the original fault.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice health check aggregation, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.