Modular Monolith Observability: Per-Module Latency and Error Metrics

Why service-level RED metrics fail inside a monolith

RED metrics (rate, errors, duration) work well when one service maps to one team. In a monolith that mapping breaks. A single process can contain billing, search, notifications, and identity code owned by four teams. They share one set of request counters and one latency histogram. When p99 latency doubles after a deploy, the dashboard tells you the deployable is slow. It does not tell you whose code made it slow.

The usual workaround is to label metrics by HTTP route. That helps less than it seems, because routes fan out. A checkout endpoint might call into cart, pricing, inventory, and payments before it responds. A route label attributes all of that time to one URL. A regression in pricing shows up on every route that touches pricing, smeared across a dozen series. Each series looks like a different problem, and none of them names an owner.

Per-module metrics fix this by recording rate, errors, and duration at the boundaries between internal modules, labeled by module rather than by route. The core idea is that attribution belongs at module boundaries, because that is where ownership and cause line up. If the pricing module's boundary histogram moves, the pricing team owns the investigation.

Two groups benefit most:

• Teams that share one deployable and need to see their own slice.
• Platform engineers who route on-call pages.

The approach pays off when three conditions hold together:

• Several teams contribute to the same codebase.
• The package structure is stable enough that module names stay meaningful from one release to the next.
• "Whose regression is this?" incidents keep recurring, and someone spends real triage time answering that question.

How boundary instrumentation works: taxonomy, entry points, and context propagation

A fixed module taxonomy

Start with a closed list of module names, derived from top-level packages or namespaces (billing, catalog, identity) and kept in one file that changes only through code review. A closed list keeps label cardinality bounded. A single owner keeps names from drifting, so billing never quietly becomes payments in half the dashboards.

Three instrumentation points

1. Inbound entry points. HTTP handlers, queue consumers, and scheduled jobs record which module owns the request. This answers which module the request belongs to.
2. Cross-module calls. Each module exposes a facade or interface, and the wrapper on that facade records latency and errors for caller → callee. This answers which module the time went to.
3. Outbound calls. Database, cache, and external API clients tag each call with the calling module. This answers which module issued the slow query.

CopyHTTP POST /checkout
      │
 [1] entry: module=orders ───────────── ctx.module = orders
      │
 orders ──[2] facade: orders→billing ──► billing   ctx.module = billing
                                           │
                                     [3] db query, module=billing
      │
 [3] cache get, module=orders

Propagating module context

Outbound calls run through a shared data access layer, and changing every call site to pass the module name does not scale. Instead, the entry point writes the module into the request context. Depending on the runtime, that is a context object, thread-local storage, or async-local storage. Each facade wrapper overwrites the value for the duration of the call and restores the previous value on return. The database and cache clients read the current value when they record a metric, so attribution follows the code that is executing, not the module that handled the original request.

Copy     HTTP POST /checkout
                │
                ▼
┌───────────────────────────────┐
│ [1] entry point               │  ctx.module = orders
│     records module=orders     │
└───────────────┬───────────────┘
                ▼
┌───────────────────────────────┐     ┌──────────────────────────┐
│ orders module                 │     │ billing module           │
│                               │ [2] │                          │
│  calls billing facade ────────┼────►│  ctx.module = billing    │
│  (orders→billing latency/err) │     │  (restored on return)    │
└───────────────┬───────────────┘     └────────────┬─────────────┘
                │                                  │
                ▼                                  ▼
┌──────────────────────────────────────────────────────────────┐
│ shared data access layer: reads ctx.module per call          │
└───────────────┬──────────────────────────────────┬───────────┘
                ▼                                  ▼
          [3] cache get                      [3] db query
          module=orders                      module=billing

A checkout request: the entry point sets ctx.module, the facade wrapper records orders→billing and swaps the context, and the shared data layer labels each outbound call with the current module.

Implementing it: Prometheus label design and boundary wrappers

Label design

Keep three labels: module, operation, and outcome. Each must come from a closed set defined in code. Define modules as an enum or constant registry so that a typo fails at compile time or startup instead of creating a new series. Give operation a fixed list per module. Limit outcome to success, client_error, and server_error.

Boundary wrappers

The wrapper takes a module constant and an operation name. It starts a timer, calls the handler, maps the result or exception to an outcome, and records one histogram observation. On failure it also increments an error counter. It reads the exception's class only to choose the outcome. The message goes to logs, never to labels.

When work moves to an executor or async task, capture the current module context at submission and restore it inside the task. Without that step, the database and cache clients record their metrics as unknown.

Series count (illustrative estimate)

Assumed inputs: 20 modules × 10 operations × 3 outcomes gives 600 label combinations. With 12 buckets plus _sum and _count, each combination produces 14 series, so the histogram has 8,400 series. Each new label multiplies that total by its number of values. A 5-value region label raises it to 42,000. A user ID with 10,000 values raises it to 84 million.

Copyimport contextvars, time
from concurrent.futures import ThreadPoolExecutor
from enum import Enum
from prometheus_client import Counter, Histogram

class Module(str, Enum):  # closed registry: Module.BILING raises AttributeError
    BILLING = "billing"
    ORDERS = "orders"

OPERATIONS = {Module.BILLING: {"charge", "refund"}, Module.ORDERS: {"place", "cancel"}}
current_module = contextvars.ContextVar("current_module", default="unknown")

LABELS = ["module", "operation", "outcome"]  # no user IDs, paths, or messages
LATENCY = Histogram("module_op_seconds", "Module boundary latency", LABELS)
ERRORS = Counter("module_op_errors_total", "Module boundary errors", LABELS)

class ClientError(Exception): pass

def boundary(module: Module, operation: str, fn, *args):
    if operation not in OPERATIONS[module]:
        raise ValueError(f"unregistered operation {operation!r}")  # fail fast, no new series
    token = current_module.set(module.value)  # shared db/cache clients read this
    start, outcome = time.perf_counter(), "success"
    try:
        return fn(*args)
    except ClientError:
        outcome = "client_error"; raise  # class picks outcome; message goes to logs only
    except Exception:
        outcome = "server_error"; raise
    finally:
        LATENCY.labels(module.value, operation, outcome).observe(time.perf_counter() - start)
        if outcome != "success":
            ERRORS.labels(module.value, operation, outcome).inc()
        current_module.reset(token)

pool = ThreadPoolExecutor(max_workers=4)

def submit_with_context(fn, *args):
    # Pool threads don't inherit contextvars; copy at submission and run inside that copy
    return pool.submit(contextvars.copy_context().run, fn, *args)

A Python boundary wrapper using an enum module registry, bounded labels, a latency histogram and error counter, and contextvars copied into executor tasks so shared-layer metrics keep the module.

Attributing latency and errors correctly

A boundary histogram measures inclusive time: everything between entering and leaving the module, including time spent in modules it calls. If checkout calls pricing, and pricing slows down by 200 ms, both modules' p99 rises. The checkout team gets paged for code they did not change.

There are two ways to fix this:

• Caller/callee labels. On cross-module calls, record a separate histogram labelled with caller and callee. You can then see how much of checkout's latency comes from its calls into pricing. Cardinality is bounded by the number of real edges in the call graph, not by every possible module pair.
• Self time. Track the time spent in child boundaries in the request context. On exit, subtract it from the elapsed time and record the result as exclusive time. This shows the cost of the module's own code, but only if every child call is wrapped. A call that bypasses a wrapper is counted as self time.

Label every panel as inclusive or exclusive, and never stack the two on one chart. Inclusive figures overlap, so summing them across modules counts the same time more than once.

Errors need explicit rules too. Attribute each failure to the module where it originated, and record where it surfaced as a separate label if you need it. Use a bounded error_class label with a small set of values such as timeout, validation, dependency, and internal. Never use raw exception messages as label values. Count the error once, at its origin, and mark it as already counted when it propagates. Otherwise one database timeout shows up as five module failures.

PromQL patterns and bridging to traces

…three errors in three modules, and the error ratio below would accuse the wrong code.

Core queries

The per-module error ratio divides errors by calls, grouped by module: sum by (module) (rate(module_errors_total[5m])) / sum by (module) (rate(module_calls_total[5m])). Use a ratio instead of a raw error rate so that a busy module does not look worse than a quiet one just because it handles more traffic.

For tail latency, use histogram_quantile(0.95, sum by (module, le) (rate(module_duration_seconds_bucket[5m]))), and swap in 0.99 for p99. The sum by (module, le) step matters. It merges buckets across instances before the quantile is estimated. If you average per-instance quantiles instead, the result is not statistically meaningful. If you drop le, histogram_quantile has nothing left to estimate from.

During an incident, ask which module's share of total request time grew. If you record exclusive (self) time, divide each module's rate(..._sum[5m]) by the total across all modules. The shares then add up to roughly one, and a module that jumps from 10% to 45% stands out immediately. With inclusive time the shares overlap and add up to more than one, so read them only as relative trends.

From spike to request

Set module as an attribute on the span your boundary wrapper already creates. If your histograms carry exemplars, a point on the p99 panel links to a trace ID from that bucket. If they don't, filter trace search on module = "billing" with duration above the spike threshold.

Metrics tell you where the time went. A trace shows you the sequence of operations inside one slow request. A profile shows the code paths that burned the CPU. Your investigation usually needs all three.

Copy# Prometheus recording rules for per-module dashboards and alerts.
# Precomputing these keeps incident-time panels fast and consistent.
groups:
  - name: module-observability
    interval: 30s
    rules:
      # Error ratio per module: errors / calls. A busy module does not
      # look worse than a quiet one just because it has more traffic.
      - record: module:errors_per_call:ratio_rate5m
        expr: |
          sum by (module) (rate(module_errors_total[5m]))
          /
          sum by (module) (rate(module_calls_total[5m]))

      # p95 and p99: merge buckets across instances first (keep `le`),
      # then estimate the quantile. Never average per-instance quantiles.
      - record: module:duration_seconds:p95_5m
        expr: |
          histogram_quantile(0.95,
            sum by (module, le) (rate(module_duration_seconds_bucket[5m])))
      - record: module:duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (module, le) (rate(module_duration_seconds_bucket[5m])))

      # Share of total request time, using exclusive (self) time so the
      # shares sum to roughly 1. The right side has no labels, so
      # ignoring(module) + group_left divides every module by the total.
      - record: module:self_time:share_rate5m
        expr: |
          sum by (module) (rate(module_self_duration_seconds_sum[5m]))
          / ignoring(module) group_left
          sum(rate(module_self_duration_seconds_sum[5m]))

Prometheus recording rules that precompute the per-module error ratio, p95/p99 from instance-merged histogram buckets, and each module's share of total self time for incident dashboards.

When to avoid it and what the alternatives are

Every label you add multiplies the series count. With 40 modules, 30 entry points, 5 outcome values, and a 12-bucket histogram, you get 72,000 series per instance before any pod label is added (illustrative estimate). On hot in-process paths, a timer plus a label lookup on every call can cost as much as the work being measured. That pushes you toward sampling or toward coarser boundaries.

Attribution is only as accurate as the boundaries are clean. A caller that imports another module's internals instead of going through its facade emits no boundary event. The dashboard then looks clean while a real dependency goes unmeasured. Work handed to thread pools or callbacks also drops the module context unless you propagate it explicitly, so that time lands on the caller or in "unknown."

Shared resources are the hardest case. When one module exhausts the connection pool, triggers a GC pause, or holds a contended lock, every module waiting on that resource shows higher latency. The labels end up blaming the victims rather than the cause.

Skip this approach for small single-team codebases, where route metrics plus a profiler already answer most questions. Also skip it for modules whose boundaries are still moving, because each refactor breaks dashboards and alerts.

Ownership, enforcement, and impact on cost and engineering time

Map modules to owners in one place

Keep the module registry as the single source of truth: each entry holds the module name, owning team, and on-call route. Do not stamp a team label on every series. Export the mapping as an info-style metric (for example module_owner_info{module, team}) and join on it in recording rules and alert expressions. A reorg then becomes a one-line registry change. A team-specific label would instead fork every series. Alertmanager can route on the joined team label, so a page for billing goes straight to the billing rotation and never sits in a shared queue.

Enforce it in CI

Add a test that enumerates the public facades of each module and fails if one lacks the boundary wrapper. Add a lint rule that rejects string literals passed as the module label, so values can only come from registry constants. Together they catch the two common failure modes: forgotten instrumentation and a typo that silently creates a new label value.

Rename and split without breaking history

Treat a rename like a schema migration:

1. Add the new name to the registry.
2. Dual-emit or map old to new with label_replace in recording rules.
3. Move dashboards and alerts to the recorded series.
4. Drop the old name once retention covers the overlap.

For a split, map each new submodule back to the parent until the alerts have been re-baselined.

Cost versus time saved

The costs are front-loaded: writing wrappers, a taxonomy review per module, and storage that grows with modules × routes × status. The savings recur. Triage starts at the right code area, fewer incidents bounce between teams, and pages reach the owner first.

Copy# prometheus rules file: rename-safe recording rule plus owner join for routing
groups:
  - name: module-slo
    rules:
      # Rename migration: fold the legacy module name into the new one so
      # dashboards and alerts read one continuous series during the overlap.
      - record: module:requests_error:ratio_rate5m
        expr: |
          sum by (module) (
            label_replace(rate(module_requests_total{status="error"}[5m]),
                          "module", "invoicing", "module", "billing_legacy")
          )
          /
          sum by (module) (
            label_replace(rate(module_requests_total[5m]),
                          "module", "invoicing", "module", "billing_legacy")
          )
      # Ownership comes from the registry-exported info metric, not from a
      # team label on every series. group_left copies `team` onto the alert.
      - alert: ModuleErrorRateHigh
        expr: |
          (module:requests_error:ratio_rate5m > 0.05)
          * on (module) group_left (team) module_owner_info
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.module }} error ratio above 5% (owner: {{ $labels.team }})"
---
# alertmanager.yml (excerpt): route on the joined team label
route:
  receiver: platform-default
  routes:
    - matchers: ['team="billing"']
      receiver: billing-oncall
    - matchers: ['team="catalog"']
      receiver: catalog-oncall

A Prometheus recording rule uses label_replace to keep a renamed module's series continuous, an alert joins module_owner_info to attach the owning team, and Alertmanager routes the page on that team label.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to modular monolith observability, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.