Monolith Module Latency Tracing with OpenTelemetry Spans

Why auto-instrumented traces of a monolith look flat

Monolith module latency tracing means wrapping the entry points of a monolith's internal modules in their own spans, so a single request's latency can be attributed to the modules that spent it.

Auto-instrumentation can't do this on its own. It hooks the places where a framework or library has a known shape: the inbound HTTP handler, the database driver, the outbound HTTP client. Calls between your own packages, such as pricing calling inventory calling tax, are ordinary function calls. No library sits in between, so nothing is intercepted. The result is a root server span with a scattering of SQL and HTTP client children. Most of the request's duration ends up in unlabeled gaps between those children.

In the illustrative trace below, about 690 ms of an 840 ms request is unaccounted for:

CopyAuto-instrumented (flat)           Module-nested (what you'll build)

POST /checkout      840ms          POST /checkout             840ms
├─ SELECT carts      40ms          ├─ module.cart             120ms
├─ SELECT prices     20ms          │  └─ SELECT carts          40ms
└─ HTTP POST tax-api 90ms          ├─ module.pricing          510ms
   (≈690ms unexplained)            │  └─ SELECT prices         20ms
                                   └─ module.tax              170ms
                                      └─ HTTP POST tax-api     90ms

This tutorial is for backend and platform engineers who run a modular monolith and already send traces to an APM or an OpenTelemetry backend. It pays off in one situation: requests are slow, and the trace shows the slow time is not in the database or in outbound calls. That means the time is being spent in your own code, and the next step is to find out which module is spending it.

Copy                     POST /checkout
                           │
┌──────────────────────────▼───────────────────────────────┐
│ Monolith process                                         │
│                                                          │
│   ┌──────────────────────────────────────────┐           │
│   │ HTTP server handler   [auto: root span]  │           │
│   └──────────────────────┬───────────────────┘           │
│                          │ plain function calls, no hook │
│   ┌──────────────────────▼───────────────────┐           │
│   │ cart ──▶ pricing ──▶ inventory ──▶ tax   │           │
│   │ invisible to auto-instrumentation        │           │
│   │ [manual: module.* spans go here]         │           │
│   └─────┬──────────────────────────┬─────────┘           │
│         │                          │                     │
│   ┌─────▼──────────────┐   ┌───────▼─────────────┐       │
│   │ DB driver          │   │ HTTP client         │       │
│   │ [auto: SQL spans]  │   │ [auto: client span] │       │
│   └─────┬──────────────┘   └───────┬─────────────┘       │
└─────────┼──────────────────────────┼─────────────────────┘
          │                          │
   ┌──────▼──────┐            ┌──────▼──────┐
   │  Database   │            │   tax-api   │
   └─────────────┘            └─────────────┘

 auto   = span emitted by auto-instrumentation hooks
 manual = module span you add at each module entry point

Auto-instrumentation hooks only the inbound handler, the database driver and the outbound HTTP client, so the plain function calls between internal modules go untraced until you add module spans at their entry points.

Who should use module boundary spans

Module spans are worth the effort when all of the following hold:

• The codebase is a monolith with recognizable internal modules. Packages or namespaces such as billing, pricing and inventory have public entry points that other code calls. If the code has no real module structure, there is no stable boundary to put a span on.
• Tracing is already in place. You have a tracer provider, an exporter and a backend that stores spans. Module spans add detail to existing traces. They do not replace the tracing pipeline.
• Unexplained latency lives in your own code. The edge spans (SQL, outbound HTTP) do not account for the slow part of the request.
• Several teams own different modules. Endpoint latency alone does not tell you which team should investigate. A per-module breakdown does.

Typical users are backend engineers who own one or more modules and want to see their module's share of a request, and platform or SRE engineers who run the tracing pipeline and need latency to be attributable to owners. Small codebases maintained by one team, and services whose latency is already explained by edge spans, usually get less from this. The last section covers those cases in more detail.

How it works

The technique relies on how OpenTelemetry tracks the "current span" inside a process:

1. A request arrives. The auto-instrumented HTTP server handler starts a root server span and stores it as the active span in the current context.
2. A module entry point is called. A boundary wrapper (a decorator or interceptor) calls the module's tracer to start a new span. Because a span started with start_as_current_span takes the active span from context as its parent, the module span becomes a child of the root span, and it is made active while the module's code runs.
3. Nested calls nest automatically. If pricing calls inventory, the inventory boundary span finds the pricing span in context and becomes its child. Auto-instrumented SQL and HTTP client spans attach to whichever module span is active at that moment, so database and network calls appear under the module that made them.
4. Context has to be carried across threads and queues. Context follows ordinary function calls on the same thread. Work handed to thread pools, some async callbacks or job queues needs the context captured and restored explicitly, or the child spans become orphan roots.
5. Spans are exported. When each span ends, a span processor hands it to an exporter, which sends it to your collector or backend. Head or tail sampling decides which traces are kept.
6. Latency is attributed offline. For each span you compute self-time: its duration minus the merged time covered by its direct children. You then group self-time by the app.module attribute across many traces to see which module spends the time, and use a profiler to look inside that module.

The rest of this article covers each step: choosing boundaries, writing the wrapper and context propagation, computing self-time, aggregating across traces, and keeping the instrumentation correct as the code changes.

Choosing module boundaries to trace

A module's public entry points are where it accepts work from the rest of the codebase. Those are the places to put spans. Service facades, command or message handlers, and repository interfaces are good candidates because callers already treat them as the module's contract. A span there measures the cost of asking the module to do something, and that is the number you can compare between modules. Spans on private helpers mostly reflect how the code happens to be organized today. That changes with every refactor, so the data stops being comparable from one release to the next.

The other constraint is overhead. Each span allocates an object, records timestamps and attributes, and must be exported or dropped. That cost is small at a facade called once per request. At a function called thousands of times inside a loop, the cost adds up, and a single trace can contain so many spans that it becomes unreadable. Put spans on the boundary and leave the hot internals to a profiler.

Write the chosen boundaries in a short list in the repository. Give each one a module name, an entry point and a span name. Code review can check new spans against this list, and the validation section later in this article uses it to detect drift.

How to implement boundary spans with the OpenTelemetry tracer API

Create one tracer per module, named after the module rather than the class or file: trace.get_tracer("app.billing"). The tracer name becomes the instrumentation scope, so you can filter or sample a whole module without parsing span names. Span names should be stable verbs such as billing.charge and billing.refund. Every span also carries app.module and app.operation attributes. Backends group and index attributes more reliably than they parse names, and the self-time math in the next section depends on these two keys being present on every span.

Keep instrumentation at the boundary

Wrap each public entry point once with a decorator, or with an interceptor if your framework exposes one. The wrapper starts the span with start_as_current_span, sets both attributes, records exceptions, and sets the span status. Business code never imports the tracer. If instrumentation is scattered through method bodies, it drifts as those bodies are refactored.

Carry context across execution boundaries

In-process context is stored in context-local state, and it does not always follow the work:

• Thread pools: a task passed to an executor runs without the caller's context, so its spans become orphan roots. Capture context.get_current() at submit time and attach/detach it inside the worker.
• Async tasks: most runtimes copy context when a task is created. Context can still be lost when tasks are created from callbacks or from a different event loop.
• Job queues: serialize the context into the job payload with the propagator's inject, then extract it in the consumer. Start the consumer span as a link or a child, depending on how you want queue wait time counted.

What never goes in names or attributes

Do not put order IDs, user IDs, SQL text, or raw input in span names or attributes. Unique values multiply the number of distinct series your backend stores, which raises cost. They can also leak personal data or secrets into a system that usually has weaker access controls than your database.

Copyimport functools
from concurrent.futures import ThreadPoolExecutor
from opentelemetry import context, trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("app.billing")  # one tracer per module = instrumentation scope

def boundary(operation):
    # The only place this module touches the tracer. Business code stays clean.
    def decorate(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            # Stable, low-cardinality name. Never put IDs, SQL text or user input here.
            with tracer.start_as_current_span(f"billing.{operation}", record_exception=False,
                                              set_status_on_exception=False) as span:
                span.set_attribute("app.module", "billing")
                span.set_attribute("app.operation", operation)
                try:
                    return fn(*args, **kwargs)
                except Exception as exc:
                    span.record_exception(exc)
                    # Use the exception type, not str(exc), which may contain user data.
                    span.set_status(Status(StatusCode.ERROR, type(exc).__name__))
                    raise
        return wrapper
    return decorate

def with_context(fn):
    ctx = context.get_current()  # capture at submit time, on the caller's thread
    def run(*args, **kwargs):
        token = context.attach(ctx)  # restore it on the worker thread
        try:
            return fn(*args, **kwargs)
        finally:
            context.detach(token)
    return run

pool = ThreadPoolExecutor(max_workers=4)

@boundary("charge")
def charge(order): ...

@boundary("settle_batch")
def settle_batch(orders):
    for order in orders:
        # Lost: pool.submit(charge, order) makes each billing.charge span an orphan root.
        # Fixed: each billing.charge span becomes a child of billing.settle_batch.
        pool.submit(with_context(charge), order)

A per-module tracer and a boundary decorator that set a stable span name, app.module and app.operation, and error status on every span, plus a wrapper that carries the caller's context into thread-pool workers so their spans no longer become orphan roots.

Computing self-time per module, including concurrent children

Ranking modules by total span duration puts the outermost caller at the top every time. A checkout span that wraps pricing, inventory, and payment will always look slowest, because its duration includes every millisecond spent in its children. To find where time actually goes, compute self-time: the span's duration minus the time covered by its direct children.

The naive approach subtracts the sum of child durations. That works only when children run one after another. When a module fans out with concurrent calls, the child intervals overlap. Summing them counts the same wall-clock time more than once, and self-time can come out too low or even negative. The fix is to subtract the union of the child intervals:

1. Take the span's direct children and clip each interval to the parent's start and end.
2. Sort the intervals by start time and merge any that overlap.
3. Sum the merged lengths and subtract that total from the parent's duration.

Illustrative estimate (assumed numbers, not a benchmark). checkout runs from 0 to 120 ms. pricing runs from 10 to 50 ms and inventory from 20 to 50 ms, in parallel. payment runs from 60 to 85 ms.

• Naive: 120 − (40 + 30 + 25) = 25 ms
• Merged: 120 − ((50 − 10) + (85 − 60)) = 120 − 65 = 55 ms

Formula: self = duration − Σ len(merge(clip(children))).

Self-time tells you where a module was blocked, not why. Lock contention, connection-pool waits, and garbage-collection pauses all show up as self-time in whichever module happened to be running when they occurred.

Copyfrom collections import defaultdict

def self_times(spans):
    """spans: dicts with span_id, parent_id, name, start_ms, end_ms.
    Returns exclusive (self) time per span_id, using the merged
    interval covered by direct children so parallel calls are not double-counted."""
    children = defaultdict(list)
    for s in spans:
        if s["parent_id"] is not None:
            children[s["parent_id"]].append(s)

    result = {}
    for s in spans:
        p_start, p_end = s["start_ms"], s["end_ms"]
        # Clip each child to the parent's bounds (async children may outlive it).
        ivals = sorted(
            (max(c["start_ms"], p_start), min(c["end_ms"], p_end))
            for c in children[s["span_id"]]
        )
        covered, cur_start, cur_end = 0.0, None, None
        for a, b in ivals:
            if b <= a:
                continue  # child lies entirely outside the parent
            if cur_end is None or a > cur_end:
                if cur_end is not None:
                    covered += cur_end - cur_start
                cur_start, cur_end = a, b
            else:
                cur_end = max(cur_end, b)  # overlap: extend merged interval
        if cur_end is not None:
            covered += cur_end - cur_start
        result[s["span_id"]] = (p_end - p_start) - covered
    return result

spans = [
    {"span_id": "a", "parent_id": None, "name": "checkout", "start_ms": 0, "end_ms": 120},
    {"span_id": "b", "parent_id": "a", "name": "pricing", "start_ms": 10, "end_ms": 50},
    {"span_id": "c", "parent_id": "a", "name": "inventory", "start_ms": 20, "end_ms": 50},
    {"span_id": "d", "parent_id": "a", "name": "payment", "start_ms": 60, "end_ms": 85},
]
print(self_times(spans)["a"])  # 55.0 (the naive sum of child durations gives 25)

A Python function that computes per-span self-time by clipping and merging direct-child intervals, applied to the assumed checkout timings from the illustrative estimate.

Aggregating module latency across traces and pairing it with profiling

Once each span carries a self-time value, group it by app.module across many traces. Report p50, p95 and p99 per module instead of means. An average blends a large population of fast requests with a few pathological ones, which hides exactly the behavior you are hunting.

The more useful comparison is between populations. Split traces into a typical set (around the median end-to-end latency) and a tail set (above p99). Then compute each module's share of total self-time in both. If billing accounts for a small share at the median and a much larger share in the tail, it is your tail driver, even if its own p50 looks harmless.

Sampling decides whether you can do this at all. Low-rate head sampling makes the keep-or-drop decision before latency is known, so it discards most of the rare slow requests. Tail sampling keeps them, but the collector must buffer every span of a trace until a decision window closes. That adds memory pressure, and in multi-collector setups it requires routing all spans of a trace to the same instance.

Spans tell you which module is slow. A sampling CPU or wall-clock profiler then tells you which code inside it. Wall-clock profiling captures off-CPU waits that CPU profiling misses. Profilers that tag samples with the active span ID let you filter to the slow traces directly.

Validating instrumentation and preventing boundary drift

Boundary spans are code, and code drifts. A refactor that inlines a service class or moves a call behind a new helper can silently delete the span that attributed its latency, and the dashboards won't show an error. They will just show the parent module getting slower.

Assert the span tree in a test

Configure the tracer provider in tests with an in-memory span exporter and a simple (synchronous) span processor, so finished spans are available as soon as the request returns. Drive one known code path, such as a synthetic checkout request against a seeded database. Then assert on the result:

• The expected parent-child structure, for example http.request → billing → pricing.
• The module attribute on each boundary span.
• That no boundary span is missing or duplicated.

Match on structure and attributes, not timings, so the test stays deterministic. When someone moves a boundary, CI fails and they have to update the expected tree on purpose.

Treat orphaned spans as propagation bugs

A boundary span with no parent, or with a new trace ID, usually means context was lost crossing a thread pool, executor, callback or async queue. Add an assertion that every span in the test trace shares one trace ID and has a parent that resolves within that trace. In production, report the orphan rate as a metric.

Give each boundary list an owner and enforce attribute names

Each module's boundary list needs a named owning team. Without one, nobody notices when a boundary stops making sense.

Attribute naming needs the same discipline. If one team emits module and another emits app.module.name, any aggregation that groups by module splits or drops data without warning. Define the attribute keys in one shared constants module, and fail the test when a span carries a module attribute that isn't in that list.

Copyfrom opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

from myapp.telemetry_conventions import MODULE_ATTR  # shared key, e.g. "app.module"
from myapp.testing import run_synthetic_checkout     # drives one known code path

EXPORTER = InMemorySpanExporter()
_provider = TracerProvider()
_provider.add_span_processor(SimpleSpanProcessor(EXPORTER))
trace.set_tracer_provider(_provider)  # global provider can only be set once per process

EXPECTED_TREE = {
    "billing": ("POST /checkout", "billing"),
    "pricing": ("billing", "pricing"),
}

def test_checkout_boundary_spans():
    EXPORTER.clear()
    run_synthetic_checkout(order_id="order-1001")  # seeded test data, no timing asserts
    spans = EXPORTER.get_finished_spans()
    by_id = {s.context.span_id: s for s in spans}

    # Orphan check: one trace, one root, every parent resolves inside this trace.
    assert len({s.context.trace_id for s in spans}) == 1
    assert len([s for s in spans if s.parent is None]) == 1
    assert all(s.parent.span_id in by_id for s in spans if s.parent is not None)

    # Boundary check: none missing, none duplicated.
    boundaries = [s for s in spans if s.name in EXPECTED_TREE]
    assert sorted(s.name for s in boundaries) == sorted(EXPECTED_TREE)
    for s in boundaries:
        parent_name, module = EXPECTED_TREE[s.name]
        assert by_id[s.parent.span_id].name == parent_name
        assert s.attributes.get(MODULE_ATTR) == module
        # Naming check: reject drifted keys such as "module" or "app.module.name".
        assert not {k for k in s.attributes if "module" in k} - {MODULE_ATTR}

A pytest test that captures spans with the OpenTelemetry SDK's InMemorySpanExporter and fails CI if boundary spans move, disappear, lose their parent context, or use a module attribute key other than the shared constant.

When to avoid it, and its impact on cost and engineering time

Module spans cost something to add and to keep, so check whether a cheaper tool already answers the question.

The cost comes in three parts. First, you write the boundary wrappers. Second, you maintain them as code moves between modules, using the drift tests described above. Third, you pay for storage, which grows with span volume:

Illustrative estimate (assumed numbers, not a benchmark) Extra spans/day = requests/day × module spans per request × sampling rate = 10,000,000 × 8 × 0.1 = 8,000,000 spans/day

Keep the module attribute to a small, fixed set of values. Putting user IDs or SQL text into span attributes inflates index cost far more than the added span count does.

The payoff is a shorter path from "this endpoint is slow" to "this module is slow." When the module attribute maps to an ownership file, alerts can route straight to the owning team. Nobody has to triage the trace by hand first.

There is also an organizational risk. A module's self-time includes time spent waiting on shared locks and connection pools, so it can look slow because of contention another module caused. Treat per-module dashboards as a shared diagnostic. Review regressions jointly before anyone files them against a team.

Alternatives to module boundary spans

Manual module spans are one option among several. Each alternative below answers part of the same question with a different trade-off:

• Sampling profilers (CPU or wall-clock). A profiler shows which functions consume time without any code changes at module boundaries. It works well for a small codebase or a single hot path. Its weakness is per-request attribution: a profile aggregates many requests, so it is hard to explain why one specific slow request was slow unless the profiler links samples to trace or span IDs.
• Profiling linked to traces. Some profilers can tag samples with the active trace or span ID. Combined with the auto-instrumented root span, this can narrow a slow trace down to functions without per-module spans. It depends on the profiler and language runtime supporting that linkage, and you still have to map functions to modules and owners yourself.
• Per-module latency metrics. Instead of spans, record a duration histogram per module entry point with the OpenTelemetry metrics API (or any metrics library), labelled with a fixed module name. Metrics are cheaper to store than spans and are not affected by trace sampling, so they suit dashboards and alerts. They lose the per-request tree, so they cannot show which module was slow inside a particular request, and they report total time rather than self-time.
• Structured timing logs. Logging start and end times at module entry points with a request or trace ID gives similar data to spans with tooling you may already have. Reconstructing nesting and computing self-time from logs is manual work, and log volume grows quickly.
• Agent-based method instrumentation. Some language agents can be configured to wrap selected methods in spans without changing source code. This avoids writing decorators, but the configuration lives outside the code, so it can drift silently when methods are renamed or moved, and the drift tests above become harder to write.
• Extracting a module into a separate service. Network boundaries are traced by auto-instrumentation, so a module running as its own service shows up as a client and server span automatically. This is an architectural decision with large costs of its own and should not be made only to gain visibility.

In practice these combine. A common split is metrics for alerting on per-module latency, boundary spans for attributing individual slow requests to modules, and a profiler for finding the code inside the module that is responsible.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to monolith module latency tracing, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.