Continuous Profiling in Production: Finding Hot Modules in a Monolith

Why is profiling a monolith harder than profiling a microservice?

In a microservice, the process boundary already answers the ownership question. If the checkout service burns CPU, the checkout team gets paged, and the per-service dashboard shows the cost. In a monolith, the billing, search, notifications and admin teams all ship code into one binary, one deployment and one set of host metrics. When CPU climbs on that fleet, the dashboard tells you that the process is hot, not whose code made it hot.

Profilers help, but a naive look at a monolith's profile is misleading. Most stack frames belong to shared infrastructure such as the HTTP router, middleware chains, the ORM, serialization, the logging library and the runtime's garbage collector. These frames sit at the top of the CPU ranking because every request passes through them. The business module that triggered the work is usually several frames deeper, or is mixed with other callers that use the same ORM query builder. A profile that says "40% of CPU is in JSON encoding" does not tell you which team's endpoint is encoding oversized payloads.

This article is for backend and platform engineers who run a shared codebase with several owning teams and need to route performance problems to the right people. The approach treats continuous profiling in production as a data source rather than a picture. A flame graph is useful when you can filter it by module or endpoint, compare it against a baseline, and attach the result to an owner.

How does continuous profiling work in production?

A continuous profiler periodically interrupts the process and records the current call stack of each running thread, typically on the order of a hundred times per second. It does not trace every function call. Each sample is a stack plus a count. The agent batches samples for a short window, often around ten seconds, attaches labels such as service, host, version, and region, and ships the batch to a profile store. The store merges stacks across windows, so you can ask for "all CPU time for /checkout between 14:00 and 14:15" and get one aggregated flame graph.

Copy+-------------------+     samples      +----------------+
|  App process      | ---------------> |  Profile store |
|  (monolith)       |   + labels       |  (aggregates   |
|  [in-proc SDK]    |                  |  by window)    |
+-------------------+                  +-------+--------+
        ^                                      |
        | stack walks                          v
+-------------------+               +---------------------+
|  eBPF agent       |               |  Query / UI         |
|  (host-level)     |               |  filter: service=,  |
+-------------------+               |  endpoint=, version=|
                                    +---------------------+

There are two ways to collect the samples. An eBPF-based agent runs once per host and walks stacks from the kernel. It needs no code changes, but it needs privileges, and it can struggle to symbolize interpreted or JIT-compiled runtimes. An in-process SDK, such as Pyroscope's language integrations, runs inside the application. It can see runtime-level frames and attach request-scoped labels, but it adds a dependency to every deploy.

Sampling is what makes it affordable to leave profiling on all the time. The work scales with the sampling rate, not with request volume. Overhead is still not zero. It depends on the runtime, the stack depth, and the collection mechanism, so measure it on your own workload before you trust it.

Copy+--------------------------- Host ---------------------------+
|                                                            |
|  +----------------------+        +----------------------+  |
|  | A: App process       |        | B: eBPF agent        |  |
|  | (monolith)           |<-------| (one per host)       |  |
|  |                      | stack  | walks stacks from    |  |
|  | +------------------+ | walks  | the kernel; no code  |  |
|  | | In-process SDK   | |        | changes; needs privs |  |
|  | | ~100 samples/sec | |        +----------+-----------+  |
|  | | + request labels | |                   |              |
|  | +--------+---------+ |                   |              |
|  +----------|-----------+                   |              |
+-------------|-------------------------------|--------------+
              | batch ~10s: stacks + counts   |
              | + labels: service, host,      |
              |   version, region             |
              v                               v
       +---------------------------------------------+
       | Profile store                               |
       | merges stacks across windows by label set   |
       +----------------------+----------------------+
                              |
                              v
       +---------------------------------------------+
       | Query / UI                                  |
       | filter: service= endpoint= version=         |
       | /checkout, 14:00-14:15 -> one flame graph   |
       +---------------------------------------------+

Option A, an in-process SDK, or option B, a host-level eBPF agent, samples stacks at low frequency and ships labeled batches to a profile store, which merges them so the query layer can return one flame graph per label filter and time window.

How do you read a flame graph to find CPU hotspots?

Each box in a flame graph is a stack frame, and its width is the share of samples in which that frame was on the stack. Width is not wall-clock duration, and the x-axis is not a timeline. Frames at the same level are usually sorted alphabetically or merged, so left-to-right position tells you nothing. The y-axis is call depth: callers sit below, callees above.

A wide frame with narrow children is the CPU hotspot you are looking for. A wide frame topped by a wide child is only passing cost upward.

In a monolith, the bottom of most stacks is the same: the HTTP server, middleware, ORM, and dispatch layers. Skip past that tower and look for wide plateaus higher up. Then find the first frame from your own code beneath each plateau, such as InvoiceRenderer.render sitting under a JSON serializer. That business frame tells you which module is responsible.

Frame quality can mislead you:

• Stripped binaries show bare addresses.
• JIT-compiled code needs runtime symbol maps.
• Inlining folds callees into their callers, which inflates self time.
• Missing frame pointers break stack unwinding and produce shallow or truncated stacks.
• Large "unknown" frames mean symbolization failed. Fix symbolization before you draw conclusions.

An aggregated flame graph shows average cost. A rare slow path that drives p99 latency can occupy a sliver too thin to notice. To find it, filter to the slow requests or compare time windows.

How do you attribute profile samples to owning modules and request context?

That filtering depends on two layers of attribution: mapping frames to modules, and tagging samples with request context.

Map frames to modules

A stack frame carries a function name such as billing.invoices.render_pdf or com.acme.search.Indexer.rank. Map those package or namespace prefixes to modules with a longest-prefix-match table, so that billing.invoices overrides billing. Do not maintain the table by hand, because it will fall out of date with the codebase. Generate it in CI from the file that already records code ownership. When the code moves, the attribution moves with it.

Label samples with context

Pyroscope-style labels attach key-value pairs to every sample collected while a block of code runs. Wrapping a handler or job in a labeled scope lets you filter one process profile into slices, such as endpoint=/checkout or job=nightly_export. The example below shows the pattern. Label APIs differ between language SDKs, so confirm the exact calls in the docs for yours.

Each unique combination of labels creates a separate series to store and query, so keep cardinality bounded:

Watch for attribution drift

Shared utilities such as JSON serializers, ORM layers and loggers appear under every caller. A sample is charged to whichever caller was on the stack when it was taken. A spike in utils.serialize therefore says nothing about who caused it. Group by the first owned frame above the shared code, not by the leaf frame.

Copyimport pyroscope

# Configure once at process startup. Static tags apply to every sample.
pyroscope.configure(
    application_name="shop.monolith",
    server_address="",
    tags={"region": "eu-west-1"},
)


def handle_checkout(request):
    # Use the route template, never the raw URL or a user ID.
    # Every sample taken inside this block carries these labels.
    with pyroscope.tag_wrapper({
        "endpoint": "/checkout",
        "module": "billing",
        "tenant_tier": request.tenant_tier,  # e.g. free / pro / enterprise
    }):
        return render_checkout(request)


def run_job(job_type, payload):
    # job_type comes from a fixed set such as "nightly_export".
    with pyroscope.tag_wrapper({"job": job_type}):
        return JOB_HANDLERS[job_type](payload)

Wrapping a request handler and a background job with bounded-cardinality labels using the Pyroscope Python SDK's tag_wrapper. Other language SDKs use different label calls, so check the docs for yours.

Worked example: from CPU alert to a ticket for the owning team

Illustrative estimate. Assumed inputs: 40 instances with 4 vCPUs each, steady-state CPU around 55%, and a deploy at 14:00.

At 14:20 a saturation alert fires: fleet CPU has reached 78% and p99 latency on checkout endpoints has doubled. Narrow the profile window to 14:00–14:20 and filter by the service_version label for the new build. Grouped by first owned frame, pricing.rules holds 27% of CPU samples.

That number alone does not prove a regression. The module might always be expensive. Diff it against 13:30–13:50 on the previous version, where pricing.rules held 6%. The diff narrows the growth to one new path: pricing.rules.evaluate now calls compile_pattern on every request, where it used to compile once.

The arithmetic shows why this frame is worth fixing:

• Before the deploy: 6% × 55% ≈ 3.3 utilisation points.
• After the deploy: 27% × 78% ≈ 21 utilisation points.

That is roughly 18 of the 23-point rise, or about 28 of the fleet's 160 vCPUs. The change drives both the saturation and the latency, so it clears the bar. A frame that grew from 0.5% to 1% would not.

The ticket goes straight to the pricing team. It includes:

• The two time windows and the label filters used
• The diff view showing the new path
• The deploy identifier
• The utilisation arithmetic above
• The suspected call site

Without this, the usual path is a shared incident channel. Checkout, platform and pricing engineers each rule out their own code, which can take hours across several teams. Here, one engineer narrows it down in under an hour, and only the team that owns the change gets pulled in.

When should you avoid continuous profiling, and what are the alternatives?

…affected code gets the ticket. That speed depends on the problem being CPU-bound, which is not always true.

Continuous profilers sample stacks at a fixed rate, so the results are statistical. A function that runs for a few milliseconds once an hour, such as a nightly reconciliation job or a cold-cache rebuild, may never appear in a sample. CPU profiles also record a stack only while a thread is on-CPU. A request blocked on a database query, a mutex or a slow downstream call adds nothing to the profile, even if it dominates latency. Profiles are aggregated across all requests, so a p99 regression that affects 1% of traffic gets diluted into the average and does not stand out in the flame graph.

Skip continuous profiling for low-traffic services. They produce too few samples to be meaningful, and their CPU bill is too small to justify the work. Also skip it when no team has capacity to act on the findings, because unread flame graphs only add storage cost. For I/O-bound monoliths, start with tracing and off-CPU profiling.

How do you roll out continuous profiling without raising cost or risk?

Enable CPU profiling first on a subset of instances, such as one canary pool or a small slice of each deployment group. If the agent misbehaves, only a fraction of traffic is affected. Over the same time window, compare p99 latency and CPU utilisation between profiled and unprofiled instances. That difference is your overhead figure. It depends on your runtime, sampling rate and workload, so a generic published number does not replace it. Widen the rollout only after the difference stays stable.

Add memory or allocation profiles next, where the runtime supports them. They catch allocation churn that appears as garbage-collection time rather than as an obvious CPU hotspot. Some allocation modes cost more than CPU sampling, so measure them separately.

Keep label sets bounded. Module, endpoint and region are safe labels. User IDs, request IDs and raw URLs are not, because the number of stored series grows with the product of distinct values per label.

Illustrative estimate (assumed numbers, not a benchmark): series ≈ modules × endpoints × regions = 40 × 60 × 3 = 7,200. Adding a tenant label with 10,000 values gives 72,000,000.

Set retention and access control for profiles as you would for logs. Function names, file paths and label values can expose customer names, internal hostnames or unreleased features.

Budget ongoing time for the module mapping, because it drifts as code moves. Give it an owner and review the share of unmapped samples on a regular schedule. Finally, agree that a wide frame alone does not justify a ticket. Optimise only when the hotspot is tied to an SLO or a measurable infrastructure cost.

Copyimport statistics

# Labels allowed on profiles. Unbounded values (user IDs, request IDs, raw URLs) stay out.
ALLOWED_LABELS = {"module", "endpoint", "region"}
MAX_SERIES = 50_000  # budget agreed with whoever pays for profile storage

def p99(latencies_ms):
    # quantiles(n=100) returns 99 cut points; the last one is p99.
    return statistics.quantiles(latencies_ms, n=100)[-1]

def overhead_report(profiled, unprofiled):
    # Both samples must come from the same time window and the same deployment group.
    p_prof, p_base = p99(profiled["latency_ms"]), p99(unprofiled["latency_ms"])
    cpu_prof = statistics.mean(profiled["cpu_util"])
    cpu_base = statistics.mean(unprofiled["cpu_util"])
    return {
        "p99_delta_pct": 100 * (p_prof - p_base) / p_base,
        "cpu_delta_pct_points": 100 * (cpu_prof - cpu_base),
    }

def check_labels(distinct_values):
    # distinct_values maps label name -> number of distinct values seen.
    unexpected = set(distinct_values) - ALLOWED_LABELS
    if unexpected:
        raise ValueError(f"labels not on allow-list: {sorted(unexpected)}")
    series = 1
    for count in distinct_values.values():
        series *= count  # stored series grow with the product of distinct values
    if series > MAX_SERIES:
        raise ValueError(f"{series} series exceeds budget of {MAX_SERIES}")
    return series

# Placeholder counts matching the illustrative estimate in the text.
print(check_labels({"module": 40, "endpoint": 60, "region": 3}))  # 7200
# check_labels({"module": 40, "endpoint": 60, "region": 3, "tenant": 10_000})  -> ValueError

A standard-library Python check that computes profiling overhead as the p99 latency and CPU difference between profiled and unprofiled instances, and rejects label sets that are off the allow-list or exceed a series budget.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to continuous profiling production, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.