What is a flame graph and how do you read it correctly?
A flame graph aggregates many sampled stack traces into one picture. A sampling profiler interrupts the process at a fixed rate, records the current call stack, and counts how often each unique stack appears. Each box is a function. The box below it is its caller, and the boxes stacked on top are its callees.
Width is the share of samples in which that function was on the stack. In the classic layout the x-axis is sorted alphabetically so identical frames merge. It is not a timeline, so left-to-right order tells you nothing about when something ran. Color is usually a random warm palette chosen for contrast. Some tools color by package or language, so check the legend before reading meaning into it.
The distinction that matters most is total time versus self time. Total time is the full width of a box. Self time is the part of that width not covered by children, meaning samples where the function itself was executing.
Worked example (illustrative, 1,000 samples)
| Frame | Total | Self |
|---|---|---|
handleRequest |
900 | 10 |
├ renderOrder |
600 | 40 |
│ └ serializeJSON |
560 | 560 |
└ loadCart |
290 | 290 |
handleRequest is the widest box, but optimizing its own code would save at most 1% of samples. The real cost sits in the leaf serializeJSON, which shows as a wide box with nothing above it. Look for wide plateaus at the top edge, not for wide boxes at the base.
Why use sampling profilers in production, and which profile type fits your symptom?
An instrumenting profiler hooks every function entry and exit. That gives exact call counts, but the cost grows with call volume, so a hot loop in a busy endpoint can slow down sharply under the profiler. A sampling profiler instead captures the current stack at a fixed frequency, such as 100 times per second. Overhead depends on that frequency and stack depth, not on how many calls your code makes. You can lower it until it is acceptable and still get a statistically useful picture of where time goes. Sampling cannot report exact call counts, but in production you want proportions rather than counts.
Choose the profile type based on the symptom, because each one measures a different resource:
| Profile type | What a sample means | Symptom it explains |
|---|---|---|
| CPU | Thread was executing on a core | CPU-bound endpoint: high CPU, latency scales with load |
| Wall-clock | Thread existed, running or waiting | I/O-bound endpoint: low CPU, high latency |
| Allocation | Bytes or objects allocated at this stack | GC pressure: frequent pauses, rising heap churn |
| Lock/contention | Time spent blocked acquiring a lock | Lock waits: throughput plateaus while cores sit idle |
The common mistake is opening a CPU profile for a slow endpoint that mostly waits. CPU profiles sample only on-CPU threads. A request blocked on a database query or a downstream HTTP call produces no samples while it waits, so the expensive part is simply missing from the graph. If CPU usage is low but latency is high, switch to wall-clock.
Why does a monolith need per-endpoint profile attribution?
In a monolith, every route runs through the same process, so a process-wide flame graph merges all of them into one picture. Shared code makes this worse. If /search, /export and /checkout all call the same ORM layer, JSON serializer and template renderer, those frames appear as one wide tower. You can see that serialization costs 30% of CPU, but not whether one export endpoint causes most of it or fifty routes each contribute a little. Those two cases call for different fixes, and the merged graph cannot tell them apart.
This matters most for backend and platform teams running large shared processes with dozens or hundreds of routes, where no single team owns the whole profile. When a CPU regression ships, the first question is "which route?", and a flat graph sends people reading call stacks instead of answering it.
Not every sample belongs to a request, though. A typical monolith also spends CPU on:
- Background work running in the same process, such as schedulers, queue consumers and cache warmers.
- Garbage collection and runtime threads, which run outside any request's context.
- Middleware for auth, logging and compression. These frames can be attributed to a request, but only if the label is set before they run.
- Connection pool maintenance, including health checks, TLS handshakes and reconnects.
Route labels close the main gap by tagging each request-handling sample with its endpoint, so you can filter or compare the graph per route. The unlabeled remainder is still useful. If it grows, the cost is coming from background work or the runtime, not from traffic.
How does continuous profiling with Pyroscope work?
Pyroscope separates capturing samples from storing and querying them. Collection stays cheap and local to the process. Retention, aggregation and analysis run elsewhere.
+---------------------------+
| Monolith process |
| +---------------------+ | push (SDK) +----------------+
| | Pyroscope SDK |--+----------------->| |
| | samples stacks, | | | Profile server |
| | attaches labels | | pull (agent) | stores, indexes|
| +---------------------+ |<-----------------| by service, |
| or a /debug profiling | scrapes | labels, time |
| endpoint | +-------+--------+
+---------------------------+ |
v
+----------------+
| Query UI |
| service + label|
| + time range |
+----------------+
There are two ways to get profiles to the server.
Push. A language SDK runs inside the application. It samples call stacks at a fixed rate and aggregates them in memory. Every few seconds it uploads a compact profile. The SDK runs in-process, so it can read request context and attach labels such as the route to each sample. That is why this tutorial uses push.
Pull. Some runtimes already expose a profiling endpoint, Go's pprof being the common case. A separate agent can scrape that endpoint on a schedule. This needs no code changes. Request-level labels are then limited to whatever the runtime records.
The server stores each profile with three things: its service name, its labels and its timestamp. A query names a service, a label filter and a time window. The server merges every matching profile into a single flame graph.
Because samples are kept continuously, you don't need to reproduce an incident to look at it. You query the window when it happened. You can also diff two windows or two label values, for example the period before a deploy against the period after it, or one route against another.
+--------------------------------+
| Monolith process |
| |
| +--------------------------+ |
| | Pyroscope SDK (push) | | push profiles
| | - samples call stacks |--+-------------------------+
| | - aggregates in memory | | every few seconds |
| | - adds labels (route) | | |
| +--------------------------+ | v
| | +---------------------+
| +--------------------------+ | scrape | Profile server |
| | Profiling endpoint |<-+--------+ | - stores profiles |
| | (e.g. Go pprof) | | | | - indexes by |
| +--------------------------+ | | | service, labels, |
+--------------------------------+ | | time |
| +---------------------+
+----------+--+ ^ |
| Agent (pull)|-------+ |
| scrapes on | forwards | query
| a schedule | profiles v
+-------------+ +---------------------+
| Query UI |
| service + labels |
| + time range |
| -> flame graph |
+---------------------+
Profiles reach the server either by an in-process SDK pushing labeled samples or by an agent scraping a runtime profiling endpoint; the server indexes them by service, labels and time for querying in the UI.
How do you tag profiles with normalized route labels?
Comparing one label value against another only works if the label has a small, stable set of values. For a monolith, the most useful label is the route template the router matched, such as /users/:id, together with the HTTP method.
Pyroscope's language SDKs expose a tag-wrapping call, named something like TagWrapper or tag_wrapper depending on the SDK; check your SDK's documentation for the exact signature. It attaches labels to every sample taken while a block of code runs. Put the call in middleware that runs after routing resolves, because the template doesn't exist until the router has matched the request. Read the template from your framework's matched-route object, never from the request path. Wrap the handler invocation itself, so that samples on that thread, goroutine or context carry the label. Give unmatched requests a fixed value such as unmatched, rather than falling back to the raw path.
The raw path is the wrong label because it embeds identifiers. /users/48213 and /users/48214 become separate series. That inflates storage and query cost, and it splits one endpoint's CPU across thousands of slivers, none of which looks hot. Query strings, user IDs, tenant IDs and request IDs fail the same way.
Keep sensitive data out of labels entirely. Labels are stored with the profiles and shown to anyone who can query them. Profiling data rarely gets the redaction, access controls or retention rules you apply to logs, so an email address or token in a label leaks quietly and stays.
# Sketch: Flask + Pyroscope Python SDK. Verify configure() and tag_wrapper()
# against the SDK version you install; signatures can differ between releases.
import pyroscope
from flask import Flask, request
pyroscope.configure(
application_name="shop-monolith", # placeholder service name
server_address="", # placeholder address
)
class TaggedFlask(Flask):
def dispatch_request(self):
# By this point the router has matched, so url_rule is populated.
rule = request.url_rule
# Use the template ("/users/<int:user_id>"), never request.path.
# Unmatched requests get one fixed value instead of the raw path.
route = rule.rule if rule is not None else "unmatched"
labels = {"route": route, "method": request.method}
# Every sample taken on this thread while the handler runs carries
# these labels. Do not add user IDs, query strings, tokens or emails.
with pyroscope.tag_wrapper(labels):
return super().dispatch_request()
app = TaggedFlask(__name__)
@app.get("/users/<int:user_id>")
def get_user(user_id):
# Labelled route="/users/<int:user_id>", so /users/48213 and
# /users/48214 land in the same series.
return {"id": user_id}
Flask middleware that labels Pyroscope samples with the matched route template and HTTP method after routing resolves, using a fixed value for unmatched requests.
How do you find slow endpoints and verify the fix?
Start from metrics, not profiles. Rank routes by total time consumed, which is p95 or p99 latency multiplied by request rate, and also by tail latency alone. A moderately slow route with heavy traffic often costs more CPU than a rare one that takes seconds, and the two need different fixes.
Narrow to one route and one window
- Filter the profile view by the route label for your shortlisted endpoint and set the time range to the window where latency was bad.
- Sort by self time, not total time. Wide frames near the root are mostly framework dispatch that every request passes through. The frames with the most self time are where the CPU actually spent its cycles.
- Follow those frames up to their callers until you reach the first function in your own code. That function is usually where you can change something, even when the hot frame is in a library such as a JSON encoder or regex engine.
Compare against a baseline
If your tooling offers a diff or comparison view, compare the slow window against a normal one, or the hour before a deploy against the hour after. Frames that grew stand out, and you don't have to remember what "normal" looked like.
A CPU profile won't show time spent blocked on a database or downstream call. Where your tooling can link spans or request IDs to profiles, use a slow trace to find where the wall-clock time went.
Verify
Reproduce the hot path under load, change the code, and confirm the frame has narrowed. Then check the route's latency metrics after deploy. A smaller frame proves you saved CPU. It doesn't prove users see faster responses.
#!/usr/bin/env bash
# Assumes a Go monolith exposing net/http/pprof on :6060 and handlers wrapped in
# pprof.Do(ctx, pprof.Labels("route", "/api/orders"), ...) so samples carry a route label.
set -euo pipefail
HOST=""
ROUTE="/api/orders" # shortlisted from latency x request-rate metrics
# 1. Capture a 30s CPU profile during the slow window.
curl -sS -o slow.pb.gz "$HOST/debug/pprof/profile?seconds=30"
# 2. Keep only samples for this route; -top sorts by flat (self) time by default.
go tool pprof -top -tagfocus="route=$ROUTE" slow.pb.gz | head -n 20
# 3. Walk up from a hot library frame to its callers to find the first function in your code.
go tool pprof -peek 'encoding/json\.' -tagfocus="route=$ROUTE" slow.pb.gz
# 4. After the fix is deployed (or for a normal-traffic window), capture again.
curl -sS -o fixed.pb.gz "$HOST/debug/pprof/profile?seconds=30"
# 5. Diff against the baseline: negative flat values mean the frame narrowed.
go tool pprof -top -diff_base=slow.pb.gz -tagfocus="route=$ROUTE" fixed.pb.gz | head -n 20
# Optional: open the diff as an interactive flame graph in the browser.
# go tool pprof -http=:8081 -diff_base=slow.pb.gz fixed.pb.gz
# A narrower frame shows CPU saved; still confirm p95/p99 for $ROUTE dropped in your metrics.
Uses Go's pprof to filter a CPU profile to one labeled route, rank frames by self time, trace a hot library frame back to its callers, and diff the profile against a baseline to confirm the frame narrowed.
When should you avoid continuous profiling, and what are the alternatives?
Continuous profiling is cheap, but it still has a cost. Overhead rises with higher sample rates, with wall-clock and allocation profiles, and with label cardinality. Before rolling it out, compare CPU and p99 latency on your own service with the agent on and off.
Sampling has blind spots. A rare 50 ms stall barely registers, and a ten-second spike disappears when you view a one-hour window. Stacks also break when JIT-compiled code, inlining, stripped binaries, missing frame pointers or async schedulers hide the real caller.
Profiles reveal function names, and labels can reveal routes or tenants, so restrict access to them the same way you restrict logs. You also take on another agent and another backend that someone has to upgrade, monitor and keep running.
| Approach | Best for | Blind spot |
|---|---|---|
| On-demand profiling | Reproducible issues you can trigger now | Anything that already happened |
| Distributed tracing | Finding which service or span is slow | Code-level CPU detail inside a span |
| APM-style route metrics | Detecting which endpoint regressed | Why it regressed |
| Load-test profiling | Catching hot paths before release | Production data shapes and traffic mix |
Continuous profiling pays off when it replaces repeated guess-and-check debugging. Illustrative estimate, assumed inputs:
- Two slow-endpoint investigations per month.
- Each takes three engineer-days without profiles and one with them.
- Running the profiling stack costs one engineer-day per month.
Under those assumptions, the net saving is about three engineer-days a month. If you hit this kind of problem once a year, on-demand profiling plus route metrics is probably enough.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to production profiling flame graphs, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.