Monolith Performance: Find and Fix the Real Bottleneck First

What monolith performance work is and when to do it

Monolith performance work means finding the few code paths that use most of your latency budget and fixing those. It does not mean adjusting every setting you can find. In most slow monoliths, a handful of endpoints or background jobs cause the bulk of user-visible slowness. Effort spent anywhere else barely changes the result. This guide is for backend and platform engineers who own a monolith that has become slow and need to know where to start.

Start when a measurable target is missed. Examples: p95 or p99 latency on an endpoint goes over its objective, throughput falls short of demand, or a resource such as CPU, memory, or database connections stays close to saturation. "The app feels slow" is not a target. Turn it into a problem statement you can test:

(Figures are illustrative.)

Before you change anything, record a baseline: the percentiles, the request rate, resource usage, and the time window you measured. Without it you can't show whether a fix helped.

Measure percentiles, not averages. Suppose 95 requests take 50 ms and 5 take 3 s. The average is about 200 ms, which looks fine, yet one user in twenty waits three seconds. Slow requests also tend to hold connections and locks longer than fast ones, so the tail is where saturation starts. If you optimize the average, you can make the common case faster while leaving the slow requests unchanged.

What is it?

Bottleneck-first performance work is a method for making a single deployable application faster by measuring where time goes and fixing the largest contributor before anything else. It is not a product or a framework. It is a disciplined loop built from tools most teams already have: request tracing, a sampling profiler, the database's query planner, and a load test. The output is a small number of targeted changes, such as a removed N+1, an added index, or a narrowed query. Each change is backed by a before-and-after measurement and protected by a test or alert so it does not regress.

Who should use it?

• Backend engineers who own the endpoints and jobs that miss their latency or throughput targets and need to know which code to change.
• Platform engineers and SREs who see saturated CPU, memory, or connection pools and need to tie that saturation back to specific code paths.
• Tech leads and architects who are being asked whether to split the monolith. Profiling data shows whether the problem is a few hot paths, which targeted fixes resolve, or real contention between domains, which may justify extraction.

The approach assumes you can get per-request timing data and run code against production-like data volumes. If you can't do either yet, build that observability first. Everything below depends on it.

When should it be avoided?

This approach is the wrong tool, or should wait, in these situations:

• There is no measurable target. If no endpoint misses an objective and no resource is near saturation, optimization adds risk and complexity without a goal to judge it against.
• You cannot measure. Without traces, profiles, or production-like data, you will be guessing. Invest in instrumentation before you tune.
• You are in an active incident. When users are failing right now, scale up, shed load, or roll back first. Do the root-cause analysis once the system is stable.
• The code is about to be retired or rewritten. Deep optimization of a path that will be deleted soon rarely pays back. A minimal fix or temporary scaling is usually enough.
• The slowness is outside the backend. If the time goes to client-side rendering, large frontend bundles, or network distance to users, server-side profiling will not find it.
• The bottleneck is already known to be structural. If earlier profiling showed that targeted fixes are exhausted and domains contend for the same resources, more local tuning will not help. The decision is architectural.

How does it work?

The method is a loop that narrows from the whole system down to one line of code, then widens back out to confirm the result:

1. Define the target and record a baseline. Write a measurable problem statement and capture percentiles, request rate, and resource usage under a known load.
2. Rank by total time. Use traces to order endpoints and jobs by tail latency multiplied by call volume, so you work on what users actually feel.
3. Split the slowest request into time buckets. Separate database time, external calls, CPU work, and waits on locks, pools, or queues. The largest bucket picks the next tool.
4. Drill down with the matching tool. Use query counts and EXPLAIN ANALYZE for database time, a sampling profiler for CPU and in-process waits, and trace signatures for external calls, contention, and GC.
5. Fix the single largest cause. Prefer the narrowest change that removes it. Add a cache or move work off the request path only after the underlying cost is understood.
6. Re-measure against the baseline. Use the same load profile and change one thing at a time.
7. Guard the gain. Add query-count assertions, latency budgets, and percentile alerts, then return to step 2 if the target is still missed.

The sections below cover each step in detail.

How to find the bottleneck: tracing and profiling top-down

Start at the request level. Use distributed tracing or per-request spans to rank endpoints by total time consumed, which is p95 or p99 latency multiplied by call volume. A slow endpoint that runs ten times a day matters less than a moderately slow one that serves most of your traffic. For each endpoint near the top of that list, break a representative slow trace into four buckets: database time, external calls, CPU work, and time spent waiting on locks, thread pools or queues. The largest bucket tells you which tool to use next.

When CPU or unexplained in-process time dominates, use a profiler:

Choose the time mode on purpose. CPU time shows computation. Wall-clock time also includes I/O and lock waits, which is usually where a monolith's latency hides. In a flame graph, width is the share of samples and vertical stacking is call depth. Look for wide plateaus near the top of the graph, because those frames are burning time themselves rather than delegating it to callees.

Profile under realistic data volume and concurrency. Ideally, run continuous low-overhead sampling in production. A staging database with a thousand rows hides missing indexes and contention. An instrumenting profiler adds a fixed cost to every call, so cheap, frequently called functions look expensive. Either one can send you after the wrong bottleneck.

Copy+-------------------------------------------------------------+
| 1. Traces / per-request spans                               |
| rank endpoints by p95/p99 latency x call volume             |
+------------------------------+------------------------------+
                               |
                               v
+-------------------------------------------------------------+
| 2. Split a slow trace into time buckets                     |
+-------+---------------+---------------+----------------+----+
        |               |               |                |
        v               v               v                v
  +-----------+   +-----------+   +-----------+   +-------------+
  | Database  |   | External  |   | CPU work  |   | Lock, pool, |
  | time      |   | calls     |   | in-proc   |   | queue waits |
  +-----------+   +-----------+   +-----+-----+   +------+------+
                                        |                |
                                        v                v
                              +--------------------------------------------+
                              | 3. Low-overhead sampling profiler          |
                              | under realistic data and concurrency       |
                              | CPU time   -> computation only             |
                              | Wall clock -> also I/O and lock waits      |
                              +---------------------+----------------------+
                                                    |
                                                    v
                              +--------------------------------------------+
                              | 4. Flame graph: width = share of samples   |
                              | wide plateaus at top = self time (hot)     |
                              +--------------------------------------------+

Top-down bottleneck hunting: rank endpoints by total time from traces, split a slow request into time buckets, then profile CPU and wait time with a sampling profiler under realistic load.

Detecting and fixing N+1 queries

An N+1 is the most common bottleneck that a trace exposes clearly. Code loads a list of parents, then each child access issues its own query: one query for 50 orders, then 50 more for their customers. Each query is fast on its own, so none of them shows up in a slow-query log. The cost comes from round trips, and it grows with the size of the page.

Detecting it

Count queries per request. Most ORMs and database drivers expose a hook or event you can use to increment a counter. In development, log every query with the endpoint that issued it. In production traces, look for many sibling spans with the same query shape that differ only in a bound parameter. One query repeated 200 times inside a single request is the signature.

Fixing it

• Eager loading: tell the ORM to fetch the association up front, either as a join or as a second query.
• Batched IN queries: collect the parent IDs and fetch all children in one query.
• Dataloader-style batching: for resolvers that run independently, as in GraphQL, defer the lookups and coalesce them per tick or per request.

Then add a test that asserts the query count for a representative request. Seed more than one record, or the N+1 will stay hidden. The test should fail when the count rises, so a regression shows up in review instead of in production.

Don't overcorrect

Eager loading a whole object graph, or joining five tables, swaps many small queries for one that returns a huge or duplicated result set. Fetch only the columns and associations the view actually renders.

Copyfrom contextlib import contextmanager
from sqlalchemy import event, select
from sqlalchemy.orm import selectinload

from myapp.models import Order, Customer  # placeholder models: Order.customer -> Customer

@contextmanager
def count_queries(engine):
    """Count every SQL statement sent to the database inside the block."""
    counter = {"n": 0}
    def on_execute(conn, cursor, statement, params, context, executemany):
        counter["n"] += 1
    event.listen(engine, "before_cursor_execute", on_execute)
    try:
        yield counter
    finally:
        event.remove(engine, "before_cursor_execute", on_execute)

def list_orders(session):
    # Without selectinload, each o.customer access lazy-loads: 1 + N queries.
    # selectinload issues one extra SELECT ... WHERE customers.id IN (...).
    # load_only keeps the batch query to the columns the view renders.
    stmt = (
        select(Order)
        .options(selectinload(Order.customer).load_only(Customer.name))
        .limit(50)
    )
    return [(o.id, o.customer.name) for o in session.scalars(stmt)]

def test_order_list_query_count(seeded_session):
    # The fixture seeds several orders across several customers. With a single
    # row, an N+1 issues the same number of queries as the fix and stays hidden.
    with count_queries(seeded_session.get_bind()) as q:
        rows = list_orders(seeded_session)
    assert len(rows) > 1
    # 1 query for orders + 1 batched IN query for customers.
    # Fails in review if someone reintroduces a lazy load.
    assert q["n"] == 2

A SQLAlchemy event hook counts queries per block, selectinload with load_only replaces per-row customer lookups with one narrow IN query, and a test pins the query count so an N+1 regression fails the build.

Database analysis: query plans, indexes, pagination and connection pools

Once a slow endpoint is down to a few queries, run each one under EXPLAIN ANALYZE against production-like data. Plain EXPLAIN shows the planner's estimates. ANALYZE executes the query and reports actual rows and timing. The most useful signal is the gap between them: a node estimated at 10 rows that returns 200,000 usually points to stale statistics or a skewed column, and the planner chose the wrong join or scan because of it. Look for sequential scans on large tables behind selective filters, sorts that spill to disk, and nested loops over large inputs.

Index the columns your hot queries filter, join, and sort on, in that composite order. Then check your database's index-usage statistics for indexes that are never read. Every index adds cost to each insert and update and takes space, so indexes added reactively during incidents tend to accumulate into bloat that slows writes.

SELECT * on wide rows drags large text or JSON columns across the network and blocks index-only scans. Select only the columns you need.

Large OFFSET pagination still reads and discards every skipped row. Keyset pagination (WHERE id > :last_seen ORDER BY id LIMIT n) seeks straight to the next page through the index.

Size connection pools against what the database can actually run in parallel, which is bounded by CPU cores and I/O, not by max_connections. Total connections equal instances × pool size. Illustrative estimate: 10 instances with 20 connections each is 200 connections competing for perhaps 16 cores. More connections or more instances past that point only move the queueing into the database, where lock and context-switch contention lower throughput.

Copy-- PostgreSQL. Run against production-like data, not an empty dev schema.

-- 1. Compare estimated vs actual rows. BUFFERS shows how many pages were read.
--    ANALYZE really executes the query, so wrap writes in BEGIN ... ROLLBACK.
EXPLAIN (ANALYZE, BUFFERS)
SELECT id, status, created_at          -- only the columns the endpoint needs
FROM orders
WHERE customer_id = 4821 AND status = 'open'
ORDER BY created_at DESC, id DESC
LIMIT 50;

-- 2. Composite index in filter-then-sort order. This lets the planner seek
--    and read rows already sorted. CONCURRENTLY avoids blocking writes.
CREATE INDEX CONCURRENTLY idx_orders_customer_status_created
  ON orders (customer_id, status, created_at DESC, id DESC);

-- 3. Find indexes that have never been scanned since stats were last reset.
--    Skip unique indexes because they enforce constraints even if never read.
SELECT s.relname AS table_name,
       s.indexrelname AS index_name,
       s.idx_scan,
       pg_size_pretty(pg_relation_size(s.indexrelid)) AS index_size
FROM pg_stat_user_indexes s
JOIN pg_index i ON i.indexrelid = s.indexrelid
WHERE s.idx_scan = 0 AND NOT i.indisunique
ORDER BY pg_relation_size(s.indexrelid) DESC;

-- 4. Keyset pagination: pass the last row's (created_at, id) from the previous
--    page instead of OFFSET. The row comparison seeks via the index above.
SELECT id, status, created_at
FROM orders
WHERE customer_id = 4821 AND status = 'open'
  AND (created_at, id) < ('2026-10-01 12:00:00+00', 987654)
ORDER BY created_at DESC, id DESC
LIMIT 50;

A PostgreSQL workflow that runs EXPLAIN (ANALYZE, BUFFERS) on a narrow query, adds a composite index in filter-then-sort order, lists never-scanned non-unique indexes from pg_stat_user_indexes, and replaces OFFSET with keyset pagination on (created_at, id).

Bottlenecks outside the database

When the database is healthy and latency is still poor, the cause is usually one of five patterns. Each leaves a distinct signature.

Contention and GC pauses mostly affect p99 because they are intermittent. Most requests never wait. The unlucky few queue behind a lock holder or land during a pause, so averages look fine while the tail grows.

Moving work out of the request path

If a result does not have to be computed before the response, compute it elsewhere. Send emails and webhooks from background jobs. Maintain precomputed aggregates instead of summing rows on every read. Send read-heavy reporting to a read replica.

Each of these trades latency for complexity:

• Eventual consistency. Users can read stale aggregates or replica data that lags behind the primary.
• Retries. Jobs fail and run again, so every handler must be idempotent. Key the side effects so that a second run does nothing.
• Monitoring. Track queue depth, job age and the dead-letter queue. A growing dead-letter queue means work is being lost without any error reaching the user.

Choosing a caching strategy deliberately

Add a cache only after you have fixed what profiling exposed. A cache placed over a slow query hides that query from your profiler. When the cache is cold, flushed or bypassed, the slow query comes back at full cost.

Base TTL and invalidation on how stale each piece of data may be. A product catalogue can be minutes old. An account balance cannot be stale at all, so do not cache it.

Most teams start with cache-aside. Read the key, for example user:42:profile:v3. On a miss, load from the database, store the result with a TTL, and return it. On a write, delete the key rather than updating it, so two concurrent writers cannot leave an old value in the cache.

• Read-through moves the loading logic into the cache layer.
• Write-through updates the cache and the store together. Reads stay fresh, but every write becomes slower.

The version suffix in the key lets you invalidate every entry at once when the object's shape changes.

Stampedes occur when many requests miss the same key at the same moment, which happens after a deploy flushes the cache or when keys expire together. Two mitigations:

• Add random jitter to each TTL so keys do not expire in sync.
• Use request coalescing, so one caller rebuilds the value while the others wait for its result.

Copyimport json, random, time
import redis

r = redis.Redis(host="localhost", port=6379)
BASE_TTL = 300  # this data may be up to ~5 minutes stale

def profile_key(user_id):
    # Bump v3 -> v4 when the cached shape changes; old entries just age out.
    return f"user:{user_id}:profile:v3"

def get_profile(user_id, db):
    key = profile_key(user_id)
    cached = r.get(key)
    if cached is not None:
        return json.loads(cached)
    # Coalesce: only the caller that wins this short lock rebuilds the value.
    lock = key + ":lock"
    if r.set(lock, "1", nx=True, ex=10):
        try:
            profile = db.load_profile(user_id)
            # Jitter keeps keys written together from expiring together.
            ttl = BASE_TTL + random.randint(0, 60)
            r.set(key, json.dumps(profile), ex=ttl)
            return profile
        finally:
            r.delete(lock)
    # Losers wait briefly for the winner's result, then fall back to the DB.
    for _ in range(20):
        time.sleep(0.05)
        cached = r.get(key)
        if cached is not None:
            return json.loads(cached)
    return db.load_profile(user_id)

def update_profile(user_id, changes, db):
    db.save_profile(user_id, changes)
    # Delete instead of overwrite so a racing writer cannot leave an old value cached.
    r.delete(profile_key(user_id))

A cache-aside read path using redis-py, with a versioned key, a jittered TTL, a simple lock-based request coalescer, and delete-on-write invalidation.

Verifying fixes, alternatives and the impact on engineering cost

Re-run the baseline load profile after every change, with the same request mix, data volume and concurrency. Change one thing at a time. If an index and a cache ship together, you cannot tell which one moved p99, or whether one hid a regression in the other.

Then add guards so the gain survives future changes:

• Query-count assertions in tests for hot endpoints, so a new N+1 fails the build.
• Latency budgets enforced by a load test in CI or before each release.
• Alerts on the percentiles you targeted, not on averages, which hide tail regressions.

Use profiling data to decide when an architectural change is justified. If one module's load still contends with everything else after you have fixed its queries and indexes, the problem is structural. If a single span dominates, it is not.

Illustrative estimate (assumed numbers, not a benchmark): Cost = engineers × weeks × weekly cost per engineer. Assume a targeted fix takes 1 engineer for 1 week, and a service extraction takes 3 engineers for 8 weeks. The extraction costs 24 times as much engineering time and leaves you with more systems to operate.

Measurement also prevents a smaller kind of waste. Micro-optimizations made without measurement make code harder to read and usually do not improve the latency users see.

How can it save engineering time?

Bottleneck-first work saves time mainly by stopping effort that would not have moved the target:

• It cuts out guesswork. Ranking by total time and splitting traces into buckets tells you which subsystem to examine before anyone opens a config file. Teams don't spend days tuning thread pools, JVM flags, or ORM settings when the time actually goes to one missing index.
• It prevents unnecessary rewrites. The profile shows whether a targeted fix is enough. When one span dominates, a change measured in days replaces a proposed extraction measured in months, as the illustrative estimate above shows.
• It shortens verification. A recorded baseline and one change at a time mean each result is unambiguous. You don't have to untangle which of several simultaneous changes helped or hurt.
• It moves regressions earlier. Query-count tests and latency budgets catch a new N+1 or a slow query in review or CI. Fixing it there is far cheaper than diagnosing it later from a production incident with on-call engineers involved.
• It keeps caches honest. Fixing the underlying query before you cache it avoids the recurring work of investigating latency spikes every time the cache goes cold.
• It produces reusable assets. The baseline load profile, the query-count hooks, and the percentile alerts carry over to the next performance problem, so each investigation starts further along than the last.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to monolith performance, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.