Where does a monolith CI pipeline actually spend its time?

This guide is for backend and platform engineers who own one large deployable and watch feedback loops stretch past the point where developers wait for results. Before you change anything, find out where the minutes go. A pipeline's critical path usually breaks into six stages:

Stage What it covers Common cause of slowness
Queue Waiting for a free runner Too few runners at peak hours
Environment setup Booting the runner, pulling base images Cold images, large base layers
Dependency fetch Downloading packages and modules No cache, or cache keys that change too often
Compile Building the application Full rebuilds of unchanged code
Test Unit, integration, end-to-end suites Serial execution, slow fixtures
Packaging Building and pushing the artifact Poor layer ordering, large uploads

Record wall-clock time per stage on every run, then compare p50 and p95 across at least a few weeks of builds. A single build misleads you. It might have hit a warm cache, an idle queue or a flaky retry. p50 shows the typical cost. p95 shows the tail that blocks merges, such as queue spikes and cache misses.

Slow CI costs more than the wait itself. A developer who switches tasks during a 25-minute build has to rebuild context when the build finishes. Merges that queue behind each other compound the delay.

Illustrative estimate (assumed numbers, not a benchmark): Daily waiting time = engineers × pushes per engineer per day × p50 pipeline minutes = 40 × 4 × 25 = 4,000 minutes, or about 67 engineer-hours per day.

Not all of that time is lost, because some of it overlaps with other work. Treat the result as an upper bound, and plug in your own numbers.

 git push
    |
    v
+-------+   +-------+   +-------+   +-------+   +-------+   +-------+
| Queue |-->| Env   |-->| Deps  |-->|Compile|-->| Test  |-->|Package|
|       |   | setup |   | fetch |   |       |   |       |   |       |
+-------+   +-------+   +-------+   +-------+   +-------+   +-------+
    |           |           |           |           |           |
    | t_queue   | t_setup   | t_deps    | t_build   | t_test    | t_pkg
    v           v           v           v           v           v
+-------------------------------------------------------------------+
|  Per-stage wall-clock timer (recorded on every run)               |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|  Timing store: run 1, run 2, ... run N  (several weeks of builds) |
+-------------------------------------------------------------------+
            |                                     |
            v                                     v
+-----------------------------+       +-----------------------------+
| p50 per stage               |       | p95 per stage               |
| typical cost                |       | tail: queue spikes,         |
|                             |       | cache misses, retries       |
+-----------------------------+       +-----------------------------+
            |                                     |
            +------------------+------------------+
                               v
            +-------------------------------------+
            | Fix the stage with worst p50/p95    |
            +-------------------------------------+

The CI critical path split into six timed stages, with each stage's wall-clock time recorded on every run and aggregated over weeks into p50 and p95 values that show which stage to fix first.

Who should use these techniques?

The techniques in this guide fit teams that ship one large deployable from one repository and want shorter feedback loops without restructuring the application. Typical signs that they apply:

  • Pull request pipelines regularly take long enough that engineers switch tasks or batch changes to avoid waiting.
  • Per-stage timings show most of the time going to repeatable work: reinstalling unchanged dependencies, recompiling unchanged modules, or running tests serially.
  • The codebase has, or can gain, some internal module boundaries, even if they are not yet enforced.
  • Someone on the team can own the pipeline configuration, because caches, shard plans and quarantine lists need ongoing maintenance.

Platform teams that support several monoliths get extra value, because patterns such as lockfile-keyed caches, staged pipelines and timing-based sharding can be reused across repositories with small changes.

How does speeding up monolith CI work?

Every technique here works by doing one of three things: avoiding repeated work, running independent work at the same time, or stopping early when a run is already known to fail.

  1. Avoid repeated work. Caching stores the output of a step together with a key derived from its inputs. When a later run produces the same key, the pipeline restores the output instead of recomputing it. Dependency caches key on the lockfile, Docker layer caching keys on each instruction and the files it copies, and remote build caches key on hashes of source files, dependency outputs, tool versions and flags. Change-based test selection applies the same idea to tests: if none of a test's inputs changed, it is skipped.
  2. Run independent work in parallel. Test sharding splits a suite across several workers. Wall-clock time then depends on the slowest shard, so balancing shards by recorded duration matters more than the number of shards.
  3. Fail early. Ordering stages from cheapest to most expensive, and cancelling sibling jobs on the first failure, means a broken commit stops before it consumes expensive runners.

All three depend on accurate inputs. A cache key that misses an input returns wrong results, a dependency graph that misses runtime coupling skips tests that should run, and timing data polluted by retries produces unbalanced shards. The sections below cover each technique and the specific way it can go wrong.

How do dependency caching and Docker layer ordering cut setup time?

Dependencies change far less often than source code, so setup is the cheapest place to start. Key the dependency cache on a hash of the lockfile, plus the OS, CPU architecture, and runtime version. The cache is then reused until a dependency actually changes and replaced as soon as one does. Keys built from branch names or dates cause either stale restores or needless misses.

A cache only helps if restoring it beats rebuilding it. Large caches have to be compressed, uploaded, and downloaded. When a fast registry mirror sits close to your runners, a fresh install can finish before a multi-gigabyte restore does. To check, run the job once with caching disabled. Then compare its install time against restore time plus install time on a warm cache. If the restore is not clearly faster, shrink the cache or remove it.

In Docker, the same principle depends on layer order. Copy only the lockfiles, install dependencies, and then copy the source. A source-only change then invalidates just the later layers, and the install layer is reused. A multi-stage build helps further. Compile in a stage that has toolchains and dev dependencies, then copy only the built output into a slim runtime stage. The final step becomes a cheap copy instead of a rebuild.

The main risk is correctness. If a key omits an input, such as a native compiler version or a postinstall script, the cache restores artifacts that no longer match the build. If a failed or compromised job writes to the cache, it can poison later builds. Allow cache writes only from trusted branches. Also budget engineering time for diagnosing unexplained misses and wrong hits, because that work is slow.

FROM node:20-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM deps AS build
COPY . .
RUN npm run build
RUN npm prune --omit=dev

FROM node:20-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=build /app/package.json ./
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
USER node
CMD ["node", "dist/server.js"]

A multi-stage Dockerfile that copies the lockfile and installs dependencies before copying source, so a source-only change reuses the cached install layer, and the slim runtime stage only copies the built output.

How do incremental builds and remote build caches work in a monolith?

Dependency caching restores inputs. Incremental builds go further and skip work whose inputs have not changed. In a monolith, that only works if the codebase has internal boundaries. Each module needs a clear source set and an explicit list of the modules it depends on. If everything compiles as one unit, any edit rebuilds everything.

Graph-aware build tools model each compile, test or packaging step as an action with declared inputs and outputs. They hash the inputs, which include source files, dependency outputs, tool versions and flags, to produce a cache key. If that key already exists in a shared remote cache, CI downloads the output instead of rebuilding it. A pull request branched from main can then reuse almost everything main already built.

Why hermeticity matters

The cache key is only as honest as the declared inputs. When an action also reads something the tool does not know about, two builds with identical keys can produce different results:

Undeclared input Failure mode
Environment variable read at build time Wrong hit: stale output reused after the value changes
Embedded timestamp or build ID Constant misses: output never matches, downstream actions rebuild
Network fetch during the build Wrong hit or nondeterminism depending on remote state
Unpinned system tool on the runner Output varies by machine under the same key

When to skip it

Moving to a graph-aware tool is a large migration. You have to declare every dependency, fix nondeterministic steps, and teach the team a new build model. For a mid-sized codebase where clean builds already take only a few minutes, that effort can cost more than the time it saves.

How should you split tests across parallel CI workers?

Splitting by file count or alphabetical order assumes every test file costs about the same to run. In a monolith that is rarely true. One integration suite that boots the full app and a database can take longer than hundreds of unit-test files combined. Wall-clock time is set by the slowest shard, so an uneven split wastes most of your parallelism. Timing-based splitting records each file's duration from previous runs and assigns files so every shard gets a similar total.

Strategy Input needed Shard balance Weakness
File count / alphabetical None Poor when file durations vary One heavy file dominates a shard
Historical timing Stored per-file durations Good New or renamed files have no history and need a default estimate

Illustrative estimate. Assume 40 minutes of serial test time and 3 minutes of fixed setup per job (checkout, dependency restore, service containers). An alphabetical split into 4 shards might produce 6, 8, 9 and 17 minutes, giving 20 minutes of wall-clock time. A timing-based split gives about 10 minutes per shard, or 13 minutes wall-clock and 52 compute-minutes. At 8 shards the wall-clock time is 8 minutes and compute is 64 minutes. At 16 shards it is 5.5 minutes and compute is 88. Setup is now most of each job, so more shards buy little speed at a large compute cost.

Reducing per-job overhead moves this curve. Use a shallow checkout. Pre-bake seed data into service images so containers do not run migrations at startup. Gate tests on a readiness check instead of a fixed sleep.

Sharding also exposes hidden coupling. A test that passed only because an earlier test seeded a row or set a global will fail once the two land on different shards. Treat these failures as real isolation bugs. Retrying them only hides the problem.

#!/usr/bin/env python3
import heapq
import json
import sys

DEFAULT_SECONDS = 30.0  # estimate for new or renamed files that have no history yet


def split(files, timings, shard_count):
    cost = {f: timings.get(f, DEFAULT_SECONDS) for f in files}
    # Min-heap of (total_seconds, shard_index, files). Each file goes to the lightest shard.
    heap = [(0.0, i, []) for i in range(shard_count)]
    heapq.heapify(heap)
    # Place the slowest files first so one heavy suite cannot pile onto a full shard.
    # Sort ties by name so every worker builds the same plan.
    for f in sorted(files, key=lambda f: (-cost[f], f)):
        total, i, assigned = heapq.heappop(heap)
        assigned.append(f)
        heapq.heappush(heap, (total + cost[f], i, assigned))
    return sorted(heap, key=lambda shard: shard[1])


if __name__ == "__main__":
    timings_path, index, count = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
    with open(timings_path) as fh:
        timings = json.load(fh)
    files = sorted(set(sys.stdin.read().split()))
    shards = split(files, timings, count)
    # Log the expected per-shard totals. Wall-clock time is set by the largest one.
    for total, i, assigned in shards:
        print(f"shard {i}: {len(assigned)} files, ~{total:.0f}s", file=sys.stderr)
    print("\n".join(shards[index][2]))

A deterministic longest-first greedy splitter that uses stored per-file durations to balance test shards, gives files with no history a default estimate, and logs per-shard totals so the slowest shard is visible.

When should you use change-based test selection, and when is it unsafe?

Change-based test selection runs only the tests that can reach the code a commit touched. The build tool or an analysis step maps changed files to their modules, walks the reverse dependency graph to find every module that depends on them, and schedules the tests that belong to those modules. In a monolith with clear internal module boundaries, a change to the invoicing package can skip the search and notification test suites entirely. That is often the largest single cut in pull request feedback time, because the work it removes is test execution rather than overhead.

The graph only records dependencies the compiler or build tool can see. A monolith also couples code at runtime in ways the graph never sees:

  • Reflection and dependency injection. A class looked up by name, or wired in by a container scanning annotations, has no static edge to the code that uses it.
  • Configuration. A changed feature flag, routing table, or serialization setting alters behavior in modules that never import the file.
  • Shared databases. A migration or a changed query in one module can break another module that reads the same tables, with no code dependency between them.

In each case the affected tests are skipped and the pull request goes green, so the regression merges silently.

For this reason, treat selection as a fast pre-merge signal and not as proof of correctness. Run the full suite on every merge to main, or at minimum before every release. When the full run catches something selection missed, trace the hidden coupling. Then either make it visible to the graph by declaring an explicit dependency, or add a rule that triggers the full suite for that path. Migrations and config directories are common candidates for such a rule.

Structuring stages for fast feedback and handling flaky tests

Order stages by cost per failure caught. Lint and formatting checks take seconds and catch a large share of trivial mistakes. Compilation comes next, because nothing downstream can pass if the code does not build. Unit tests follow. Only after those pass should the pipeline start integration and end-to-end stages, which need databases, browsers or seeded services and run on the most expensive runners. In the configuration below, each expensive stage depends on the cheap ones, so a missing import ends the run in a minute or two instead of after twenty minutes of provisioning. Inside a stage, configure jobs to cancel their siblings on the first failure where your CI system supports it.

Flaky tests belong in this discussion because they cost build time directly. Automatic retries hide that cost: a test that fails once and passes on retry shows up green, but the worker spent twice as long on it. They also distort timing-based splitting. Recorded durations include retry time, so the shards that hold flaky tests get fewer tests, and the shards stay unbalanced. Move known-flaky tests into a quarantine job that runs without blocking merges. Track each one with an owner and a date, and remove tests that stay quarantined too long rather than keeping them forever.

The trade-off is maintenance. Every stage adds configuration, dependency edges and another place where a failure can appear. Keep the stage count small enough that an engineer can find the failing job without searching through the pipeline view.

name: ci
on: [push, pull_request]

jobs:
  # Cheapest check first: fails in seconds on style and lint errors.
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make lint

  # Nothing downstream can pass if the code does not compile.
  build:
    needs: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make build

  # Sharded unit tests. fail-fast cancels sibling shards on the first failure.
  unit:
    needs: build
    runs-on: ubuntu-latest
    strategy:
      fail-fast: true
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - run: make test-unit SHARD=${{ matrix.shard }} TOTAL_SHARDS=4

  # Expensive stages start only after the cheap ones are green.
  integration:
    needs: unit
    runs-on: ubuntu-latest  # in practice, a larger runner label
    steps:
      - uses: actions/checkout@v4
      - run: make test-integration

  # Known-flaky tests: run and record results, but never block the build.
  # Do not mark this job as a required check for merges.
  quarantine:
    needs: build
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: make test-quarantine  # reads tests listed in quarantine.txt (owner, date)

A CI workflow where lint, build and sharded unit tests gate the expensive integration stage, and known-flaky tests run in a separate non-blocking quarantine job.

When should you avoid these optimizations?

Not every slow pipeline benefits from this work, and some of the techniques are risky in the wrong setting. Avoid or defer them when:

  • The pipeline is already fast enough. If engineers get results before they would switch tasks, added caching and sharding mostly add configuration to maintain.
  • The time is not in repeatable work. If per-stage timings show most of the cost in queueing, the fix is runner capacity, not caching. If one integration suite dominates and cannot be split without shared fixtures, sharding will not help until that suite is restructured.
  • Builds are not deterministic. Remote build caches and aggressive dependency caching depend on accurate cache keys. If builds read undeclared environment variables, fetch from the network or embed timestamps, fix that first. Otherwise you risk wrong cache hits that ship stale artifacts.
  • The codebase has no internal boundaries. Incremental builds and change-based selection need a dependency graph with more than one meaningful node. In a tightly coupled codebase, nearly every change invalidates everything, and the tooling adds cost without saving time.
  • Correctness requirements leave no room for skipped tests. If a release cannot tolerate a regression that a partial run might miss, do not let change-based selection gate merges or releases. Use it only as an early signal and keep full runs mandatory.
  • No one will maintain it. Cache keys, timing files and quarantine lists go stale. Without an owner, a stale cache or an ever-growing quarantine list can cost more than the original slow build.

What are the alternatives when CI tuning stops paying off?

Eventually each new optimization saves less than the one before it. Before you take on another round of CI tuning, check whether the pipeline is the actual constraint.

Option Engineering cost What it buys Main risk
Continued CI tuning Low per change, ongoing Small gains on the current structure Diminishing returns
Graph-aware build tool High migration, moderate upkeep Accurate caching and affected-target selection Build definitions drift from the code
Internal modularization Moderate, spread over time Enforced boundaries that caching can exploit Boundaries erode without checks
Separate services High, plus operational load Independent pipelines and deploys Network contracts, distributed debugging

Coupling is the limit when most changes touch a shared core that invalidates every cache. It is also the limit when change-based selection keeps returning "run everything", or when test splitting is capped by one large suite that cannot be divided without shared fixtures. In those cases CI work only postpones a decision about module boundaries. A graph-aware build tool has the same problem: a dependency graph of a tangled codebase is still one large node.

Internal modularization is usually the cheapest next step. It produces the boundaries that a build tool, or a later service split, needs to work well. Enforce those boundaries with dependency rules in CI. Without enforcement, they erode within a few months.

Whatever you choose, track p50 and p95 pipeline duration as weekly trends rather than individual runs. A rising p50 usually comes from new tests or dependencies. A widening p95 points to cache misses or flaky retries. Watching both lets you catch regressions before engineers start working around the pipeline.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to speed up monolith ci build, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.