Monolith Test Impact Analysis: Module-Level Test Selection in CI

What module-level test impact analysis is and who should use it

Monolith test impact analysis at module granularity answers one question per change: which modules could this diff have affected? You take the list of changed files from version control, map each file to the module that owns it, add every module that depends on those modules, directly or transitively, and run only the tests belonging to that set. Everything else is skipped. Module granularity is a deliberate compromise. It is coarser than per-test coverage tracking but far cheaper to maintain, and it is easy to reason about when a selection looks wrong.

The payoff is CI speed. A change to the billing module should not wait on the reporting suite, and engineers get a pass or fail signal while the change is still fresh in their heads.

Who it fits

Backend and platform engineers on monoliths with real module boundaries and suites long enough that the wait hurts. If the full suite already finishes in a few minutes, the added machinery is not worth the risk of skipping a test you needed.

Defining modules

A module is whatever unit your codebase already treats as one: a language package, a top-level directory, or a build-system target. Build targets are the strongest choice when you have them, because they declare their dependencies explicitly.

Boundaries in name only

Avoid this approach, or fix the boundaries first, when the boundaries are not real. Watch for these signs:

• Circular imports between modules. A cycle means the modules have to be treated as one unit anyway.
• Imports of another module's internals rather than its public interface. These create dependencies your map may not show.
• Shared mutable state, such as globals, singletons, or a shared database schema. This coupling never appears in the import graph, so a change can break a test in a module that selection skipped.

How it works: from changed files to an affected test set

The pipeline has four steps, and each one narrows the set or widens it on purpose.

First, collect changed files with git diff --name-only against the merge base of the branch and its target, not against the target's current tip. A two-dot diff against origin/main attributes every commit that landed on main since you branched to your change. The three-dot form (origin/main...HEAD) diffs from the merge base and shows only your work. Deleted and renamed paths still appear in the output, and they matter: deleting a file affects its old module's dependents.

Second, map each path to its owning module, usually by longest matching directory prefix. Treat paths that match no module as unowned rather than dropping them.

Third, invert the module dependency map. The map records "A depends on B", but you need "who depends on B". Walk the reversed edges breadth-first from every changed module to get the transitive set of dependents. Stopping at direct dependents misses the case where B changes a type that A re-exports to C.

Fourth, select the tests owned by every module in that set.

The diff base is easy to get wrong without any error message. Shallow CI clones often lack the merge-base commit entirely. A stale local main that was never fetched produces an ancient base. A branch stacked on a parent that was later squash-merged keeps the parent's commits in its diff, which over-selects but stays safe. Rebases move the base, so you have to compute it fresh on every run. Each of these fails without an error, so cover the merge-base logic with tests that build real throwaway repositories for each scenario.

Copy +----------------------+      +-------------------------------+
 | merge-base logic     |      | 1. git diff --name-only       |
 | origin/main...HEAD   |----->|    paths incl. deleted and    |
 | fresh every run      |      |    renamed files              |
 +----------------------+      +-------------------------------+
            ^                                  |
            |                                  v
 +----------------------+      +-------------------------------+
 | tested against real  |      | 2. map path -> owning module  |
 | throwaway repos:     |      |    by longest dir prefix;     |
 | shallow clone, stale |      |    no match -> unowned        |
 | main, squash, rebase |      +-------------------------------+
 +----------------------+                      |
                                               v
 +----------------------+      +-------------------------------+
 | module dep map       |      | 3. BFS on reversed edges to   |
 | A depends on B       |----->|    transitive dependents of   |
 | invert: who uses B   |      |    every changed module       |
 +----------------------+      +-------------------------------+
                                               |
                                               v
                               +-------------------------------+
                               | 4. select tests owned by      |
                               |    every module in the set    |
                               +-------------------------------+
                                               |
                                               v
                               +-------------------------------+
                               | affected test set -> CI       |
                               +-------------------------------+

Test impact analysis pipeline: a freshly computed merge base feeds the diff, paths map to modules, the inverted dependency graph yields transitive dependents, and their owned tests form the affected set.

Building the module dependency map and selection script

Start with signals the build already trusts. Build-file dependency declarations give you module-to-module edges directly. Import statements give finer-grained edges and catch cases where code reaches into a module the build file never declared, which the build may still tolerate through transitive classpaths or path tricks. Generate the map from both. Commit it as a plain file, such as one module per line with its direct dependencies, and regenerate it in CI. If the regenerated map differs from the committed one, fail the build. A dependency change then shows up as a reviewable diff, which makes surprises visible.

Static analysis cannot see edges that are resolved at runtime. Reflection, DI container wiring, plugin registries, string-based routing tables, generated code, and resource files such as templates, schemas, and config all create real dependencies with no import behind them. Keep these in a separate, hand-maintained overrides file. Put a comment on each edge explaining why it exists, so a reviewer can tell when an edge has become stale.

Map each unit test file to the module whose directory contains it. Integration and end-to-end suites span modules by design, so don't force them into the graph. Give each suite its own group, and either run it always or tie it to an explicit list of trigger modules.

The selection script is short. It loads the generated map and the overrides, then inverts the edges. It maps each changed file to its owning module, then walks the inverted graph breadth-first to collect every module that depends on a changed module, directly or transitively. It prints the sorted set. A file that maps to no module should trigger a full run rather than being skipped.

Copy# Usage: git diff --name-only origin/main...HEAD | python3 select_modules.py
# Map lines look like "services/billing: libs/money libs/db" (module = its dir).
import sys
from collections import defaultdict, deque

def load(path):
    edges = {}
    with open(path) as fh:
        for line in fh:
            line = line.split("#", 1)[0].strip()  # overrides carry '# why' comments
            if not line:
                continue
            mod, _, deps = line.partition(":")
            edges.setdefault(mod.strip(), set()).update(deps.split())
    return edges

def owner(path, modules):
    # Deepest module directory containing the file wins (handles nested modules).
    hits = [m for m in modules if path.startswith(m + "/")]
    return max(hits, key=len) if hits else None

graph = load("deps.map")                      # generated, committed, diff-checked in CI
for mod, deps in load("deps.overrides").items():  # hand-maintained runtime edges
    graph.setdefault(mod, set()).update(deps)

rdeps = defaultdict(set)                      # invert: dependency -> its dependents
for mod, deps in graph.items():
    for dep in deps:
        rdeps[dep].add(mod)

modules = set(graph) | set(rdeps)
queue = deque()
for path in sys.stdin.read().split():
    mod = owner(path, modules)
    if mod is None:                           # unknown file: never skip, run everything
        print("ALL")
        sys.exit(0)
    queue.append(mod)

selected = set(queue)
while queue:                                  # BFS = reverse transitive closure
    for dependent in rdeps[queue.popleft()] - selected:
        selected.add(dependent)
        queue.append(dependent)
print("\n".join(sorted(selected)))

A minimal selection script that merges the generated dependency map with hand-written overrides, inverts the edges, maps changed files to their owning modules, and walks the reverse graph breadth-first, printing ALL when a changed file belongs to no known module.

Failing safe: explicit run-everything triggers

That rule generalizes. Some changes affect every module at once, or affect behaviour the import graph cannot see. For these, the script should skip selection and emit a single "run everything" signal.

Keep this list in the selection script as explicit path patterns, next to the module map. If the list lives somewhere else, it drifts out of date and nobody notices.

Engineers also need a way to force a full run by hand. Use a commit message tag or a PR label, and have the selector check for it first. Make every use visible: print the reason in the job log and count overrides per week. If nobody reviews them, overrides can become a habit. Each one is a deliberate full run, and enough of them quietly erase the time savings. A rising count usually means engineers don't trust the map, and the map is what you should fix.

Wiring selection into CI and validating the map

Trust comes from evidence, so the pipeline should produce some.

Pipeline layout

A provider-neutral layout has three stages:

1. Select job. It runs first and is cheap. It diffs the branch against the merge base, resolves the affected modules, and publishes the test list as a job output or artifact.
2. Test jobs. They depend on the select job and read its output. If the list is empty, they exit successfully instead of being skipped. That way required status checks still report.
3. Full runs. Every merge to main and a nightly schedule run the complete suite, ignoring selection. This catches anything selection missed within hours, not weeks.

Shadow mode first

Before selection gates any merge, run it in shadow mode. Compute the affected set on every PR, but still execute the full suite. Log every failure that falls outside the selected set. Each one is a candidate miss, and usually it traces back to a dependency edge the map doesn't record. Switch to gating only after these candidates stop appearing for a sustained period.

Separate flaky tests

Flaky tests corrupt this comparison. A flaky failure outside the selected set looks like a miss, and a flaky pass inside it can hide a real one. Quarantine known-flaky tests into their own tracked job so they don't pollute the shadow data.

Freshness check

Add a check that fails CI when a new module or a new cross-module import isn't reflected in the map. Without it, the map slowly drifts from the code.

Copy# Provider-neutral sketch: map these keys onto your CI system's own syntax.
# tools/*.py, test-map.json and the 'flaky' pytest marker are placeholders.
jobs:
  select:                      # runs first on every PR, cheap
    run: |
      git fetch origin main
      base=$(git merge-base HEAD origin/main)
      # Freshness: fail if a module or cross-module import is missing from the map
      python tools/check_map_fresh.py --map test-map.json
      git diff --name-only "$base" HEAD > changed.txt
      python tools/select_tests.py --map test-map.json --changed changed.txt > selected.txt
    outputs: [selected.txt]    # publish as artifact/job output

  test:                        # the required status check
    needs: [select]
    run: |
      # Empty selection exits 0, so the required check still reports
      if [ ! -s selected.txt ]; then echo "No affected tests"; exit 0; fi
      xargs pytest -m "not flaky" < selected.txt

  shadow:                      # PRs only, until selection is trusted
    needs: [select]
    run: |
      # Run everything, never gate; record failures outside selected.txt
      pytest -m "not flaky" --junitxml=full.xml || true
      python tools/report_misses.py --selected selected.txt --results full.xml

  quarantine:                  # flaky tests tracked separately, non-blocking
    run: pytest -m flaky --junitxml=flaky.xml || true

  full:                        # ignores selection entirely
    when: [push to main, nightly schedule]
    run: pytest -m "not flaky"

A provider-neutral pipeline sketch where a select job runs a map freshness check and publishes the affected test list, test jobs pass on an empty list, a shadow job runs the full suite to log candidate misses, flaky tests run in their own job, and full runs happen on main and nightly.

Alternatives and trade-offs

Module-level selection sits between two extremes, and the right choice depends on how much precision you need versus how much infrastructure you're willing to own.

Sharding never skips a regression, which is why it's the baseline to beat. Every selection approach trades that guarantee for speed. When selection misses something, the failure surfaces on main, where it blocks every other merge until someone reverts or fixes it.

Coarse granularity means a one-line change in a widely imported module selects most of the suite. You lose speed in that case, but you don't lose safety. Map rot is the more dangerous failure because it's silent.

No static map captures coupling through a shared database, test-order dependence, or shared fixtures. Two modules with no import relationship can still break each other through a table or a global seed.

The cultural risk matters too. If post-merge full runs catch misses reliably, engineers start treating pre-merge results as advisory. Track how many misses post-merge runs catch, and treat a rising count as a defect in the map.

Impact on cost and engineering time

Measure the effect with three numbers, all taken from CI logs rather than estimates:

• Selected-test ratio: tests run on the PR ÷ tests in the full suite. Report it per PR, not just as an average, because the distribution is bimodal.
• CI wall-clock time: queue-to-result time for the pre-merge job. Fixed costs such as checkout, build, and environment setup don't shrink when you select fewer tests, so this falls more slowly than the ratio.
• Escaped failures: tests that failed on main but were skipped on the PR that caused them. This is the cost side of the ledger.

Worked example

Illustrative estimate (assumed numbers, not a benchmark). Assume a 5-minute fixed overhead and a 35-minute full test phase. 30% of PRs touch a core module whose fan-in pulls in 90% of tests. The other 70% touch leaf modules and select 15%.

Expected time = overhead + test_time × Σ(share × ratio) = 5 + 35 × (0.3 × 0.9 + 0.7 × 0.15) = 5 + 35 × 0.375 ≈ 18 min

Nearly all the savings come from leaf changes. Core fan-in sets a ceiling: if every PR touched core, expected time could never drop below 36.5 minutes.

Net value = PRs × minutes saved × fraction of waiting that blocks work − (map maintenance + shadow-run compute + escapes × triage cost per escape). Count waiting minutes conservatively, because engineers often switch to other tasks while CI runs.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to monolith test impact analysis, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.