Speed Up Integration Tests in a Monolith: Rollback, Templates, Parallelism

Where integration test time actually goes in a monolith

Before changing fixtures or adding workers, find out where the time goes. In most monolith suites, the assertions are cheap. The expensive part is returning the database to a known state between tests. Every truncate, cascading delete or reseed hits disk, takes locks and rebuilds rows that the next test may never read. That cost repeats for every test, so it grows with the suite even when individual tests stay small.

Measure in three buckets

Start with the total wall-clock time of the suite. Then split it into three parts:

1. Schema setup and migrations. This is the one-time cost of creating the database and applying migrations.
2. Per-test reset. This covers truncating, deleting or reseeding before or after each test.
3. Test bodies. This is the time spent on the code under test and its assertions.

Use your test runner's slowest-tests report, such as pytest's --durations, to find outliers. Then put simple timers around the setup and teardown fixtures and log the totals. The runner usually reports per-test time, but it often doesn't separate fixture time from test time unless you instrument it.

Illustrative estimate (assumed numbers, not a benchmark): reset time = number of tests × reset per test 2,000 tests × 150 ms = 300 s, or about 5 minutes of reset alone, before any test logic runs.

Who benefits

Backend and platform engineers who own large shared suites gain the most from this work. It pays off when reset time is the largest of the three buckets and CI wait times are delaying merges. If test bodies dominate instead, faster database resets won't help much.

Copy# conftest.py: split suite time into schema setup, per-test reset and test bodies.
# Run with: pytest --durations=20   (the built-in report still lists slow outliers)
import time
from collections import defaultdict

import pytest

# Placeholders: swap in your project's own migration and reset helpers.
from myapp.testing import apply_migrations, truncate_all_tables

TOTALS = defaultdict(float)  # per process; with parallel workers, sum the per-worker output


@pytest.fixture(scope="session", autouse=True)
def db_schema():
    start = time.perf_counter()
    apply_migrations()  # bucket 1: one-time schema setup and migrations
    TOTALS["schema_setup"] += time.perf_counter() - start
    yield


@pytest.fixture(autouse=True)
def db_reset(db_schema):
    start = time.perf_counter()
    truncate_all_tables()  # bucket 2: the reset that repeats for every test
    TOTALS["per_test_reset"] += time.perf_counter() - start
    yield


@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
    # Bucket 3: wraps only the test function, so fixture time is excluded.
    start = time.perf_counter()
    yield
    TOTALS["test_bodies"] += time.perf_counter() - start


def pytest_terminal_summary(terminalreporter):
    terminalreporter.write_sep("-", "time by bucket")
    for name, seconds in TOTALS.items():
        terminalreporter.write_line(f"{name:>16}: {seconds:8.1f} s")

A pytest conftest.py that times schema setup, per-test reset and test bodies separately and prints the totals for each bucket at the end of the run, so you can see whether reset time dominates before you change anything.

Transaction rollback per test and the savepoint pattern

The cheapest reset is no reset at all: let the database discard the test's writes. A per-test fixture checks out one connection, runs BEGIN, hands that connection to the test, and on teardown runs ROLLBACK. Nothing is ever committed, so there are no rows to delete or tables to truncate, and the next test sees the seeded baseline.

This only works if every query in the test goes through that one connection. The application code, the HTTP test client, factories and assertion helpers all have to use it. A second pooled connection would not see the uncommitted rows, and anything it wrote would survive the rollback. In practice, you override the app's connection or session factory in the test setup so that it returns the fixture's connection instead of drawing from the pool.

Application code that calls commit() or opens its own nested transaction is the usual obstacle, because a real COMMIT ends the outer test transaction. The fix is savepoints. After BEGIN, the fixture issues SAVEPOINT test_sp. The app's commit is then mapped to RELEASE SAVEPOINT followed by a fresh SAVEPOINT, and its rollback is mapped to ROLLBACK TO SAVEPOINT test_sp. The app's transaction boundaries behave as written, but none of them escape the outer transaction.

Many ORMs and test helpers support this directly as a "join an external transaction" mode. You bind the session to a connection that already has an open transaction, and the session's commits become savepoint operations. Use that mode if your stack offers it rather than hand-patching commit calls.

Copyimport pytest

# `pool`, `app` and the override hooks are placeholders for your project's own
# connection pool and session/connection factory.
@pytest.fixture
def db_conn(pool, app):
    conn = pool.checkout()  # one physical connection for the whole test
    # Assumes the driver is in autocommit mode, so these statements
    # control the transaction directly.
    conn.execute("BEGIN")  # outer test transaction, never committed
    conn.execute("SAVEPOINT test_sp")

    class JoinedTransaction:
        """Handed to app code in place of a real transaction."""

        def commit(self):
            # App 'commits' end the savepoint, not the outer transaction.
            conn.execute("RELEASE SAVEPOINT test_sp")
            conn.execute("SAVEPOINT test_sp")

        def rollback(self):
            # Undo back to the last savepoint. The outer BEGIN stays open.
            conn.execute("ROLLBACK TO SAVEPOINT test_sp")

    # App code, the test client and helpers all get this same connection
    # instead of drawing a new one from the pool.
    app.override_connection_factory(lambda: (conn, JoinedTransaction()))
    try:
        yield conn
    finally:
        app.reset_connection_factory()
        conn.execute("ROLLBACK")  # discards everything, including 'committed' work
        pool.release(conn)

A pseudo-fixture that wraps each test in BEGIN/ROLLBACK, routes all app queries through one connection, and maps the app's commit and rollback to savepoint operations so they never end the outer test transaction.

When to avoid rollback: commit-time behavior and shared connections

Rollback isolation works because nothing in the test ever commits. That same property makes some tests pass when the code they cover is broken.

Behavior that only happens at commit

• Deferred constraints. A foreign key or unique constraint declared DEFERRABLE INITIALLY DEFERRED is checked at COMMIT. If the test rolls back, the check never runs, so a violation goes unnoticed.
• After-commit callbacks. ORM hooks such as "send email after commit" or "enqueue job after commit" never fire. The test can't tell whether the hook is wired up.
• Visibility. Uncommitted rows are invisible to every other connection. A worker, a second request, or a reporting query reading from a separate connection sees an empty table.

What a shared connection hides

When every code path goes through the test's single connection, pool exhaustion, wrong isolation levels and race conditions cannot occur, so the suite can't catch them. It can also hang. If application code starts a thread or async task that opens its own connection, that connection may block on rows locked by the test's open transaction while the test waits for the task to finish.

A separate tier

Put tests that run DDL, test locking, run background jobs or span processes in their own tier. Some DDL commits implicitly or refuses to run inside a transaction. This tier commits for real and resets state with a fresh template clone or with truncation.

This adds fixture complexity. You now maintain two setup paths, and data for the committing tier must be committed before the test starts. Make the fast tier the default and require an explicit marker, such as a committing tag, to opt into the slower tier. Document in the contributing guide which of the cases above require the marker.

Copy# conftest.py: rollback by default; tests opt into real commits with @pytest.mark.committing
# CI fast job:  pytest -m "not committing"    CI slow job:  pytest -m committing
import pytest
import psycopg2

DSN = "dbname=app_test user=app_test password=example-only host=localhost"
TABLES = "orders, order_items, outbox_jobs"  # tables the committing tier may write to

def pytest_configure(config):
    # Register the marker so a typo fails under --strict-markers instead of silently running in the fast tier.
    config.addinivalue_line(
        "markers",
        "committing: commits for real (deferred constraints, after-commit hooks, "
        "DDL, locking, background jobs, multi-process)",
    )

@pytest.fixture
def db(request):
    conn = psycopg2.connect(DSN)
    if request.node.get_closest_marker("committing"):
        # Slow tier: each statement commits, so deferred checks run, after-commit
        # hooks fire, and workers on other connections can see the rows.
        conn.autocommit = True
        yield conn
        conn.close()  # release locks before resetting
        reset = psycopg2.connect(DSN)
        reset.autocommit = True
        with reset.cursor() as cur:
            cur.execute(f"TRUNCATE {TABLES} RESTART IDENTITY CASCADE")
        reset.close()
    else:
        # Fast tier: psycopg2 opens a transaction implicitly; rolling it back
        # discards everything, so commit-time behavior is never exercised here.
        yield conn
        conn.rollback()
        conn.close()

A pytest fixture that uses rollback isolation by default and switches to real commits plus truncation for tests tagged with a registered committing marker, so CI can run the two tiers as separate jobs.

Template databases: build once, clone with CREATE DATABASE ... TEMPLATE

Rollback isolates individual tests, but each run still needs a database with the full schema in place. Running every migration from scratch on every CI job costs time in proportion to your migration history. That history only grows, and it often includes data backfills that no test needs. PostgreSQL offers a shortcut. Build one fully migrated and seeded database, mark it as the template, then create test databases from it with CREATE DATABASE test_w1 TEMPLATE app_template. Because the clone copies files at the storage level, its cost depends on database size rather than on how many migrations produced it.

Two requirements make this reliable:

• No connections to the template during cloning. PostgreSQL refuses the copy while any other session is connected to the template, and it reports that the source database is being accessed by other users. In CI this usually comes from a leftover migration process or a pooled connection that never closed. Before cloning, close every connection to the template and terminate any stragglers.
• A rebuild trigger. Hash the migration files and seed scripts, or a dump of the resulting schema, and store the hash in the template's name or in a comment on the template. If the hash differs from the current one, drop the template and rebuild it. Without this check, the template goes stale and tests run against a schema that differs from the code under test.

Because cloning is a file copy, a large seed dataset can make cloning per test slower than rolling back. Clone once per worker or per test group, and keep using rollback inside each clone.

Copy#!/usr/bin/env bash
set -euo pipefail
# Connection settings (PGHOST, PGUSER, PGPASSWORD) come from the CI environment.
TEMPLATE=app_template
WORKERS=${WORKERS:-4}
admin() { psql -v ON_ERROR_STOP=1 -d postgres -tAc "$1"; }

# Fingerprint every file that shapes the schema or the baseline seed data.
HASH=$(find db/migrations db/seeds -type f -print0 | sort -z \
  | xargs -0 sha256sum | sha256sum | cut -c1-16)

# The template stores its fingerprint in a database comment.
CURRENT=$(admin "SELECT shobj_description(oid, 'pg_database')
                 FROM pg_database WHERE datname = '$TEMPLATE'")

if [ "$CURRENT" != "schema:$HASH" ]; then
  echo "Template missing or stale (have '$CURRENT', want 'schema:$HASH'); rebuilding"
  admin "DROP DATABASE IF EXISTS $TEMPLATE"
  admin "CREATE DATABASE $TEMPLATE"
  for f in db/migrations/*.sql db/seeds/*.sql; do
    psql -v ON_ERROR_STOP=1 -q -d "$TEMPLATE" -f "$f"
  done
  admin "COMMENT ON DATABASE $TEMPLATE IS 'schema:$HASH'"
  # Block new sessions so nothing can hold the template open later.
  admin "ALTER DATABASE $TEMPLATE ALLOW_CONNECTIONS false"
fi

# Cloning fails if any session is still connected; terminate stragglers.
admin "SELECT pg_terminate_backend(pid) FROM pg_stat_activity
       WHERE datname = '$TEMPLATE' AND pid <> pg_backend_pid()" > /dev/null

# One clone per worker; tests inside each clone still use transaction rollback.
for i in $(seq 1 "$WORKERS"); do
  admin "DROP DATABASE IF EXISTS test_w$i"
  admin "CREATE DATABASE test_w$i TEMPLATE $TEMPLATE"
done

A CI setup script that rebuilds the migrated, seeded template only when the hash of the migration and seed files changes, terminates any sessions still connected to the template, and then clones one test database per worker with CREATE DATABASE ... TEMPLATE.

Parallel test runs with one database per worker

Parallel runs keep that per-worker clone and give each worker its own copy. Most runners expose a stable worker ID, such as PYTEST_XDIST_WORKER in pytest-xdist or JEST_WORKER_ID in Jest. Derive the database name from it, for example app_test_gw3. On startup, each worker drops any stale database with that name and clones a fresh one from the template. Workers never touch each other's rows, so tests need no extra isolation logic.

Copy              test runner
       ┌──────────┼──────────┐
   worker 0   worker 1   worker 2
       │          │          │
   app_test_0 app_test_1 app_test_2
       └──────────┼──────────┘
          cloned from app_template

Keep each worker's connection pool small, at two to five connections. Every worker multiplies the pool, and the total must stay under max_connections with headroom left for admin sessions. Size the worker count against both limits:

workers = min(usable CPU cores, (max_connections − reserved) ÷ pool size)

Illustrative estimate: assume 12 cores, max_connections = 100, 10 connections reserved and a pool of 5. The connection limit allows 18 workers, so the cores are the binding limit at 12. If the database server shares the host with the workers, start lower, around 8, because the server needs CPU too.

The database is only one shared resource. Each worker also needs its own:

• fixed ports
• cache key prefixes or cache DB indexes
• queue names
• temp and upload directories
• process-wide singletons

Replace hardcoded IDs and sequence assumptions with generated values.

Measure as you add workers. Past some count, the database server saturates on CPU, I/O or locks. After that point, extra workers make runs slower and timeouts flakier.

Tuning a disposable test database server for speed

Once the server itself is the bottleneck, make each write cheaper. A test database holds nothing you need to keep, so you can turn off the durability guarantees that cost the most I/O.

Put the data directory in memory

Run PostgreSQL in a container and mount its data directory (PGDATA) as tmpfs. Most container runtimes support this with a --tmpfs option on the run command. Every page write then goes to RAM rather than disk. When the container stops, the data disappears, which suits a throwaway instance. Size the tmpfs mount to fit your template database plus one clone per worker. If it fills up, writes fail.

Disable durability settings

Pass these server flags after the image name: -c fsync=off -c synchronous_commit=off -c full_page_writes=off.

• fsync=off stops the server from forcing WAL and data files to stable storage.
• synchronous_commit=off lets a commit return before its WAL record is flushed.
• full_page_writes=off stops the server from writing whole pages to WAL after each checkpoint. This cuts WAL volume.

On tmpfs, fsync costs little anyway. The other two settings still reduce WAL work and commit latency.

Never copy these settings into production

With fsync and full_page_writes off, a crash or power loss can corrupt the cluster beyond repair. With synchronous_commit off, commits the client saw succeed can be lost. Keep these flags in the test container definition only, never in a shared config file.

These settings also change how the server behaves around crashes. Recovery, replay of committed transactions and lost-write behavior will not match production. Run tests that cover crash handling, failover or durability on a separate instance with production settings.

Choosing a strategy, alternatives, and rolling it out

No single isolation strategy fits every test. Most monoliths end up with transaction rollback as the default and template clones for the minority of tests that need real commits.

Rolling it out

Put the new fixture behind a flag and migrate one test directory at a time. Before you switch a directory over, run its tests under both the old and new strategies in CI. A test that passes only under the old one usually depends on rows leaked from an earlier test. Track suite duration and flake rate for several weeks so you can tell real regressions apart from noise.

Expect flaky tests to rise at first. Most of these failures expose ordering bugs and hidden coupling that already existed; the old strategy was masking them. Budget engineering time to fix them rather than adding retries.

The fixtures you end up with are harder to maintain than a truncate script, and someone has to own them. That cost should be weighed against the benefit: every engineer gets faster feedback on every run, and the time saved adds up across the whole team.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to speed up integration tests monolith, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.