What does it mean to parallelize integration tests in CI, and when is it worth it?
To parallelize integration tests in CI is to split one long serial run into pieces that execute at the same time, either across several CI machines (sharding) or across worker processes on one machine. Adding runners is the easy part. The real work is isolation, so that tests running side by side don't share databases, queues, ports, or files, and balancing, so that no single shard holds the slowest tests and sets the wall-clock time for the whole pipeline.
The reason to do it is feedback time. On a large monolith, a serial integration suite can turn every push into a long wait. Engineers then batch changes, switch context, and merge with less confidence. Parallelism helps most when test execution dominates fixed per-job costs such as checkout, dependency install, image pulls, and schema migration. Every shard pays those costs again, so a suite that is mostly setup gains little from splitting.
Before you shard, check whether a cheaper fix exists:
| Symptom | Better first move | Why |
|---|---|---|
| Many tests cover the same path end to end | Delete or merge redundant tests | Fewer tests beat faster duplicates |
| Integration tests asserting pure business logic | Move them to unit tests | No I/O, so they run in milliseconds |
| A handful of tests take most of the runtime | Profile and fix those tests | One slow test caps the speedup of any shard |
| Setup dominates the job | Cache dependencies, prebuild images | Sharding multiplies setup cost |
Sharding can also hide a suite that has become unhealthy. Growing runtime, rising flaky test counts, and tangled fixtures stay out of sight while the wall-clock number looks fine.
Who should parallelize integration tests?
This approach fits teams that own a single large codebase where many engineers push to the same pipeline and the integration suite is the bottleneck. Typical owners are platform or developer-productivity engineers who maintain the CI configuration, and the application team leads who own the test suite and its fixtures. Both groups need to be involved. The CI side controls shard counts, runner sizes and artifact plumbing. The application side has to fix shared state, order dependence and slow fixtures, which the CI team cannot do alone.
It suits teams that:
- Run integration tests against real databases, queues or services, so tests are I/O-bound and slow per test.
- Have enough concurrent CI capacity, whether self-hosted runners or a hosted pool, to run several jobs at once.
- Can spend engineering time on test isolation, because parallelism exposes coupling that a serial run hides.
Small teams with a short suite, or teams without spare runner capacity, usually get more from the cheaper fixes in the table above.
When should you parallelize integration tests?
Parallelize when all of these hold:
- Test execution dominates job time. Your CI step timings show that most of the job is spent running tests, not on checkout, install, image pulls or migrations.
- The suite has already been pruned. Redundant tests are gone, pure logic lives in unit tests, and the worst individual tests have been profiled.
- No single file dominates. The slowest file is shorter than the shard duration you are aiming for. If it isn't, split or speed up that file first.
- Slow feedback is changing behavior. Engineers batch commits, the merge queue backs up, or people stop waiting for CI before moving on.
- You can isolate state. Each worker can get its own database or schema, port range, queue names and temp directory.
When should you avoid parallelizing integration tests?
Hold off, or keep parallelism minimal, when:
- Setup dominates the job. If most of each job is fixed overhead, sharding multiplies that cost for little wall-clock gain. Cache dependencies and prebuild images first.
- Tests depend on a shared external resource you cannot duplicate, such as a single sandbox account with a strict rate limit or a licensed service that allows one connection. Parallel shards will race on it and fail in ways that look like flakiness.
- The suite is already unstable. If flaky failures are frequent in a serial run, parallelism will add order-dependent failures on top, and you lose the ability to tell new problems from old ones. Stabilize first.
- CI capacity is the constraint. If runners are already saturated, extra shards sit in the queue and can make total pipeline time worse while raising billed minutes.
- The suite is short. When the serial run is already fast enough that engineers wait for it, the added plumbing (timing caches, aggregation jobs, isolation scripts) costs more to maintain than it saves.
How does parallelizing integration tests work?
At a high level, the pipeline does five things:
- Measure. Collect per-test and per-file durations from JUnit-style reports over several runs.
- Plan. Split the test files into N shards so each shard has roughly equal total runtime, using historical timings.
- Fan out. Start N CI jobs. Each job does its setup once, then runs its assigned files with several in-process workers.
- Isolate. Each worker derives unique names for its database, queues, cache prefixes, ports and temp directories from its shard index and worker ID, and clones a pre-migrated template database instead of replaying migrations.
- Fan in. A final job collects every shard's reports, checks that all shards reported, merges results, and saves fresh timing data from main-branch runs for the next plan.
The sections below cover each step in detail.
What are the alternatives to parallelizing integration tests?
Parallelism makes the same amount of work finish sooner. The alternatives reduce the amount of work, or move it out of the critical path. They are often combined with parallelism rather than replacing it.
| Alternative | What it does | Trade-off |
|---|---|---|
| Prune and push tests down the pyramid | Delete redundant integration tests and rewrite logic-only tests as unit tests | Requires test-by-test review; the effort pays off permanently |
| Test selection based on changed code | Run only tests affected by the files a change touches, mapped from coverage or build dependency data | Mapping can miss indirect dependencies in a monolith; still needs a periodic full run |
| Speed up fixtures and setup | Cache dependencies, prebuild images, use template databases, replace fixed sleeps with readiness checks | Helps every job, including sharded ones; limited by how much setup exists |
| Tiered pipelines | Run a fast subset on every push and the full suite in the merge queue, after merge or on a schedule | Some regressions are caught later, after more code has piled on top |
| Contract tests between modules | Verify interfaces between components instead of exercising full end-to-end paths | Needs well-defined boundaries, which many monoliths lack |
| Larger runners only | Use in-process workers on bigger machines without sharding | Single machine sets the ceiling; no repeated setup cost |
| Split the monolith | Separate codebases with independent, smaller suites | A large architectural change; not a CI fix |
How do you measure a test suite before splitting it?
Start with data you already have. Most test runners can emit JUnit-style XML, where each <testcase> element carries classname, name, and time attributes. Collect these reports from the last few dozen CI runs rather than one. A single run mixes real cost with noise from a busy runner, while a median across runs shows what each test reliably costs. Aggregate by file or class as well as by individual test. Most shard schedulers assign whole files, so per-file totals are the units you will actually balance.
Sort descending and look at the shape. Slow suites usually have a long tail: a handful of files that seed large datasets, boot extra services, or wait on timeouts account for much of the total. If one file runs longer than your target shard duration, no split can fix it. Break it up or speed it up first.
Next, separate test time from job time. The time attributes cover test execution only, and depending on the runner they may exclude suite-level setup hooks. The wall-clock job also includes checkout, dependency install, container startup, and schema migrations. Get those from your CI step timings. Every shard pays this overhead again, and that caps the speedup.
Illustrative estimate, assumed inputs: 4 minutes of fixed setup and 40 minutes of tests. Eight shards finish in about 4 + 5 = 9 minutes. Sixteen shards finish in about 6.5 minutes. Doubling the shard count saves 2.5 minutes of wall-clock time and doubles the total setup compute you pay for.
import glob, os, statistics, sys
import xml.etree.ElementTree as ET
from collections import defaultdict
test_times = defaultdict(list) # (classname, name) -> seconds, one entry per run
file_times = defaultdict(list) # file or class -> total seconds, one entry per run
for run_dir in sys.argv[1:]:
per_file = defaultdict(float)
for path in glob.glob(os.path.join(run_dir, "**", "*.xml"), recursive=True):
for tc in ET.parse(path).getroot().iter("testcase"):
t = float(tc.get("time") or 0)
cls = tc.get("classname", "?")
unit = tc.get("file") or cls # some runners emit file=; schedulers balance files
test_times[(cls, tc.get("name", "?"))].append(t)
per_file[unit] += t
for unit, total in per_file.items():
file_times[unit].append(total)
files = sorted(((statistics.median(v), k) for k, v in file_times.items()), reverse=True)
total = sum(m for m, _ in files) or 1.0
print(f"median test time: {total / 60:.1f} min across {len(files)} files")
running = 0.0
for m, unit in files[:15]: # the long tail: how much do the top files account for?
running += m
print(f"{m / 60:7.2f} min {running / total:6.1%} cumulative {unit}")
for m, (cls, name) in sorted(((statistics.median(v), k) for k, v in test_times.items()), reverse=True)[:10]:
print(f"{m:8.1f} s {cls}::{name}")
setup = float(os.environ.get("SETUP_MIN", "4"))
longest = files[0][0] / 60 if files else 0.0
for n in (4, 8, 16):
floor = max(longest, total / 60 / n) # no shard can finish faster than its slowest file
print(f"{n:3d} shards: ~{setup + floor:.1f} min wall clock, {setup * n:.0f} min of setup compute")
A standard-library Python script reads JUnit XML from several CI runs and takes the median time per test and per file. It ranks the long tail by cumulative share, then estimates shard wall-clock time from an assumed fixed setup cost and the slowest single file.
How does test sharding work alongside in-process parallelism?
There are two independent levels of parallelism, and most slow suites need both.
In-process parallelism runs several test workers on one CI machine. The test runner forks or spawns workers and hands each one a slice of the tests. It is cheap because setup happens once: one checkout, one dependency install, one compiled build. The limit is the machine. Once the worker count passes the number of available cores, workers compete for CPU. Each worker also holds its own application process, connection pool and caches, so memory runs out before you expect it to. Past that point, extra workers make the run slower.
Sharding splits the suite across separate CI jobs, usually on separate machines. Each job runs a deterministic subset, often chosen by a shard index and a shard total that the CI system passes in. Sharding scales past one machine, but every shard repeats the full setup. As the previous section showed, that fixed cost eventually dominates. Shards also wait in the queue for runners, so on a busy pool sixteen shards may not actually start together.
The two levels combine multiplicatively. Eight shards with four workers each give 32 concurrent test processes. Choose the worker count from the runner's size and use the shard count to cover the rest.
The trade-off is isolation cost. If each worker gets its own database schema, cache instance or service container, a four-worker runner hosts four copies of everything. On an undersized runner, queries slow down, connection and startup timeouts fire, and tests that wait on fixed sleeps start failing. That flakiness comes from resource starvation, not from bugs in the tests. Size runners for the per-worker footprint, not for the application alone.
+--------------------------------------------+
| CI pipeline: SHARD_INDEX / SHARD_TOTAL = 8 |
+--------------------------------------------+
|
+------------------------+----------------------+
| | |
v v v
+----------------------------+ +----------------+ +----------------+
| Shard 1 of 8 (runner A) | | Shard 2 of 8 | | Shard 8 of 8 |
+----------------------------+ +----------------+ +----------------+
| Setup, once per shard: | | same layout: | | same layout: |
| checkout, install, build | | setup + | ... | setup + |
| Test runner, 4 workers: | | 4 workers | | 4 workers |
| [W1] [W2] [W3] [W4] | +----------------+ +----------------+
| | | | | | Limits on one machine:
| v v v v | - workers beyond core count contend
| [D1] [D2] [D3] [D4] | - N copies of DB/cache eat memory
| per-worker DB schema, | - undersized runner -> slow queries,
| cache, service container | timeouts, failing fixed sleeps
+----------------------------+
Limits across machines: every shard repeats the full setup,
and shards wait in the runner queue, so they may not start together.
Combined: 8 shards x 4 workers = 32 concurrent test processes
Sharding splits the suite across CI jobs that each repeat setup, while in-process workers inside each job share one setup but each carry their own database and services, so per-machine CPU and memory set the worker count and the shard count covers the rest.
Which test sharding strategy gives balanced shards?
Once each shard has enough resources, the next question is which tests go to which shard. Shards finish only as fast as the slowest one, so the goal is to balance wall-clock time per shard, not test count.
| Strategy | Balances runtime? | Stable assignment? | Extra pipeline work |
|---|---|---|---|
| Round-robin / alphabetical | No | Shifts when files are added | None |
| Hash of test ID | No | Yes | None |
| Timing-based bin packing | Yes | Shifts as durations change | Store and restore timing data |
Round-robin and alphabetical splits assume every test takes roughly the same time. Integration suites rarely do. One file that boots a full application context and seeds a large dataset can take longer than fifty unit-style tests together, and whichever shard gets it becomes the critical path. Hashing test IDs (hash modulo shard count) fixes a different problem. Each test lands on the same shard every run, which makes failures easier to reproduce. It still ignores duration, though.
Timing-based splitting sorts tests by historical duration and assigns each one, longest first, to the currently lightest shard. This greedy bin packing keeps shard totals close together because the large items get placed while there is still room to even out around them.
The data needs a fallback. New or renamed tests have no recorded time, so assign them a default, such as the median duration, and spread them out instead of piling them onto one shard. Stale data degrades gracefully: shards drift out of balance rather than breaking.
The cost is pipeline plumbing. Every shard has to emit timing reports. A post-run job merges them and saves the result as a CI artifact or cache entry. The next pipeline restores it before planning shards. Save timings only from the main branch so that a broken feature branch can't skew the data.
import heapq
import json
import statistics
import sys
def plan_shards(test_files, timings, shard_count):
known = [timings[t] for t in test_files if t in timings]
# New or renamed tests get the median duration, so they are spread out, not piled up.
default = statistics.median(known) if known else 1.0
durations = {t: timings.get(t, default) for t in test_files}
# Min-heap of (total_seconds, shard_index): the top is always the lightest shard.
heap = [(0.0, i) for i in range(shard_count)]
shards = [[] for _ in range(shard_count)]
# Longest first, so large items are placed while there is room to balance around them.
# The name tie-break keeps the plan deterministic across runs.
for test in sorted(test_files, key=lambda t: (-durations[t], t)):
total, idx = heapq.heappop(heap)
shards[idx].append(test)
heapq.heappush(heap, (total + durations[test], idx))
return shards
if __name__ == "__main__":
# Usage: plan_shards.py timings.json SHARD_INDEX SHARD_COUNT < test_list.txt
path, index, count = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
try:
with open(path) as f:
timings = json.load(f)
except FileNotFoundError:
timings = {} # No cache yet: every test gets the default and the split is roughly even by count.
tests = [line.strip() for line in sys.stdin if line.strip()]
# Print only this shard's tests, one per line, for the test runner to consume.
print("\n".join(plan_shards(tests, timings, count)[index]))
A shard planner that assigns test files longest-first to the lightest shard using cached historical timings, falls back to the median duration for tests with no recorded time, and degrades to an even split when the timing cache is missing.
How do you isolate state per shard and per worker?
Every resource a test touches needs a name that no other shard or worker can produce. Build one run key from the shard index and the worker ID, both read from environment variables your CI and test runner expose. Use it as a suffix for the database or schema name, queue and topic names, and cache key prefixes. Add it to a base port number for any server the tests start, and append it to temp directory paths. If you derive every name from that one key, two workers cannot collide, and a leftover resource tells you which worker created it.
Run migrations once per job, not once per worker. Migrate a single template database, then clone it for each worker. PostgreSQL's CREATE DATABASE ... TEMPLATE does this directly. On other engines, restore a schema snapshot or dump. Cloning copies files instead of replaying hundreds of migrations, so worker startup no longer grows with migration history.
Load shared, read-only reference data such as countries, plans and permission tables into the template. Create per-test fixtures inside each test so they cannot leak from one test to the next.
Wrapping each test in a transaction and rolling it back is fast, but it only works when every write goes through one connection and nothing commits. It fails in these cases:
- Explicit commits in the code under test end the wrapping transaction.
- Multiple connections, such as a pool, a second service or a subprocess, cannot see uncommitted rows.
- Triggers with external side effects are not undone.
- Deferred constraints never run, so violations go unnoticed.
- Background jobs run on their own connections.
Tests that hit any of these need truncation or a fresh clone instead.
#!/usr/bin/env bash
set -euo pipefail
SHARD="${SHARD_INDEX:-0}"
WORKER="${WORKER_ID:-0}"
WORKER="${WORKER//[!0-9]/}" # strip prefixes like 'gw' so arithmetic works
WORKER="${WORKER:-0}"
RUN_KEY="s${SHARD}_w${WORKER}"
export DATABASE_NAME="app_test_${RUN_KEY}"
export QUEUE_NAME="jobs_${RUN_KEY}"
export TOPIC_NAME="events_${RUN_KEY}"
export CACHE_PREFIX="test:${RUN_KEY}:"
export APP_PORT=$((18000 + SHARD * 100 + WORKER))
export TEST_TMPDIR="${TMPDIR:-/tmp}/itest_${RUN_KEY}"
rm -rf "$TEST_TMPDIR" && mkdir -p "$TEST_TMPDIR"
psql -v ON_ERROR_STOP=1 -d postgres \
-c "DROP DATABASE IF EXISTS ${DATABASE_NAME};" \
-c "CREATE DATABASE ${DATABASE_NAME} TEMPLATE app_test_template;"
echo "worker ${RUN_KEY}: db=${DATABASE_NAME} port=${APP_PORT} tmp=${TEST_TMPDIR}"
A per-worker setup script that builds one run key from the shard index and worker ID, derives database, queue, topic, cache prefix, port and temp directory names from it, and clones a pre-migrated PostgreSQL template database instead of running migrations per worker.
How do you find hidden coupling and handle flaky tests without masking regressions?
Splitting a suite onto parallel workers changes which tests run together and in what order. That surfaces coupling that a single serial run happened to hide. The usual sources are:
- Order dependence: a test passes only because an earlier one created a row, warmed a cache or set an environment variable.
- Global singletons such as configuration objects, feature-flag clients and in-memory registries that keep state between tests.
- Shared fixtures mutated in place, so the second test to touch the fixture sees modified data.
- Wall-clock time: assertions that break near midnight, across a DST change or under a slower worker.
- Fixed IDs or auto-increment assumptions, for example "the new user is ID 1".
- External services with shared accounts, where two shards race on the same sandbox tenant or rate limit.
To find the cause, randomize test order and record the seed in the job log. A failure is only reproducible if you know the order that produced it. Run the suspect test alone and repeatedly. If it still fails, the problem is inside the test. If it passes, the problem is a neighbor, so bisect the tests that ran earlier on the same worker until you find the one that leaks state.
To reproduce a failure locally, you need three things from the CI log: the shard index, the seed and the exact ordered test list. With those you can replay the shard on your machine rather than guess at it.
For flaky tests, allow retries per test rather than per job, and cap them at one or two. Record every test that needed a retry. A test that passed on retry still failed once, and that failure may be a real race condition. Quarantine chronic offenders in a non-blocking job, and give each one an owner and a deadline. Without those, quarantine becomes a graveyard of tests nobody runs or fixes.
Expect a wave of failures in the first weeks after you turn on parallelism, and set aside engineering time to work through them.
How much does sharding cost, and how do you keep the split healthy?
Treat sharding as a pipeline with a fan-in step, not N independent jobs. Add a final aggregation job that runs even when upstream shards fail. It should download every shard's JUnit XML and coverage file, merge them, and compare the number of reports received against the configured shard count. A missing report fails the check, the same as a red test. A crashed or cancelled shard produces no failures, and without that count it can look like a pass. Tag each test result with its shard index and job URL so a failure in the merged report links straight to the log that contains it.
Illustrative estimate (assumed numbers, not a benchmark). Assume a 60-minute serial suite (S), 4 minutes of per-job overhead (O) for checkout, dependencies and database startup, and perfect balance. Wall clock ≈ S/N + O. Billed minutes ≈ S + N·O.
| Shards (N) | Wall clock (min) | Billed minutes |
|---|---|---|
| 1 | 64 | 64 |
| 4 | 19 | 76 |
| 8 | 11.5 | 92 |
| 16 | 7.75 | 124 |
Going from 8 to 16 shards saves under 4 minutes of wall clock and costs 32 more billed minutes. Fixed overhead dominates at that point, so cutting O usually beats adding shards.
Keep the split healthy without manual effort:
- Refresh the timing data automatically from main-branch runs only, so slow or broken PR runs don't skew it.
- Alert when the slowest shard drifts noticeably above the median shard.
- Revisit N when the suite grows or when overhead starts to rival per-shard test time.
The real saving is engineer time, not compute. Shorter waits mean fewer context switches per pull request and faster merge-queue throughput. Billed minutes go up. Weigh that increase against developer hours, not against the old CI bill.
How can parallelizing integration tests save engineering time?
The time saved comes from several places, not only the shorter wait itself:
- Less waiting and less context switching. When CI finishes while the change is still fresh, engineers fix failures immediately instead of returning to them after switching to other work.
- Smaller, safer changes. Fast feedback removes the incentive to batch several changes into one push to avoid paying the wait more than once. Smaller changes are easier to review and easier to bisect when something breaks.
- Higher merge-queue throughput. If the queue tests each change before merging, the suite duration limits how many changes merge per hour. Shorter runs reduce queue backups and the rebases that follow them.
- Faster diagnosis. The plumbing that parallelism requires (per-test timings, recorded seeds, ordered test lists per shard, results tagged with shard and job) also makes failures quicker to reproduce and attribute.
- Cleaner test suites over time. Isolation work removes order dependence and shared state, which cuts down on flaky reruns that cost time in a serial pipeline too.
Illustrative estimate (assumed inputs, not a measurement). Assume 20 engineers, each waiting on the integration suite 3 times per working day. Using the table above, going from 1 shard (64 minutes) to 8 shards (11.5 minutes) shortens each wait by 52.5 minutes. Assume only a quarter of that waiting time was actually lost, because engineers did other useful work during the rest. That gives 20 × 3 × 52.5 × 0.25 ≈ 790 minutes, or roughly 13 engineer-hours per day. On the CI side, the same 60 daily runs add 28 billed minutes each (92 − 64), about 1,680 extra CI minutes per day. Plug in your own team size, push frequency, CI pricing and loaded engineering cost to see whether the trade holds for you.
Count the one-time costs as well: building the shard planner and aggregation job, isolating shared state, and working through the wave of order-dependent failures that parallelism exposes. Those costs come at the start, and the time savings keep coming for as long as the suite stays balanced.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to parallelize integration tests ci, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.