Nx Atomizer for Backend E2E Tests: Split Suites per File in CI

What Nx Atomizer does and how per-file tasks work

Atomizer is a capability of Nx's inferred-task plugins. When the plugin scans a project, it finds each test file that matches its configured pattern and generates a separate task for it. Instead of one e2e target that runs the whole suite in a single process, you get one task per file and a parent target, commonly named something like e2e-ci, that depends on all of them. Running the parent target runs every file-level task. The parent does no test work of its own and exists only to collect results.

This changes the unit of work Nx sees in CI:

• Scheduling. Each file is a separate task, so Nx can place them on different machines and stop treating a 40-minute suite as one indivisible block.
• Caching. Results are cached per file. If a file's inputs haven't changed, its task can be restored from cache while the rest run.
• Retries. A flaky failure reruns one file, not the whole suite.

Copy                   e2e-ci  (parent target)
                      │
     ┌────────────┬───┴────────┬─────────────┐
     ▼            ▼            ▼             ▼
 orders.spec  users.spec  billing.spec  auth.spec
     │            │            │             │
  Agent 1      Agent 1      Agent 2       Agent 3

Target names, the naming pattern for file-level tasks, and the options that enable or rename the CI target vary by plugin and version. Before you wire anything into a pipeline, check them in the docs for the plugin you actually have installed, and inspect the inferred targets in your project's graph.

Copy  Inferred-task plugin scans the project for files matching its pattern
  and generates one task per file plus an aggregate parent target:

    ┌──────────────────────────────────────────────────┐
    │ e2e-ci  (parent target; exact name varies)       │
    │ no test work; depends on every file-level task   │
    └─────┬────────────┬────────────┬────────────┬─────┘
          ▼            ▼            ▼            ▼
      ┌────────┐   ┌────────┐   ┌────────┐   ┌────────┐
      │ orders │   │ users  │   │ billing│   │ auth   │  ◄ one task per file
      │  .spec │   │  .spec │   │  .spec │   │  .spec │  ◄ own cache entry
      └───┬────┘   └───┬────┘   └───┬────┘   └───┬────┘  ◄ retried alone
          │            │            │            │
          └─────┬──────┘            │            │
                ▼                   ▼            ▼
           ┌─────────┐         ┌─────────┐  ┌─────────┐
           │ Agent 1 │         │ Agent 2 │  │ Agent 3 │
           └─────────┘         └─────────┘  └─────────┘

  Check target names and plugin options in your installed plugin's docs.

The parent CI target runs no tests itself; it depends on one inferred task per test file, so Nx can distribute, cache and retry each file independently across agents.

Who should atomize a backend e2e suite, and when to avoid it

Atomizing pays off when a backend e2e suite has become the slowest required check on every pull request and is long enough that splitting it across machines would shorten the wait. The people who usually drive the change are platform engineers who own the pipeline and already watch queue times. Product teams benefit too, because their merges stop waiting on one serial job.

Backend suites are harder to split than frontend ones because the state sits outside the test process. A frontend e2e file often gets a fresh browser context. A backend file may share a database, a message broker, a schema migrated once at suite start, or a server process that stays up for the whole run. Over time, tests come to rely on that sharing without anyone noticing. One file may expect rows another file inserted. Another may assume a queue is empty, or that migrations already ran. Serial execution in a fixed order hides all of this. Per-file tasks running in parallel on different agents remove the ordering and the shared setup, and the hidden assumptions fail.

Delay or skip atomization when:

• Files are many and tiny. Each task pays fixed costs for checkout, install and service startup. If a file runs in seconds, that overhead outweighs the parallelism.
• Coupling is unresolved. Fix the shared-state dependencies first. Splitting a coupled suite does not remove its flakiness. It spreads the flakiness across more machines, where it is harder to reproduce.
• You can't provision matching agent environments. Every agent needs the same databases, brokers and configuration that the suite expects.

Step one: make every test file independent before splitting

Atomizing assumes any file can run alone, on any agent, in any order. Most backend suites never had to meet that bar, because a single long-lived process hid the shortcuts. Before you enable split tasks, remove them.

Give each file its own state

• Setup and teardown belong to the file. Each file starts the resources it needs in its own beforeAll and releases them in afterAll. A global setup that seeds data for "everyone" becomes a hidden dependency once files land on different machines.
• Name resources per task. Derive the tenant, schema or database name from the test file path plus a short random suffix. Two tasks then never write to the same rows, even if they share a database server. The code example below shows this, along with dropping the schema in teardown.
• Seed what you read. If orders.e2e expects a customer that customers.e2e created, it passes today only because of alphabetical ordering. Have each file insert its own fixtures.
• Make migrations idempotent. Several tasks may migrate at the same moment, and an agent may run against a database another task already migrated. Use "create if not exists" semantics or a migration lock so that a second run does nothing.

Find coupling before CI does

Run each file by itself locally, for example by passing a single path to your test runner. Then run the full suite in shuffled order, if your runner supports randomization. A file that fails alone or fails only in certain orders depends on another file.

When failures appear right after splitting, assume they were already latent. Splitting removes the accidental ordering and shared state that were masking them.

Copy// test/support/isolated-schema.ts
import { randomBytes } from 'node:crypto';
import path from 'node:path';
import { Client } from 'pg';

// Derive a schema name from the test file path plus a random suffix,
// so two tasks never share rows even on the same database server.
export function schemaNameFor(testFile: string): string {
  const base = path.basename(testFile).replace(/\W+/g, '_').toLowerCase();
  const suffix = randomBytes(4).toString('hex');
  // Postgres truncates identifiers at 63 bytes; keep the suffix intact.
  return `e2e_${base}`.slice(0, 50) + `_${suffix}`;
}

export async function createIsolatedSchema(testFile: string) {
  const schema = schemaNameFor(testFile);
  const client = new Client({ connectionString: process.env.DATABASE_URL });
  await client.connect();
  // IF NOT EXISTS keeps setup safe if a retry reuses the same name.
  await client.query(`CREATE SCHEMA IF NOT EXISTS "${schema}"`);
  await client.query(`SET search_path TO "${schema}"`);
  return { client, schema };
}

export async function dropIsolatedSchema(client: Client, schema: string) {
  // Teardown belongs to the file that created the schema.
  await client.query(`DROP SCHEMA IF EXISTS "${schema}" CASCADE`);
  await client.end();
}

// orders.e2e.ts: the file creates, seeds and drops its own state.
// let db: { client: Client; schema: string };
// beforeAll(async () => {
//   db = await createIsolatedSchema(__filename);
//   await runMigrations(db.client); // idempotent: safe to run twice
//   await db.client.query("INSERT INTO customers (id, name) VALUES (1, 'Test Co')");
// });
// afterAll(() => dropIsolatedSchema(db.client, db.schema));

A per-file helper for Postgres via node-postgres that derives a unique schema name from the test file path, creates the schema and sets it as the search path in beforeAll, then drops it in afterAll, so each file seeds and cleans up only its own data.

Handling service startup, resources and caching per task

Once files run as separate tasks, each one needs a reachable API, database and queue. There are three common ways to provide them.

On a single agent, concurrency limits come from the dependencies, not from Nx. Each in-task server opens its own database connection pool, so five parallel tasks with a pool of ten can exhaust a default Postgres connection limit. Use random or OS-assigned ports instead of a fixed port. Set parallelism from measured memory per task, not from CPU count.

Cache conservatively. A file-level task's inputs should include:

• the test file
• the source it exercises
• fixtures and seed data
• any environment variables or config that change behaviour

If a result depends on a third-party API, live data or the current time, disable caching for that target. A missing input does not cause a cache miss. It causes a false hit that reports green for code that was never tested.

As a sketch: named inputs and per-target inputs usually go in targetDefaults in nx.json. The atomized target name is set in the test plugin's options. Parallelism is set with the --parallel flag or the parallel setting. Verify each of these keys against your installed Nx version.

Copy{
  "namedInputs": {
    "e2eFixtures": [
      "{projectRoot}/fixtures/**/*",
      "{projectRoot}/seed/**/*"
    ],
    "e2eEnv": [
      { "env": "DATABASE_URL" },
      { "env": "API_BASE_URL" },
      { "env": "FEATURE_FLAGS" }
    ]
  },
  "plugins": [
    {
      "plugin": "@nx/jest/plugin",
      "include": ["apps/api-e2e/**/*"],
      "options": {
        "targetName": "e2e",
        "ciTargetName": "e2e-ci"
      }
    }
  ],
  "targetDefaults": {
    "e2e-ci--**/*": {
      "cache": true,
      "inputs": [
        "default",
        "^production",
        "e2eFixtures",
        "e2eEnv"
      ]
    }
  },
  "parallel": 3
}

Hedged nx.json sketch: file-level inputs (tests, upstream source, fixtures, env vars), the atomized CI target name, and a parallel cap of 3; set cache to false for suites hitting live services, and verify keys against your Nx version.

Distributing with Nx Agents and an illustrative speedup estimate

The parent target (for example e2e-ci) does no testing itself. It depends on the per-file tasks, and the orchestrator assigns each task to whichever agent is free. With Nx Agents, the main CI job sends the task graph to a pool of machines. Another multi-machine setup must do the same thing: give each machine a subset of tasks and collect the results. Any task can land on any agent, so every agent needs the same runtime, environment variables, secrets and service containers (database, broker, cache). If one agent lacks a secret, you get failures that look random and depend on scheduling.

Illustrative estimate (assumed numbers, not a benchmark)

Assumed inputs: 120 files, 45 s average runtime, 20 s boot overhead per task, one outlier file that takes 20 min. The average stays at 45 s.

• Serial test time = files × avg = 120 × 45 s = 90 min
• Total compute = files × (avg + overhead) = 120 × 65 s = 130 min
• Ideal wall-clock = total compute ÷ agents
• Real wall-clock ≥ max(ideal, longest task + overhead)

Two things follow from this. First, atomizing without adding agents makes the run slower, because you pay boot overhead 120 times instead of once. Second, adding agents stops helping once the longest file sets the floor, so split or trim outliers before you buy more machines. Agent count and task granularity set wall-clock together. Expect total compute minutes to rise as wall-clock falls.

Rollout, retries and debugging in an atomized pipeline

Roll out in two phases. Keep the existing single e2e target and add the atomized parent alongside it. Point CI at the parent while the old target stays runnable locally and as a fallback. Over a few weeks, compare failure rates and wall-clock durations between the two paths. A file that passes in the monolithic run but fails alone usually depends on state that an earlier file left behind, which section 3 should have removed. Once results are stable, delete the old target so the two paths cannot drift apart. Recheck per-file durations at the same time. Split any new outliers, or move their slow cases into separate files, so one long task does not set the floor again.

Atomizing also makes retries cheap, because a failed file reruns alone instead of rerunning the whole suite. That convenience can hide real problems. Record which files needed a retry to pass and review that list on a schedule. A file that keeps passing on its second attempt usually has a race condition or a shared-resource collision, not random flakiness.

To debug, reproduce the exact task that failed. The parent runs as something like nx run api-e2e:e2e-ci, and a single file as nx run api-e2e:e2e-ci--src/orders.spec.ts. The target names depend on how the plugin is configured in your workspace, so run nx show project api-e2e to see the real ones. Write logs, service output and artifacts to a path keyed by the task name, so a failure on any agent can be traced back to one file.

Copy#!/usr/bin/env bash
# Run one atomized e2e task with a single retry, keeping logs per task
# and recording files that only passed on retry.
# Usage: ./run-e2e-task.sh 'api-e2e:e2e-ci--src/orders.spec.ts'
# Target names depend on plugin config; list the real ones with:
#   npx nx show project api-e2e --json | jq -r '.targets | keys[]'
# The whole atomized suite runs via the parent: npx nx run api-e2e:e2e-ci
set -uo pipefail

TASK="$1"
# Turn 'api-e2e:e2e-ci--src/orders.spec.ts' into a filesystem-safe name.
SAFE_NAME="$(echo "$TASK" | tr '/:' '__')"
OUT_DIR="e2e-artifacts/${SAFE_NAME}"
mkdir -p "$OUT_DIR"

# Our own convention: test setup writes service output and artifacts here.
export E2E_ARTIFACTS_DIR="$OUT_DIR"

run_attempt() {
  # Skip the cache so a retry actually re-executes the file.
  npx nx run "$TASK" --skip-nx-cache > "$OUT_DIR/attempt-$1.log" 2>&1
}

if run_attempt 1; then
  exit 0
fi

echo "First attempt failed for $TASK, retrying once" >&2
if run_attempt 2; then
  # Passed only on retry: record it so a race condition is not hidden.
  echo "$(date -u +%FT%TZ) $TASK" >> e2e-artifacts/retried-files.txt
  exit 0
fi

echo "$TASK failed twice; logs in $OUT_DIR" >&2
exit 1

A CI wrapper that runs one atomized e2e task by name, stores its logs and artifacts in a directory keyed by the task name, and appends to retried-files.txt whenever a file passes only on its second attempt.

Alternatives and the impact on cost and engineering time

Atomizing is only one way to cut e2e wall time. The other options trade setup effort against isolation and retry precision differently.

Atomizing has real costs. Each task repeats service startup, so total compute minutes go up even as wall time goes down. You also need to provision and size agents, maintain a larger task graph, and read noisier output with one log per file. The inferred targets come from Nx plugins, so their names and options can change between releases, and someone has to check them on every upgrade.

Illustrative estimate (assumed numbers, not a benchmark): extra compute = files × per-task startup. With 60 files and 20 seconds of startup each, that is 60 × 20 s = 20 extra minutes per full run.

The engineering time saved usually outweighs that compute. Feedback arrives sooner. A flaky failure reruns one file instead of the whole suite. Files in unchanged areas hit the cache and don't run at all. A red task names the exact file that failed, so nobody has to dig through a single combined log to find it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx atomizer, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.