Nx Distributed Task Execution for Sharded Backend E2E Tests

How does Nx distributed task execution work for E2E tests?

Nx distributed task execution splits one logical CI run across several machines while the run still behaves like a single command. A main job (the coordinator) computes the project and task graph for the targets you request, such as e2e across affected projects. It sends that graph to Nx Cloud, which hands out tasks to agent machines as their dependencies are satisfied. Each agent runs its task, uploads the outputs and terminal logs to the remote cache, and asks for more work. The main job pulls those results back, so logs, exit codes and build artifacts appear as if everything had run locally.

The scheduling is dynamic. An agent that finishes a short suite immediately picks up another task. Build outputs that an E2E task depends on are restored from cache instead of being rebuilt on every machine. Manual sharding works differently: each CI job receives a fixed index and runs the slice of tests assigned to it, regardless of what the other jobs are doing.

Backend and platform teams get the most out of this when they run slow, uneven E2E suites across many services in a single monorepo. That is the case where static splits waste the most agent time.

Copy +------------------------------------+
 | Main job (coordinator)             |
 | 1. compute task graph (e2e targets)|
 | 6. collect logs, exit codes,       |
 |    artifacts as if run locally     |
 +------------------------------------+
     | task graph            ^ results
     v                       |
 +------------------------------------+     +----------------------+
 | Nx Cloud                           |<--->| Remote cache         |
 | 2. hand out tasks when deps are met|     | outputs + term logs  |
 +------------------------------------+     +----------------------+
     |              |              |                  ^
     | 3. pull      | pull         | pull             | 5. upload
     v              v              v                  |
 +----------+   +----------+   +----------+           |
 | Agent 1  |   | Agent 2  |   | Agent 3  |           |
 | build    |   | e2e: api |   | e2e: auth|           |
 | 4. run   |   | 4. run   |   | 4. run   |           |
 +----------+   +----------+   +----------+           |
     |              |              |                  |
     +--------------+--------------+------------------+
       done -> ask for next task (dynamic scheduling)

 Manual sharding (contrast):
 +-----------+   +-----------+   +-----------+
 | Job idx 0 |   | Job idx 1 |   | Job idx 2 |   fixed slice each,
 | build+e2e |   | build+e2e |   | build+e2e |   no shared queue
 +-----------+   +-----------+   +-----------+

The main job computes the task graph and Nx Cloud gives tasks to agents as their dependencies are met; agents upload outputs and logs to the remote cache and the main job collects them, unlike manual sharding, where each job runs a fixed slice and repeats its own builds.

Why can't a single monolithic e2e target be distributed?

Nx schedules work at the level of a task, meaning one target on one project. When the coordinator hands out work, it can send api-e2e:e2e to an agent, but it cannot reach inside that task and give half of its test files to another machine. To Nx, the test runner process is opaque: it starts, runs every spec it finds, and exits with one status code and one set of outputs. A 40-minute suite in a single target is a 40-minute task, and adding agents does not shorten it.

To make the suite distributable, split it into several targets so that each one is a separate task. You can do this by hand in the project configuration. Define one target per feature area, such as e2e-orders, e2e-billing and e2e-auth, and have each target pass a file pattern or directory to the runner. Then add an aggregate e2e target that depends on all of them, so developers still run one command locally. Some Nx test runner plugins can infer these targets for you. They create one target per spec file, plus a parent target that groups them. This "atomized" or split mode is configured in the plugin options, so check the documentation for your plugin and Nx version.

The finest split is not always the fastest. Each task has a fixed cost: the process starts, test-runner configuration loads, and often a database or server boots. If that setup takes 20 seconds and a spec file runs for 5, per-file tasks spend most of their time on overhead. In that case, group small, related specs into targets that each run for minutes rather than seconds. Keep per-file splits for the long, uneven files that actually determine how long the run takes.

Copy{
  "name": "api-e2e",
  "projectType": "application",
  "targets": {
    "e2e-orders": {
      "executor": "nx:run-commands",
      "options": {
        "command": "npx jest --config apps/api-e2e/jest.config.ts apps/api-e2e/src/orders"
      }
    },
    "e2e-billing": {
      "executor": "nx:run-commands",
      "options": {
        "command": "npx jest --config apps/api-e2e/jest.config.ts apps/api-e2e/src/billing"
      }
    },
    "e2e-auth": {
      "executor": "nx:run-commands",
      "options": {
        "command": "npx jest --config apps/api-e2e/jest.config.ts apps/api-e2e/src/auth"
      }
    },
    "e2e": {
      "executor": "nx:noop",
      "dependsOn": ["e2e-orders", "e2e-billing", "e2e-auth"]
    }
  }
}

A hand-written project.json that splits one Jest e2e suite into three feature-area targets. Nx can schedule each target as its own task on a different agent. The no-op e2e target depends on all three, so nx run api-e2e:e2e still runs the whole suite with one command.

How do you isolate backend E2E tasks so they can run on any agent?

Distribution changes one assumption your suite probably relies on: that tests run in a known order on a machine nothing else is using. Under Nx distributed task execution, any task can land on any agent. An agent may run several tasks at once, and a retried task may run somewhere new. Each task has to bring its own world with it.

Start with state. Give every task a unique database name or schema, derived from the project name plus a random suffix, so two suites on the same agent never truncate each other's tables. Apply the same rule to ports: bind servers to port 0 and read back the port the OS assigned, instead of hardcoding 3000 and hoping. Message brokers need the same treatment. Use a per-task virtual host, topic prefix or queue name, so one suite's consumers don't pick up another suite's events.

Next, remove global setup that assumes it runs once per pipeline, such as a single migration step or a seed script in a CI job before the tests start. Move that work into each task's own setup, where it runs against that task's isolated resources.

Each task should start its own container or ephemeral database and tear it down in a finally block or shell trap, so cleanup also runs when assertions fail. Label these resources with the task ID. Then a sweeper can remove leftovers when an agent is killed mid-run.

Expect the first rollout to fail in ways that look random. Usually the cause is hidden coupling: shared fixtures, or tests that only passed because an earlier suite created the data they needed.

Copy#!/usr/bin/env bash
# Run one project's backend E2E suite against resources owned by this task only.
# Usage: e2e-isolated.sh <project-name>
set -euo pipefail

PROJECT="${1:?usage: e2e-isolated.sh <project-name>}"
SUFFIX="$(od -An -N4 -tx4 /dev/urandom | tr -d ' \n')"
# Unique per run, so retries and parallel tasks on one agent never collide.
TASK_ID="${PROJECT//[^a-zA-Z0-9]/-}-${SUFFIX}"
CONTAINER="e2e-pg-${TASK_ID}"
DB_NAME="e2e_${SUFFIX}"

# Cleanup runs on success, failed assertions, and Ctrl-C/SIGTERM.
cleanup() { docker rm -f -v "$CONTAINER" >/dev/null 2>&1 || true; }
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

# Label lets a sweeper remove leftovers if the agent dies mid-run, e.g.:
#   docker ps -aq --filter label=e2e.task-id | xargs -r docker rm -f -v
# '127.0.0.1::5432' asks Docker for a random free host port.
docker run -d --name "$CONTAINER" --label "e2e.task-id=${TASK_ID}" \
  -e POSTGRES_PASSWORD=e2e-only-password -e POSTGRES_DB="$DB_NAME" \
  -p 127.0.0.1::5432 postgres:16-alpine >/dev/null

# Check over TCP so we wait for the real server, not the init-time one.
for _ in $(seq 1 30); do
  docker exec "$CONTAINER" pg_isready -h 127.0.0.1 -U postgres -q && break
  sleep 1
done
docker exec "$CONTAINER" pg_isready -h 127.0.0.1 -U postgres

HOST_PORT="$(docker port "$CONTAINER" 5432/tcp)"
export DATABASE_URL="postgres://postgres:e2e-only-password@127.0.0.1:${HOST_PORT##*:}/${DB_NAME}"
# App code prefixes queue/topic names with this to keep broker traffic separate.
export E2E_NAMESPACE="$TASK_ID"

# Migrations and seeds run per task, not once per pipeline.
npm run migrate
npm run test:e2e

A per-task wrapper script that starts a labeled Postgres container on a random host port with a unique database name, runs migrations and tests against it, and removes the container through a trap even when tests fail or the task is interrupted.

How should target dependencies and cache inputs be declared?

Declare the dependency graph so the E2E target pulls in what it needs rather than building it inline. Give each backend e2e target a dependsOn entry on the application's build target, or on ^build if it relies on libraries. Then set outputs on that build so the artifacts can be stored and restored. If outputs are missing, a cache hit restores nothing, and the agent has to rebuild or fails with a missing bundle. When outputs are declared, the build runs once wherever the coordinator schedules it, and every other agent downloads the result.

Inputs control the hash, so mistakes here go one of two ways:

• Inputs too narrow (false cache hits). If the e2e target's inputs leave out migration files, seed data, the compose file for the test database, or an environment variable that changes behaviour, a real change produces the same hash. Nx then replays the old result and the suite never runs. This failure is silent, so it costs the most. If you cannot list every input with confidence, mark the e2e target as non-cacheable and cache only the builds.
• Inputs too broad (cache misses). If build inputs include test files, generated timestamps, or a whole-repo glob, the hash changes on unrelated commits. Every agent then rebuilds the same artifact and you lose the benefit of distribution.

Treat the cache as storage you do not control. Nx caches terminal output along with artifacts, so a test that logs a connection string will replay it on other machines and keep it in remote storage. Inject secrets at runtime, never write .env files into declared output directories, and redact secrets in logs.

Copy// nx.json (Nx accepts comments in its JSON config files)
{
  "namedInputs": {
    "default": ["{projectRoot}/**/*", "sharedGlobals"],
    "sharedGlobals": ["{workspaceRoot}/package-lock.json"],
    // Build hash ignores tests so test-only commits don't force rebuilds on every agent
    "production": ["default", "!{projectRoot}/**/*.spec.ts", "!{projectRoot}/**/*.e2e-spec.ts"]
  },
  "targetDefaults": {
    "build": {
      "dependsOn": ["^build"],
      "inputs": ["production", "^production"],
      // Without outputs, a cache hit restores nothing and the e2e agent has no bundle
      "outputs": ["{workspaceRoot}/dist/{projectRoot}"],
      "cache": true
    },
    "e2e": {
      // Build the app and its libs first; other agents pull the artifact from cache
      "dependsOn": ["^build"],
      "inputs": [
        "default",
        "^production",
        // Things outside the project that change test results must be in the hash
        "{workspaceRoot}/db/migrations/**/*",
        "{workspaceRoot}/db/seed/**/*",
        "{workspaceRoot}/docker-compose.e2e.yml",
        { "env": "E2E_FEATURE_FLAGS" }
        // DATABASE_URL and other credentials are injected by CI at runtime,
        // never listed here and never written into a declared output directory
      ],
      // Set to false if you cannot list every input with confidence
      "cache": true
    }
  }
}

An nx.json that makes e2e depend on cached, output-declared builds. It keeps test files out of the build hash, adds migrations, seed data, compose file and behaviour-changing env vars to the e2e hash, and leaves credentials out of inputs and outputs.

Configuring the CI pipeline: one coordinator, N agents and a stop condition

A distributed run has two kinds of jobs. The coordinator checks out the code, tells the distribution service that a run is starting, and invokes the affected or run-many commands as usual. It does not execute most tasks itself. Instead, the service hands each task to whichever agent is free, and the coordinator streams back logs and outputs as tasks finish.

The agents are N identical jobs that start, register with the service, pull tasks, and run them. Every agent must match the coordinator and every other agent:

• Same tool versions. Pin Node, the package manager, and any CLI the tests shell out to.
• Same service containers. Use the same database image tag and the same message broker configuration.
• Same environment variable names. Values can be injected per job.

Drift is costly because Nx hashes inputs and assumes any agent can produce the same result. If one agent runs a newer Postgres minor version, a test that depends on query ordering can pass on three agents and fail on the fourth. The failure follows whichever agent picks up the task, so it looks random. Define the environment once and reference it from both job types.

You also need a stop condition. Tell the coordinator which targets are the last stage, so agents shut down as soon as those targets finish instead of idling until a timeout. Idle agents still bill as CI minutes.

Keep a fallback job that runs the same targets on a single machine without distribution, triggered by a variable or a manual flag. If the hosted service is down or its pricing changes, you lose speed but still get a passing or failing pipeline.

How many Nx agents do you need for your E2E suite?

Agent count should come from the shape of your task durations, not from the size of the suite. Two numbers matter: the total task time across all E2E shards, and the duration of the single longest task. Wall-clock time can never drop below the longest task plus per-agent setup (checkout, dependency install, starting the database container). Past that floor, extra agents only wait.

Illustrative estimate

Assumed inputs: 40 split E2E tasks totalling 120 minutes of task time, an uneven distribution with the longest task at 12 minutes, and 3 minutes of setup on every agent. Wall-clock is approximated as setup + max(total ÷ agents, longest task). This is a best case, because real scheduling never packs tasks perfectly.

Going from 4 to 8 agents cuts 15 minutes of developer wait for 12 extra agent-minutes. That is usually worth it, because an engineer blocked on CI costs far more per minute than a runner. Going from 8 to 16 saves 3 minutes and nearly doubles the bill, because the 12-minute task now sets the pace while most agents sit idle.

When the table flattens like this, adding agents will not help. Split the longest task further, or make setup cheaper with a pre-built image or a shared database snapshot. Either change lowers the floor itself. Re-check the distribution periodically, since suites drift as tests are added.

How do you debug distributed E2E failures and handle flaky tests?

When a split target fails on an agent, rerun that one target locally with nx run <project>:<target>, using the exact task name from the run output, and add --skip-nx-cache so you get a real execution. If it passes locally but fails in CI, compare the environment the agent had: which database state it started from, which ports and env vars it saw, and whether a seed or migration task ran first. Logs are captured per task, so read the failing task's own output first. The interleaved coordinator log mixes unrelated tasks and hides causality.

Order-dependent tests

A test that fails only when split usually depended on a sibling for setup, such as a user row, a migrated schema or a warmed cache. In the monolithic run, file order supplied that setup by accident. To confirm, run the failing file alone against a fresh database. If it fails there too, the test has a hidden dependency. Move the setup into the test file or a shared fixture. Do not reintroduce ordering between targets.

Deliberate flaky-test handling

Scope retries to E2E targets only. Unit tests should not need them, and retrying them masks real bugs. Cap retries at one or two. Record every task that passed only after a retry, with the test name and failure message, in a place the team reviews. A test that keeps needing retries is a defect to fix, not noise. In backend suites it often points to a real race condition, such as an async job finishing after the assertion or two requests competing for the same row. Track the retry rate per test over time, so a spike after a backend change gets investigated rather than absorbed silently.

When should you avoid distributed E2E tests, and what are the alternatives?

Distribution has a fixed cost: every agent has to check out the repo, restore dependencies, start databases or containers, and pull cache artifacts before it runs anything. If your whole E2E suite finishes in a few minutes on one machine, that overhead can be a large share of the total. You pay for more compute and wait about as long as before. The same applies when each task needs heavy setup, such as seeding a large database or building service images. Splitting the suite repeats that setup on every agent. Small teams with a handful of E2E projects often get more from a stable pipeline than from a faster one. Compliance can also rule distribution out. If logs, artifacts or test data must stay on specific runners or must not reach a remote cache, the coordinator-and-agents model may conflict with those rules.

Running only affected projects combines well with any of the others. Start there, and add distribution only once affected runs are still too slow.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx distributed task execution, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.