Nx Distributed Task Execution for Backend CI Across Many Agents

How Nx distributed task execution works

Nx distributed task execution (DTE) moves scheduling out of your CI YAML and into the task graph. The main CI job runs your normal Nx command, for example running test, build, and lint for affected projects. Nx computes the full graph of tasks and their dependencies, then hands that graph to Nx Cloud instead of executing it locally. Nx Cloud orchestrates the run. It knows which tasks are ready because their dependencies have finished, and it assigns them to whichever agent asks for work next.

Agents are interchangeable machines that each check out the same commit. An agent never needs to build a library itself just because a later task depends on it. When an agent finishes a task, its outputs, such as compiled dist folders or generated clients, go to the remote cache. When a dependent task lands on another agent, that agent restores those outputs from the cache instead of rebuilding them. Terminal output and artifacts are also collected back to the main job, so logs and exit codes look the same as a single-machine run.

Teams often try a simpler approach first: split the project list into N chunks and send one chunk to each CI job. This ignores the graph, so the same dependencies get rebuilt in several jobs. Balance also suffers, because one chunk can contain the 20-minute integration suite while another finishes in two minutes.

The biggest gains go to backend and platform teams with large monorepos. These repos tend to have many services and libraries, each with independent build, test, and integration-test targets. That produces a wide graph with many tasks that can run at the same time. A narrow, deeply chained graph gains much less, because each task still has to wait for the one before it.

Copy+-----------------------------------------------+
| Main CI job (same commit on every machine)    |
|  nx affected -t test build lint               |
|  computes task graph, hands it to Nx Cloud    |
+-----------------------+-----------------------+
                        | task graph
                        v
+-----------------------------------------------+
| Nx Cloud orchestrator                         |
|  ready queue: tasks whose deps have finished  |
+-----+-----------------+-----------------+-----+
      ^ |               ^ |               ^ |
 pull | | assign   pull | | assign   pull | | assign
      | v               | v               | v
+-----------+     +-----------+     +-----------+
| Agent 1   |     | Agent 2   |     | Agent 3   |
| build lib |     | test api  |     | lint web  |
+-----+-----+     +-----+-----+     +-----+-----+
      | ^               | ^               | ^
 push | | restore  push | | restore  push | | restore
      v |               v |               v |
+-----------------------------------------------+
| Remote cache: dist/, generated clients        |
+-----------------------------------------------+
  logs + exit codes collected back to main job

The main CI job hands the task graph to Nx Cloud, agents pull tasks as their dependencies finish, and outputs move between agents through the remote cache, so a task such as test api restores the library that Agent 1 built instead of rebuilding it.

Minimal setup: starting a distributed CI run

The smallest working setup changes only the start of your main CI job. Before any Nx commands run, call npx nx-cloud start-ci-run with a --distribute-on value such as "5 linux-medium-js". The number sets how many agents to start and the second part sets their machine type. After that, run nx affected -t lint test build exactly as you would on a single machine. Nx sees that a distributed run is active, sends tasks from the graph to the agents, and streams their output back.

The main job still controls the run. It prints logs for every task, waits for all of them to finish, and exits non-zero if any task fails. Your CI provider's pass/fail status, required checks, and log viewer keep working unchanged, even though the work ran on other machines.

Decide early what should happen when Nx Cloud is unreachable. You can let the job fail, so outages are obvious. Or you can fall back to running everything on the main machine, which keeps merges moving but can make a five-minute pipeline take an hour. Check how your Nx version behaves by default, then choose explicitly.

Distribution also makes the remote cache security-sensitive. Agents restore outputs from it without rebuilding, so a poisoned cache entry can ship as a real artifact. Give CI a read-write token only on trusted branches. For untrusted fork PRs, use a read-only token or disable remote caching entirely, so outside code can never write entries that later builds will trust.

Copy# .github/workflows/ci.yml : main job for a distributed Nx run
name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  main:
    runs-on: ubuntu-latest
    env:
      # Trusted pushes to main get a read-write token so they can populate the cache.
      # Pull requests get a read-only token and cannot write cache entries.
      # GitHub does not pass repository secrets to fork PRs on `pull_request`,
      # so fork PRs end up with no token. Confirm how your setup behaves in that case.
      NX_CLOUD_ACCESS_TOKEN: ${{ github.event_name == 'push' && secrets.NX_CLOUD_RW_TOKEN || secrets.NX_CLOUD_RO_TOKEN }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # `nx affected` needs history to compare against the base commit

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci

      # Start the distributed run before any Nx command executes.
      # "5 linux-medium-js" means 5 agents on the linux-medium-js machine type.
      - run: npx nx-cloud start-ci-run --distribute-on="5 linux-medium-js" --stop-agents-after="build"

      # Sets NX_BASE and NX_HEAD so `affected` knows what changed.
      - uses: nrwl/nx-set-shas@v4

      # This is the same command you would run on one machine. Tasks run on the agents,
      # and this job streams their logs back and exits non-zero if any task fails.
      # Before relying on this, check your Nx version's default behavior when
      # Nx Cloud is unreachable (fail the job, or run everything locally).
      - run: npx nx affected -t lint test build

A GitHub Actions main job that starts an Nx Cloud distributed run on five agents and runs nx affected while keeping logs and exit status in the main job, with a read-write cache token used only for trusted pushes to main.

Making the task graph trustworthy before you distribute it

Distribution changes one assumption your local runs quietly relied on: that every task runs on the same disk. On an agent, a task sees only the files in the checkout plus whatever Nx restores from the cache. Nx restores only the paths you declared as a target's outputs.

Declare what each target reads and writes

For every cacheable target, define three things:

• inputs: the files and environment variables that affect the result. Include source files, config files, and variables declared with the env input type.
• outputs: every path the target writes, such as {projectRoot}/dist or a generated-code directory.
• dependsOn: ordering rules. "^build" on test means dependencies build before the project's tests run.

Missing output: a worked example

Suppose a codegen target writes a generated API client into src/generated, but its outputs lists only dist. Locally, test passes because the generated files are still sitting on your disk. In distributed CI, the scheduler runs codegen on agent A and test on agent B. Agent B restores dist and nothing else. The import of the generated client fails, and the result looks like a flaky agent when it is really a missing declaration.

Missing input: false cache hits

The opposite failure is worse because nothing visibly breaks. If DATABASE_URL or a shared tsconfig isn't listed in inputs, changing it leaves the hash the same. Nx replays the old passing result, and code that was never tested gets merged.

Deploy and publish steps get their build artifacts back through the cache in the same way. Any build output you haven't declared is missing from what you ship.

Copy// apps/orders-api/project.json (Nx parses this file with comment support)
{
  "name": "orders-api",
  "targets": {
    "codegen": {
      "command": "node tools/gen-client.js apps/orders-api/openapi.yaml",
      "cache": true,
      // Only the spec affects generated code
      "inputs": ["{projectRoot}/openapi.yaml"],
      // Declare the generated dir so other agents get it from the cache
      "outputs": ["{projectRoot}/src/generated"]
    },
    "build": {
      "command": "tsc -p apps/orders-api/tsconfig.app.json",
      "cache": true,
      // Generate this project's client, then build its dependencies first
      "dependsOn": ["codegen", "^build"],
      "inputs": ["production", "^production", "{workspaceRoot}/tsconfig.base.json"],
      // Deploy and publish steps get dist back only if it is declared here
      "outputs": ["{projectRoot}/dist"]
    },
    "test": {
      "command": "jest -c apps/orders-api/jest.config.ts",
      "cache": true,
      "dependsOn": ["codegen", "^build"],
      "inputs": [
        "default",
        "^production",
        // Shared config outside the project must be listed explicitly
        "{workspaceRoot}/tsconfig.base.json",
        // A change to this env var must change the hash, or Nx replays a stale pass
        { "env": "DATABASE_URL" }
      ],
      "outputs": ["{workspaceRoot}/coverage/apps/orders-api"]
    }
  }
}

This project.json declares inputs, outputs and dependsOn for each target. Generated code, build output, shared config and environment variables are all included, so cache restores on any agent are complete and the cache hash changes when anything the result depends on changes.

Preparing agents for backend workloads and integration tests

Each agent is a fresh machine that runs whatever slice of the task graph the orchestrator gives it. Nx assumes a task produces the same result on any agent. That only holds if every agent is set up exactly like the main job: same Node and language runtime versions, same lockfile-based install, same code generation (ORM clients, protobuf stubs, OpenAPI types), and the same environment variables. If one agent skips codegen or has an older compiler, the failure shows up in a random project, depending on which agent picked it up. If an env var changes behaviour but isn't declared as a cache input, a wrong result can be cached and replayed everywhere. Put agent setup in one shared script so the main job and the agents can't drift apart.

Integration tests that need Postgres, Redis or a message broker are the harder part. Agents run several tasks in parallel, so two tasks can end up targeting the same database.

These approaches work best together. Give each agent its own containers, have each task derive its schema name from the project name, and keep these tests in their own target so they can run on larger agents with lower parallelism.

Be realistic about scope. Hard-coded ports, fixed queue names and shared seed data are spread across test suites. Untangling them usually takes more work than the Nx configuration itself.

Copy// test/db-schema.ts: gives each Nx task its own Postgres schema.
// Nx sets NX_TASK_TARGET_PROJECT for every task it runs, so parallel tasks
// on the same agent (and the same per-agent container) never share tables.
import { Client } from 'pg';

// Postgres on the agent's ephemeral container; the shared setup script exports this.
const url = process.env.TEST_DATABASE_URL ?? 'postgres://test:test@localhost:5432/test';

export function schemaName(): string {
  const project = process.env.NX_TASK_TARGET_PROJECT ?? 'local';
  // Identifiers must be safe to interpolate: lowercase, [a-z0-9_] only.
  // The pid suffix keeps two runs of the same project apart.
  const safe = project.toLowerCase().replace(/[^a-z0-9_]/g, '_');
  return `t_${safe}_${process.pid}`.slice(0, 63); // 63 = Postgres identifier limit
}

export async function createTestSchema(): Promise<string> {
  const schema = schemaName();
  const client = new Client({ connectionString: url });
  await client.connect();
  try {
    await client.query(`DROP SCHEMA IF EXISTS "${schema}" CASCADE`);
    await client.query(`CREATE SCHEMA "${schema}"`);
  } finally {
    await client.end();
  }
  // App config reads this instead of a hard-coded schema, then sets search_path.
  process.env.DB_SCHEMA = schema;
  return schema;
}

export async function dropTestSchema(): Promise<void> {
  const client = new Client({ connectionString: url });
  await client.connect();
  try {
    await client.query(`DROP SCHEMA IF EXISTS "${schemaName()}" CASCADE`);
  } finally {
    await client.end();
  }
}

A test helper that builds a unique Postgres schema name from the Nx task's project, so integration tests running in parallel on one agent don't write to the same tables.

Splitting long-pole tasks and sizing agent counts

No amount of agents can make a pipeline shorter than its slowest single task. If one integration suite takes 15 minutes, the run takes at least 15 minutes plus agent setup, while the other agents finish early and sit idle.

Splitting the long pole

Nx task splitting, sometimes called the Atomizer, uses inferred targets from test-runner plugins such as Jest, Playwright and Cypress. When enabled, the plugin creates a CI-specific target for each test file, plus a parent target that depends on all of them. Each file then becomes its own cacheable task that Nx can schedule on any agent.

If your runner has no splitting support, split the suite yourself. Move slow integration specs into a separate project, or define several targets that each select a directory or tag. Nx can then distribute and cache these targets independently.

Sizing agents

The illustrative estimate below assumes 40 tasks of 3 minutes each, a 2-minute setup cost per agent and perfect load balancing:

Beyond about 10 agents, each extra agent saves little wall-clock time but still adds its full setup cost in agent-minutes. To avoid paying for agents a run doesn't need, have Nx Cloud scale the agent count with the size of the changeset. A small change that affects a few projects should run on a few agents, and only broad changes should get the full fleet.

Handling flaky tasks and verifying the setup is correct

Nx Cloud can detect and retry flaky tasks automatically. Its definition of a flaky task is precise: the same task, with the same computation hash, has both passed and failed. Because the hash covers every declared input, a mixed result on identical inputs means something outside those inputs affected the outcome. Treat retries as a way to keep the pipeline moving, and use the flaky-task list as a backlog to fix.

Some failures look flaky but are not. If a task fails on one agent and passes on another, check whether the agents differed in ways the hash ignores. Tests that depend on execution order break when they share a database or a temp directory. Tests can also depend on the default time zone or locale, or on a tool installed on some images but not others. A single machine hides these problems because everything shares one environment. Pin TZ and locale in the agent setup, and give each test run its own schema or port range.

When a task runs on a remote agent, its output streams back to the main job's log. The full log for each task is also available on the run's page in Nx Cloud, so you don't need shell access to the agent.

Verification routine

1. On one commit, run the same targets on a single machine with the cache skipped. Then run them distributed. The task lists and results should match.
2. Re-run the distributed job without changes. Every task should be a cache hit. Any task that misses has an input that changes between runs, such as a timestamp or a generated file.
3. Edit a file inside a single library. Confirm that only that library and the projects that depend on it re-run.

When to use it, when to avoid it, alternatives and cost impact

Distribution pays off when a typical pull request affects a large part of the graph, the expensive tasks split into many similar-sized pieces, and the graph is accurate enough that a cache hit produced on one agent can be trusted on another. It is the wrong tool when most changes touch two or three projects, because provisioning agents, installing dependencies and fetching artifacts can take longer than running everything on one machine. It also does little when a single unsplittable task sets the critical path. Heavily stateful integration tests are a third poor fit: if they share a schema or fixed ports, they cannot run concurrently without isolation work first.

Illustrative estimate (assumed numbers, not a benchmark). These figures are invented to show the arithmetic:

• One machine runs a PR in 30 minutes.
• 6 agents finish in 11 minutes of wall time each, including 3 minutes of setup.
• The team opens 40 PRs a day.

Extra agent-minutes per day = PRs × (N × T_dist − T_single) = 40 × (6 × 11 − 30) = 1,440.

Wall-clock minutes saved per day = PRs × (T_single − T_dist) = 40 × 19 = 760.

Multiply each figure by its price per minute. Agent-minutes are priced from your CI bill. For saved minutes, use what an engineer's waiting time costs you. Then subtract the ongoing cost of debugging failures that span several machines.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx distributed task execution, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.