Nx Distributed Task Execution: Splitting CI Work Across Agents

How Nx distributed task execution works

In Nx distributed task execution (DTE), your CI pipeline stops being a fixed script that each machine runs end to end. It becomes a single declaration of intent. The main CI job states what needs to happen, typically nx affected across the lint, test and build targets. Nx Cloud then decides where each piece runs.

The flow

1. The main job declares the work. It computes the affected project graph and sends the resulting task graph to a coordinator in Nx Cloud. The main job does not execute those tasks itself.
2. The coordinator schedules tasks. It splits the graph into individual tasks, such as api:test or shared-utils:build, and hands them to idle agents in dependency order. A task is only dispatched once everything it depends on has finished.
3. Agents execute and upload. Each agent runs its assigned task and writes the outputs and terminal logs to the remote cache.
4. The main job reassembles results. It streams back exit codes and logs, so the output looks as though everything ran on one machine. If a task fails, the main job fails too.

Why remote caching is required

Agents do not share a filesystem. Suppose api:build depends on shared-utils:build and the two tasks land on different agents. The second agent has no local copy of the library's compiled output. It pulls that output from the remote cache instead of rebuilding it.

Without the cache, every agent would have to rebuild its dependencies from scratch. That would erase most of the time saved by parallelizing. The cache is effectively the transport layer that moves build outputs between agents.

Copy+----------------------+ (1) task graph +---------------------------------+
| Main CI job          |--------------->| Nx Cloud coordinator            |
| nx affected          |                | splits graph into tasks,        |
| -t lint test build   |                | dispatches in dependency order  |
+----------------------+                +---------------------------------+
           ^                                 |           |           |
           | (4) exit codes              (2) |           |           |
           | + terminal logs                 v           v           v
           |                            +---------+ +---------+ +---------+
           |                            | Agent 1 | | Agent 2 | | Agent 3 |
           |                            | shared- | |api:build| | api:test|
           |                            | utils:  | |         | |         |
           |                            | build   | |         | |         |
           |                            +---------+ +---------+ +---------+
           |  (3) upload outputs + logs      |         ^   |         |
           |                                 v         |   v         v
           |                  +-------------------------------------------+
           +------------------| Nx Cloud remote cache                     |
                              | task outputs + terminal logs from agents  |
                              | api:build pulls shared-utils:build output |
                              +-------------------------------------------+

The main CI job sends the affected task graph to the Nx Cloud coordinator (1), which dispatches tasks to agents in dependency order (2); agents upload outputs and logs to the remote cache and pull their dependencies' outputs from it (3), and the main job streams back exit codes and logs (4).

DTE versus static sharding and other alternatives

Distribution is one of several ways to make a large Nx workspace finish faster in CI, and the alternatives fail in different ways.

Static sharding splits projects into fixed groups before the run, usually by name or count, and gives each group its own CI job. The split ignores the dependency graph. A shard may have to build libraries that another shard is also building, or it may wait on work it cannot see. Shard sizes also drift as the codebase changes, so one slow shard ends up setting the pipeline time while the others sit idle.

One large machine running nx affected with local parallelism keeps full graph awareness and avoids any coordination. Its limits are the cores and memory of a single host. Past that point, adding tasks only makes the queue longer.

Remote caching alone skips tasks that have already run somewhere else. It helps a lot on repeat work, but cache misses still run on a single machine.

DTE keeps the graph intact and hands out tasks as agents become free and dependencies finish. It can use historical task durations to schedule long tasks early, so the run does not end with a single long task running on one agent while the rest are idle.

Who should use it and when it pays off

The strongest fit is a platform or backend team running a large monorepo where a typical pull request touches many projects. DTE helps when the affected graph is big, wide and cache-friendly. "Big" means there is enough work to share. "Wide" means many tasks can run at the same time rather than waiting on a long dependency chain. "Cache-friendly" means agents can reuse outputs instead of rebuilding them. A small repo, or one where most tasks sit on a single critical path, gives the scheduler little to work with.

Every agent has fixed overhead before it does useful work. It has to boot, check out the repo, install dependencies, and pull artifacts produced by upstream tasks on other machines. DTE only pays off when the time saved by parallel scheduling is larger than that overhead. If your affected tasks take a few minutes in total, adding nx agents can make CI slower.

Signs you are ready:

• Long serial CI time. The pipeline is dominated by task execution, not by setup or deployment steps.
• Many independent projects. The task graph stays wide even after you filter to affected projects, so several agents have work at once.
• Healthy cache hit rates. Remote caching already works reliably. That shows tasks are reproducible, which distribution depends on.
• Correct inputs and outputs. Each task declares what it reads and what it writes. Without this, an agent can miss files another agent produced, and failures become hard to reproduce.

If the last two points are shaky, fix caching first. DTE will expose those gaps rather than hide them.

Setting up the main job and agents

A DTE setup has two roles. The main job opens a CI run with Nx Cloud and declares how agents will be provisioned. It then runs nx affected for the targets you want, such as lint, test and build, and stops the agents when it finishes. It does not execute most of the work itself. Instead, it hands tasks to Nx Cloud, which schedules them across agents and streams logs and outputs back. Agent jobs start the Nx Cloud agent process and wait for assignments. Exact command names and options change between releases, so check the current Nx docs before writing your pipeline.

Under either model, every machine must compute the same affected set and resolve the same task hashes. Mismatched SHAs or toolchains cause cache misses at best and wrong results at worst.

Each distributable target needs explicit inputs and outputs (see the example below). The inputs determine the task's hash. The outputs determine which files Nx copies back to the machines that need them.

Keep deploys, migrations, publishes and other non-cacheable or side-effecting tasks on the main job. Run them after the distributed work completes, so that they execute once and in a known order.

Copy{
  "namedInputs": {
    "default": ["{projectRoot}/**/*", "sharedGlobals"],
    "production": [
      "default",
      "!{projectRoot}/**/*.spec.ts",
      "!{projectRoot}/jest.config.ts"
    ],
    "sharedGlobals": ["{workspaceRoot}/package-lock.json", "{workspaceRoot}/tsconfig.base.json"]
  },
  "targetDefaults": {
    "build": {
      "dependsOn": ["^build"],
      "inputs": ["production", "^production"],
      "outputs": ["{projectRoot}/dist"],
      "cache": true
    },
    "test": {
      "inputs": ["default", "^production"],
      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"],
      "cache": true
    },
    "lint": {
      "inputs": ["default", "{workspaceRoot}/.eslintrc.json"],
      "outputs": [],
      "cache": true
    },
    "deploy": {
      "dependsOn": ["build"],
      "cache": false
    }
  }
}

An nx.json target configuration where build, test and lint declare explicit inputs (which files feed each task's hash) and outputs (which files Nx copies back from agents), while deploy is marked non-cacheable so it runs only on the main job after the distributed work finishes.

Making tasks distributable: hermeticity, splitting and environment parity

A task is only safe to distribute if its cache key and output contract match what it actually reads and writes. On a single machine, an undeclared output still sits on disk, so the next task finds it anyway. On agents, only declared outputs are restored, so a downstream task on another machine fails or builds against nothing. Missing inputs fail more quietly: the hash ignores a file the task depends on, and an agent gets a stale cache hit. Distribution surfaces both problems sooner because tasks stop sharing a working directory.

Plan for refactors where tests share state:

• Shared database. Give each task its own schema or database name. Tests that assume one shared instance will race once they run concurrently on other agents.
• Fixed ports. Replace hard-coded ports with ports allocated at runtime.
• Shared directories. Replace writes to a common temp or report directory with per-project paths declared as outputs.

Large e2e or test targets limit parallelism because each target is one indivisible task. Task atomization splits a target into per-file or per-group tasks that Nx can schedule independently. Some Nx plugins generate these split targets through inferred configuration. Support depends on the plugin and its version, so check what targets your setup actually produces before you depend on them.

Environment parity means agents need the same toolchain versions, lockfile-installed dependencies, environment variables and secrets as the main job. Environment variables declared as task inputs feed into the hash. A value that differs per machine, such as a timestamp or a runner ID, causes a cache miss on every run.

Copy{
  "namedInputs": {
    "sharedGlobals": [
      { "runtime": "node --version" },
      { "env": "NODE_ENV" }
    ],
    "default": ["{projectRoot}/**/*", "sharedGlobals"],
    "production": [
      "default",
      "!{projectRoot}/**/*.spec.ts",
      "!{projectRoot}/test-results/**/*"
    ]
  },
  "targetDefaults": {
    "build": {
      "cache": true,
      "dependsOn": ["^build"],
      "inputs": ["production", "^production"],
      "outputs": ["{projectRoot}/dist"]
    },
    "test": {
      "cache": true,
      "dependsOn": ["^build"],
      "inputs": ["default", "^production", { "env": "TEST_DB_ENGINE" }],
      "outputs": [
        "{workspaceRoot}/coverage/{projectRoot}",
        "{projectRoot}/test-results"
      ]
    }
  }
}

An nx.json fragment that hashes the Node version and only stable environment variables (never runner IDs or timestamps), and declares per-project coverage and report directories as outputs, so agents restore every file that downstream tasks need instead of relying on a shared directory.

Sizing the agent pool and impact on cost and engineering time

Illustrative estimate (assumed numbers, not a benchmark)

Assume a run with 120 minutes of serial task time, a critical path of 15 minutes (the longest chain of dependent tasks), and 3 minutes of setup per agent for checkout, dependency install and cache restore. A rough lower bound for wall-clock time with N agents is:

wall-clock ≈ setup + max(critical path, serial time ÷ N)

Agents are usually billed until the run finishes, so total compute is roughly N × wall-clock.

Beyond 8 agents, the critical path sets the floor. Extra agents finish their share early and then sit idle, waiting on dependencies that only one machine can run. Total compute rises at every step because setup runs once per agent and idle time is still paid for. Real runs perform worse than this bound, because the scheduler cannot pack tasks perfectly. The point where adding agents stops helping therefore arrives sooner than the formula suggests.

Engineering time is saved in two places. Developers wait less for feedback on each push, and nobody has to rebalance hand-written shard lists as the repository grows. It is spent up front on fixing non-hermetic tasks, splitting long tasks that sit on the critical path, and teaching the team to read distributed logs.

Pull serial time, critical path and setup cost from real runs in your own repository. Then choose the agent count where the drop in wall-clock time starts to level off, rather than the largest pool your budget allows.

When to avoid it, debugging and rolling it out safely

When DTE is a poor fit

Skip it if most pull requests affect only a handful of projects. In that case agent startup, checkout and dependency installation can take longer than the tasks themselves. If your goal is lower spend rather than faster feedback, adding machines usually raises the bill. A long critical path, such as one slow build that everything else waits on, will not shrink with more agents. Split or speed up that task first. Non-hermetic tasks are also a poor fit, because their results depend on which agent happened to run them.

DTE also depends on an external coordination service that receives task artifacts and logs. Send that data flow through security and compliance review before adopting it. If a self-hosted option is available, evaluate it against your actual requirements rather than assuming it solves the problem.

Debugging

Visibility is split between two places. CI logs show agent setup and environment details. The Nx Cloud view shows which agent ran each task and when. Use the Nx Cloud view to find the agent that ran the failing task, then open that agent's CI log.

In the task timeline, look for two patterns:

• Idle agents point to an oversized pool or a dependency bottleneck.
• A single long chain of tasks is your critical path.

To reproduce a failure locally, run the single task with nx run project:target. Match the commit, environment variables and tool versions the agent used.

Rolling out

Start with lint or unit tests, which are cheap and well isolated. Compare wall-clock time and compute cost against your current pipeline over many runs, not one. Expand to builds and end-to-end tests only once cache hit rates and agent use look healthy.

Copy#!/usr/bin/env bash
# Reproduce a task that failed on a DTE agent by running only that task locally.
# Fill in the values from the agent's CI log and the Nx Cloud task view.
set -euo pipefail

FAILED_SHA="3f9c2ab"          # commit the agent checked out (placeholder)
PROJECT="payments-api"         # project of the failing task (placeholder)
TARGET="test"                  # target of the failing task (placeholder)

# 1. Use the exact commit the agent ran, not your branch tip.
git fetch origin
git checkout "$FAILED_SHA"

# 2. Install dependencies from the lockfile, as CI does.
npm ci

# 3. Print tool versions and compare them with the agent's CI log.
node --version
npm --version
npx nx report

# 4. Copy any environment variables the task reads from the agent's
#    CI log. Use dummy values for secrets.
export NODE_ENV="test"
export API_BASE_URL=""

# 5. Run only the failing task and bypass the cache so it really runs,
#    instead of replaying a cached result.
npx nx run "$PROJECT:$TARGET" --skip-nx-cache

# If the task passes here but fails on the agent, it may not be hermetic.
# Diff the environment and tool versions between the two machines.

A shell script that reproduces a failed DTE task locally by checking out the agent's commit, matching its environment and tool versions, and running that single task with the Nx cache bypassed.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx distributed task execution, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.