Nx Distributed Task Execution: Nx Agents vs Manual CI Sharding

How Nx distributed task execution works

A large monorepo's CI run is mostly a long list of build, test, and lint targets across many projects. On one machine, even with nx affected trimming the list and local parallelism enabled, the run is limited by that machine's cores and memory. Both approaches in this article spread that work across several CI machines. They differ in who decides which machine runs which task, and when that decision is made.

The dynamic model

With Nx distributed task execution, one main job checks out the repository, computes the affected projects, and builds the task graph. This is the full set of targets to run and the dependency edges between them, such as app:build depending on lib:build. The main job does not split this list up front. It keeps a queue of tasks whose dependencies have all finished and hands them out to agents, which are separate CI machines running the same checkout, as each agent becomes free. When an agent finishes a task, its dependents may become ready and enter the queue.

This is why the model balances load well. A slow test suite holds one agent while the others keep pulling work. No agent sits idle because a precomputed shard was smaller than the others.

Moving outputs between machines

Agents do not share a disk. When lib:build finishes on agent A, its declared outputs are uploaded to the remote cache. Before agent B runs app:build, it restores those outputs from the cache. This keeps agents stateless and interchangeable, but correctness depends on every target declaring its outputs accurately.

Copy+-------------------------------------------------------------+
|  MAIN JOB: checkout -> nx affected -> build task graph      |
|                                                             |
|  lib:build -----> app:build -----> app:test                 |
|                                                             |
|  ready queue: tasks whose dependencies have all finished    |
|  task done on agent -> its dependents join the queue        |
+-------------------------------------------------------------+
        |                     |                     |
        | ready task          | ready task          | ready task
        v                     v                     v
+---------------+     +---------------+     +---------------+
|   AGENT A     |     |   AGENT B     |     |   AGENT C     |
| lib:build     |     | app:build     |     | app:test(slow)|
+---------------+     +---------------+     +---------------+
        |                     ^                     |
        | upload outputs      | restore outputs     | upload outputs
        v                     |                     v
+-------------------------------------------------------------+
|                         REMOTE CACHE                        |
|         declared outputs of finished tasks                  |
+-------------------------------------------------------------+

The main job builds the task graph and hands each ready task to the next free agent, while agents exchange task outputs only through the remote cache, never a shared disk.

How manual CI sharding works and how to implement it

Manual sharding removes the coordinator. Instead of one orchestrator handing out tasks, you define N identical CI jobs, usually through your provider's matrix feature. You give each job two values, a shard index and a shard total (for example SHARD_INDEX=2 and SHARD_TOTAL=4). Each job then decides what to run without talking to the others.

A typical shard script does four things:

1. Compute the affected set. It runs nx show projects --affected -t test, passing the same --base and --head commits as every other shard. Every job must produce an identical list, so the base commit has to be resolved the same way in all of them.
2. Sort the list. The output order is not something you should depend on. Sorting alphabetically makes the slicing deterministic.
3. Pick a slice. It keeps the projects whose position modulo SHARD_TOTAL equals SHARD_INDEX. A stable hash of the project name also works, and it keeps assignments steadier as projects are added.
4. Run the slice. It calls nx run-many -t test -p <slice> and exits cleanly when the slice is empty.

The job layout is one matrix job per target, such as lint, test and build, each fanned out to N shards. Some teams add a final job that waits on all shards and reports one status.

This design asks for very little: no external service, and nothing beyond shell and the Nx CLI. The cost is ownership. The script encodes decisions about base-commit resolution, ordering, empty slices, and which targets get sharded. Anyone who changes CI or adds a target has to understand those decisions, and the script has no test suite beyond your pipeline.

Copy#!/usr/bin/env bash
# shard.sh: run one deterministic slice of the affected projects for one target.
# Usage: SHARD_INDEX=2 SHARD_TOTAL=4 NX_BASE=<sha> NX_HEAD=<sha> ./shard.sh test
# Job layout: one matrix job per target (lint, test, build), each fanned out to
# SHARD_INDEX=0..SHARD_TOTAL-1, plus an optional final job that waits on all shards.
set -euo pipefail

TARGET="${1:?usage: shard.sh <target>}"
: "${SHARD_INDEX:?zero-based shard index required}"
: "${SHARD_TOTAL:?total shard count required}"
# Every shard must resolve the same base/head, or the slices will overlap or leave gaps.
: "${NX_BASE:?base commit required}"
: "${NX_HEAD:?head commit required}"

# 1. Compute the affected set. 2. Sort with a fixed locale so the order is identical in every job.
mapfile -t projects < <(
  npx nx show projects --affected -t "$TARGET" --base="$NX_BASE" --head="$NX_HEAD" | LC_ALL=C sort
)

# 3. Keep the projects whose position modulo SHARD_TOTAL equals SHARD_INDEX.
slice=()
for i in "${!projects[@]}"; do
  if (( i % SHARD_TOTAL == SHARD_INDEX )); then
    slice+=("${projects[$i]}")
  fi
done

# An empty slice is normal when few projects are affected, so exit successfully.
if (( ${#slice[@]} == 0 )); then
  echo "shard ${SHARD_INDEX}/${SHARD_TOTAL}: no projects for '${TARGET}'"
  exit 0
fi

# 4. Run the slice. run-many accepts a comma-separated project list.
project_list=$(IFS=,; echo "${slice[*]}")
echo "shard ${SHARD_INDEX}/${SHARD_TOTAL}: ${TARGET} on ${project_list}"
npx nx run-many -t "$TARGET" -p "$project_list"

A provider-neutral shard script that computes the affected projects, sorts them, keeps the slice matching its shard index, and runs nx run-many on that slice. It exits cleanly when the slice is empty.

Dependent targets and load balancing: where static shards lose time

A static shard runs a fixed list of projects, and every target those projects depend on must finish inside that shard or before it. Suppose api and worker both need core:build, but the slicing script puts them in different buckets. Each bucket then has to build core itself, or the pipeline needs a serial build stage that runs before any shard and hands off artifacts. The first wastes compute. The second puts the build on every run's critical path, including runs where only one test needed it.

A static pipeline finishes only when its slowest bucket does. Splitting by project count assumes all projects cost the same, and they rarely do. Balancing by measured duration fixes this, but you then have to collect per-task timings, store them between runs, and keep them current as the code changes. A dynamic scheduler avoids that work because it assigns each task to the next free agent once the task's dependencies are done.

Illustrative estimate (assumed numbers, not a benchmark). Assume 4 machines and these test durations in minutes: api 10, web 9, worker 8, auth 7, billing 5, search 4, admin 3, docs 2. Assume core:build takes 6 minutes and is needed by api and worker.

For static shards, wall clock = max over shards of Σ(task durations in that shard, including duplicated builds).

For dynamic agents, wall clock ≈ max(total task time ÷ agents, critical path) + scheduling overhead. In this example that is max(54 ÷ 4 = 13.5, 6 + 10 = 16) = 16. Both calculations ignore agent startup time and cache hits.

The remote cache and declared inputs and outputs

Both models depend on a remote cache, though for slightly different reasons. With Nx Agents, the orchestrator assumes that any task's outputs can be restored on whichever agent needs them next. If app:build runs on agent 3 and depends on lib:build from agent 1, agent 3 pulls the library's artifacts from the cache rather than rebuilding them. With manual shards, there is no coordinator. Each shard must compute its own dependency graph, so without a shared cache every shard rebuilds the libraries that its assigned projects depend on. That repeated work is often what erases the expected speedup.

The cache can only be as correct as the inputs and outputs you declare for each target. Nx hashes the declared inputs to produce the cache key and stores only the declared outputs. If a build reads an undeclared file, such as a root tsconfig or an environment variable, a change to that file will not change the hash. The cache then returns a stale artifact. If a target writes to a directory that is missing from outputs, that directory is never stored or restored.

On a single machine, missing outputs stay hidden because the files are still sitting on disk from the earlier task. On a fresh agent, that directory does not exist. The downstream task then fails, or it silently reads whatever an older checkout left behind. Distribution turns these latent configuration bugs into visible, often intermittent, failures.

The example below declares production inputs, includes dependencies' inputs through ^production, and lists the build output path explicitly.

Copy// nx.json (Nx parses this file with comments allowed)
{
  "namedInputs": {
    // Everything in the project, plus workspace-level files that affect every build
    "default": ["{projectRoot}/**/*", "sharedGlobals"],
    "sharedGlobals": [
      // Root tsconfig is read by every build, so it must be part of the hash
      "{workspaceRoot}/tsconfig.base.json",
      // An env var read at build time changes the output, so hash it too
      { "env": "API_BASE_URL" }
    ],
    // Production builds ignore test files, so editing a spec does not bust the build cache
    "production": [
      "default",
      "!{projectRoot}/**/*.spec.ts",
      "!{projectRoot}/jest.config.ts"
    ]
  },
  "targetDefaults": {
    "build": {
      // Build dependencies first; on another agent their outputs come from the remote cache
      "dependsOn": ["^build"],
      // Hash this project's production files and the production files of its dependencies
      "inputs": ["production", "^production"],
      // Only this path is stored and restored; anything written elsewhere is lost on a fresh agent
      "outputs": ["{workspaceRoot}/dist/{projectRoot}"],
      "cache": true
    }
  }
}

An nx.json that hashes the root tsconfig, an env var and dependencies' production inputs, and declares the build output path so a fresh agent can restore it.

Sizing the agent count: total task time vs critical path

Two numbers set the floor and ceiling for any agent count. Total task time is the sum of every task's duration in the affected graph. It is what a single machine would spend running everything serially. The critical path is the longest chain of dependent tasks, for example build of a shared library, then build of the app, then its e2e. No number of agents can finish faster than the critical path, because each task in that chain must wait for the one before it.

A rough model for wall-clock time with N agents:

wall time ≈ agent startup + max(critical path, total task time ÷ N)

compute cost ≈ N × wall time

Agent startup covers checkout, dependency install and cache restore. Every agent pays it, so it multiplies with N.

Illustrative estimate (assumed numbers, not a benchmark):

• Assumed inputs: total task time 60 min, critical path 12 min, startup 3 min.
• With 5 agents: 3 + max(12, 12) = 15 min wall time, and 5 × 15 = 75 agent-minutes.
• With 10 agents: 3 + max(12, 6) = 15 min wall time, and 150 agent-minutes. You pay double the compute for no speedup.

The useful ceiling is therefore roughly total task time ÷ critical path. Beyond that, extra agents sit idle or wait on dependencies.

Small affected sets push this the other way. Suppose a change touches three projects: 4 min of tasks, with a 3 min critical path. Distribution can save at most one minute. That saving disappears once you add coordination latency or one agent with a slow install, and you still pay startup on every agent.

For pull requests that usually touch little, run tasks on a single machine below a threshold of affected work. Reserve distribution for large affected graphs and full main-branch runs.

Operational behavior: logs, retries, flaky tasks and agent failures

With manual sharding, every shard is an ordinary CI job. Logs, exit codes and the "re-run job" button all come from your CI system, and a failure stays inside one shard. The task-to-shard assignment is computed deterministically, so a re-run puts the same task back on the same shard.

With Nx Agents, the main job orchestrates and the agents execute. Task output is collected back into the main job and grouped per task in the Nx Cloud run view. Environment evidence stays on the agent that ran the task: setup steps, disk pressure, memory. That evidence is not in the main job's log.

This split is what makes debugging harder. Suppose a test fails because of leftover state, a port collision or a slow disk. Under dynamic assignment, that test can run on a different machine next time, next to different tasks, and pass. To debug it:

• Record which agent ran the failing task.
• Reproduce the task in isolation with the cache bypassed.

Nx Cloud can flag tasks that both pass and fail with identical inputs as flaky, and can retry them. Treat that flag as a signal to fix the test, not as the fix.

In either approach, set a timeout on the main job so a stalled run fails instead of hanging.

Choosing between Nx Agents and manual sharding: trade-offs and decision checklist

Both approaches cost engineering time. Nx Agents moves that cost to a vendor. Manual sharding keeps it on your team. The useful question is which ongoing cost is smaller for your repo.

Nx Agents adds an external orchestration service to every pipeline. If that service is slow or unavailable, your CI is too. It is billed separately from your CI provider, under the vendor's current pricing model. It also receives task metadata, and often cached artifacts, from outside your network, so security or compliance review may block it. Manual sharding sends nothing outside your network, and you can debug it by reading your own CI files. Its weakness is that shard balance drifts as projects grow and the dependency graph changes. Nothing fails when that happens. Wall-clock time just creeps up until someone retunes the split. Both lock you in somewhat: CI behavior built around Nx is work to undo if you later leave Nx.

Decision checklist

• If most rows favor one column, start there.
• If compliance rules out the external service, choose manual sharding. No other factor outweighs that.
• If you choose manual sharding, put shard rebalancing on a recurring schedule. Shards drift without any visible failure, so a regular check is the only way to catch it.
• If you choose Nx Agents, decide in advance what CI does when the orchestration service is unavailable.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx distributed task execution, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.