How does manual CI sharding work, and why does it degrade over time?
Manual sharding splits the work before any job starts. You define a CI matrix of N parallel jobs. Each job receives its shard index and the total count as environment variables. A typical job computes the affected projects for the current change and sorts that list so every job sees the same order. It then keeps only the projects whose position modulo N equals its own index, and runs the test or build target on that slice. Other common schemes hash project names into buckets or read from a hand-maintained list of projects per shard. All of them assign work without knowing how long any task will take.
The wall time of the pipeline is the duration of the slowest shard, and shards drift out of balance for several reasons:
- Task durations are uneven. A modulo split balances the number of projects per shard, not the time each one takes.
- One slow project pins a shard. If a single test suite takes longer than an even share of the total, the shard that holds it finishes last no matter how the rest are arranged.
- Dependency edges cross shard boundaries. A shard that tests a project may first have to build upstream libraries that belong to another shard. That work is repeated on both machines.
- Lists go stale. Projects that are added or renamed after the list was written either land in a default shard or are not run at all.
Illustrative estimate (assumed numbers, not a benchmark): wall time ≈ max over shards (setup + sum of task durations). Assume 4 shards, 40 minutes of total task time, and one 18-minute project. The ideal split is 40 / 4 = 10 minutes per shard, but the actual wall time is at least 18 minutes plus setup.
name: ci
on: pull_request
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [0, 1, 2, 3] # N is fixed here, before any timing data exists
env:
SHARD_INDEX: ${{ matrix.shard }}
SHARD_TOTAL: 4
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # full history so the affected calculation can diff against main
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
- run: npm ci
- name: Test this shard's slice of affected projects
run: |
# Every shard computes the same sorted list so the slices don't overlap
npx nx show projects --affected --base=origin/main --head=HEAD | sort > affected.txt
# Keep projects whose position modulo SHARD_TOTAL equals this shard's index.
# This balances project count, not duration: one slow suite still pins its shard.
MINE=$(awk -v i="$SHARD_INDEX" -v n="$SHARD_TOTAL" '(NR-1) % n == i' affected.txt | paste -sd, -)
if [ -z "$MINE" ]; then
echo "No projects assigned to shard $SHARD_INDEX"
exit 0
fi
# Upstream dependencies of these projects are built here as well,
# even when another shard builds the same libraries.
npx nx run-many -t test --projects="$MINE"
A GitHub Actions matrix of four jobs. Each job lists the affected Nx projects in sorted order, keeps the ones whose position modulo 4 matches its shard index, and runs their tests, so all work is assigned without any runtime information.
How does Nx Agents distributed execution work?
Nx Agents replaces the static split with a scheduler. One CI job, the main job, runs your Nx command with distribution enabled. It computes the project and task graph for the affected projects and hands it to Nx Cloud, which acts as the coordinator. Separate CI jobs, the agents, connect to Nx Cloud and ask for work. You still provision these agents in your CI provider by defining the jobs, their machine size, and how many run. Nx Cloud does not create the agent machines for you in this model.
Manual sharding (assigned up front) Nx Agents (pulled at runtime)
shard 1: [A][B][C] -> done ┌─ task queue (Nx Cloud) ─┐
shard 2: [D][E] -> idle │ D E F G H ... │
shard 3: [F.........] -> still busy └──┬──────┬──────┬────────┘
shard 4: [G][H] -> idle ▼ ▼ ▼
agent1 agent2 agent3
(each takes the next task
when it finishes one)
The pull model fixes the drift problem from the previous section. Nobody has to keep a duration table current, because an agent that finishes early takes the next ready task. A slow machine simply ends up running fewer tasks.
A task is not offered to agents until its dependencies have finished. A build of a library has to complete before the test targets that consume it become available. When that build runs on agent 1 and the dependent test lands on agent 3, agent 3 does not rebuild the library. It restores the library's outputs from the remote cache. Agents return terminal output and declared outputs to the main job, so the main job reports results and can use the artifacts as if it had run every task itself.
┌───────────────────────────────────────┐
│ Main job │
│ runs nx with distribution enabled, │
│ computes project + task graph │
└─────┬──────────────────────▲──────────┘
│ task graph │ terminal output,
▼ │ declared outputs
┌────────────────────────────┴────────────────────────────────┐
│ Nx Cloud (coordinator) │
│ │
│ task queue: a task is offered only after its deps finish │
│ [lib:build] [lib:lint] [app:test: waits on lib:build] │
│ │
│ remote cache: outputs written by one agent, restored by │
│ any other agent instead of rebuilding │
└──────▲──────────────────▲──────────────────▲────────────────┘
│ pull next task │ pull next task │ pull next task
│ push outputs │ (repeats) │ restore outputs
┌──────▼─────┐ ┌──────▼─────┐ ┌──────▼─────┐
│ agent 1 │ │ agent 2 │ │ agent 3 │
│ lib:build │ │ lib:lint │ │ app:test │
│ stores │ │ │ │ restores │
│ outputs │ │ │ │ lib outputs│
└────────────┘ └────────────┘ └────────────┘
The main job and agent jobs run in your CI provider: you define them,
their machine size and how many agents run.
The main job sends the task graph to Nx Cloud, agents pull ready tasks in dependency order and share outputs through the remote cache, and results flow back to the main job.
Why does remote cache correctness become a hard requirement?
With manual sharding, a shared cache is an optimization. Each shard checks out the repository and builds whatever it needs. If a cache entry is missing or wrong, the shard rebuilds, and the worst case is lost time. Distribution removes that safety net. The remote cache is how a library built on agent A reaches the test task on agent B, so a wrong hash means agent B receives the wrong files, or none.
On a single machine, undeclared state stays hidden. A build writes dist/, and the next task finds those files on disk whether or not anything declared them. Across agents, only declared outputs travel. Suppose a target writes to tmp/generated without listing it. It succeeds on its own agent, and then a dependent task fails on another agent with a missing-file error that nobody can reproduce locally.
In a target's configuration in project.json, or in targetDefaults, declare three things:
inputs: the files that affect the result. These are often named inputs such asproduction, plus^productionfor the files of dependencies.- Environment entries:
{ "env": "API_URL" }for each environment variable the task reads. outputs: paths such as{projectRoot}/dist.
Each omission fails differently. A missing input lets two different builds share a hash, so a stale artifact is replayed. A missing env entry can serve a staging build to a production job. Some tasks depend on machine state, such as local tool versions, absolute paths or the current time. Fix those, or mark them non-cacheable, before you distribute them.
// apps/api/project.json (Nx parses this file as JSON with comments)
{
"name": "api",
"targets": {
"build": {
"executor": "nx:run-commands",
"options": { "command": "node tools/build.js apps/api" },
"cache": true,
"inputs": [
// This project's source files, using the 'production' named input from nx.json
"production",
// Production files of every dependency, so a library change produces a new hash
"^production",
// The build inlines this value. Without this entry, staging and prod builds share a hash
{ "env": "API_URL" }
],
// Only these paths are stored in the cache and restored on other agents
"outputs": [
"{projectRoot}/dist",
// A downstream task reads generated code from here, so declare it as well
"{projectRoot}/tmp/generated"
]
},
"stamp-build-info": {
"executor": "nx:run-commands",
// Embeds the current time and host tool versions, so the result is not reproducible
"options": { "command": "node tools/stamp.js apps/api" },
// Never replay a cached result from another machine
"cache": false
}
}
}
A project.json build target that declares source, dependency and environment inputs plus every output directory, and a machine-dependent target marked non-cacheable.
Why can one large test target still dominate wall time?
Nx schedules work at the task level: one target on one project, such as api:test. A task is the smallest unit it can move between machines. Manual sharding has a similar floor, set by whatever you put in a single job. If api:test takes 18 minutes, adding agents cannot finish it in less than 18 minutes. The pipeline's wall time is bounded below by the longest chain of dependent tasks (the critical path), however much parallelism is available.
A useful lower bound is:
wall time ≥ max(critical path, total task time ÷ agents) + per-agent setup
Illustrative estimate (assumed numbers, not a benchmark): Assume 40 tasks totalling 60 minutes, 8 agents and 2 minutes of setup. One test target takes 18 minutes and nothing depends on it. Without splitting, the bound is max(18, 60 ÷ 8) + 2 = 20 minutes. Now split that target into six groups of about 3 minutes each. The bound becomes max(3, 7.5) + 2 = 9.5 minutes. The extra agents only started paying off after the split.
If one target dominates, give the scheduler smaller units to balance:
- Split the project along real domain boundaries. This also narrows what "affected" selects on each change.
- Add separate targets, such as unit versus integration, or per-directory test groups.
- Generate per-file or per-group test targets, which some Nx plugins can infer for end-to-end and unit test runners.
Before adopting distribution, examine the shape of your task graph. Teams sometimes expect a large speedup, but their graph is one long task surrounded by short ones. In that case distribution adds machine cost and setup overhead without reducing the time engineers wait.
How should you compare the two fairly, and what does it cost?
Compare the approaches on three numbers rather than on wall time alone:
- Critical-path length: the longest chain of dependent tasks. No amount of distribution goes below it.
- Total machine-minutes: what you pay for, including idle time on agents that are waiting for work.
- Setup share: per-machine checkout, dependency install, toolchain, and cache restore as a fraction of wall time. Every added machine pays this again.
The simplified formulas, with N machines, setup S, and total task work W:
- Dynamic wall time ≈ S + max(critical path, W / N)
- Static-shard wall time ≈ S + heaviest shard
- Machine-minutes ≈ N × S + W, plus any idle time
Illustrative estimate (assumed numbers, not a benchmark). These figures assume S = 3 min, W = 40 min, a critical path of 8 min, and static shards that drifted so the heaviest one holds 14 min of work.
| Setup | Wall time | Machine-minutes |
|---|---|---|
| One machine | 3 + 40 = 43 | 43 |
| 4 static shards | 3 + 14 = 17 | 12 + 40 = 52 |
| 4 dynamic agents | 3 + max(8, 10) = 13 | 12 + 40 = 52 |
Now assume a highly cached run where W drops to 6 min and the critical path is 2 min. One machine finishes in 9 min and uses 9 machine-minutes. Four agents finish in about 3 + 2 = 5 min but use 12 + 6 = 18 machine-minutes. That saves 4 minutes of wall time and doubles the machine cost, and setup accounts for most of the wall time.
Machine cost is only part of the picture. Multiply the minutes engineers spend waiting on a run by the number of runs per day and by the number of engineers who block on those runs. Then add the hours someone spends rebalancing static shards whenever test timings drift. Waiting time and rebalancing time usually cost more than compute, but only if the reduction in wall time is real for your task graph.
What operational differences will engineers notice day to day?
With manual sharding, everything lives in your CI provider. Each shard is a job with its own log, and a failure points to one job you can rerun. With Nx Agents, the CI job mostly waits on the orchestrator, so task logs and failure details are often clearer in the Nx Cloud view than in the CI console. Engineers end up checking two places.
Retries also work differently. Static shards retry whole jobs, so one flaky test reruns the entire slice. An orchestrator can retry individual tasks, which is cheaper. It can also hide flakiness unless someone watches the retry counts. When an agent dies mid-task, the orchestrator can schedule the work elsewhere. A dead static shard just fails the pipeline. Reproducing failures is harder with agents, though, because you need the same task inputs and environment, not "shard 3 of 8."
| Concern | Manual sharding | Nx Agents |
|---|---|---|
| Logs and failures | CI provider only | Split across CI and Nx Cloud |
| Retry unit | Whole shard job | Individual task |
| Agent loss | Shard fails | Work can be reassigned |
| External dependency | None | Nx Cloud availability and network egress |
| Compliance review | Existing CI scope | New vendor handling cache artifacts and logs |
| Exit cost | Low, but scripts rot | Rework of orchestration and caching |
Nx Cloud also becomes part of your failure surface. An outage or a blocked egress rule can stall CI. Security teams will want to review what source-derived artifacts leave your network. Usage-based pricing turns CI volume into a line item someone has to forecast. Hand-rolled scripts avoid all of this and stay portable, but they decay quietly because nobody owns them.
When should you use Nx Agents, and when should you avoid it?
Manual sharding on top of affected-only runs is enough when the graph is small, task durations are roughly uniform, or policy forbids sending build metadata and artifacts to an external service. In those cases a static split stays close to balanced. The scripts are cheap to keep correct because the inputs they depend on rarely change.
Dynamic distribution pays off when the graph is large and uneven, when dependency chains are deep, or when projects are added and reshuffled often. Static shards cannot respect build order across machines, and they go stale every time the graph changes. A scheduler that reads the graph on every run avoids both problems.
| Situation | Better fit | Why |
|---|---|---|
| Small graph, similar task times | Affected + manual shards | Static split stays balanced |
| Strict vendor or network rules | Affected + manual shards | Nothing leaves your network |
| Large graph, skewed durations | Nx Agents | Scheduler packs uneven work |
| Deep dependency chains | Nx Agents | Order is respected across machines |
| Frequently changing graph | Nx Agents | No shard lists to update |
Before adopting either approach, check the cheaper alternatives:
- One larger CI machine. This removes network and coordination overhead entirely.
- Affected-only runs without sharding. These are often fast enough on their own.
- Splitting the dominant target. This fixes the wall-time floor that no scheduler can lower.
- Test-file-level splitting in the test runner. This balances work inside a single target.
The real trade is engineering time. Nx Agents removes the ongoing work of maintaining shards. In exchange, you take on responsibility for keeping cache inputs accurate, and you accept an external service in your critical path.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nx agents distributed execution, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.