What is a flaky task in Nx, and how does task hashing make it detectable?
An Nx task is a single unit of work identified by three parts: a project, a target, and an optional configuration. api:test and api:test:ci are different tasks. Before running one, Nx computes a hash from everything declared as affecting its result:
- Source files matched by the target's
inputs(oftennamedInputsdefined innx.json) - Dependencies, meaning the hashes of upstream projects and the resolved versions of external packages
- Runtime inputs, the output of commands such as a Node version check
- Environment inputs, the values of declared environment variables
If none of these change, the hash stays the same. The hash serves as the cache key and also as an identity for the task's inputs.
That identity is what makes flakiness detectable. If the same hash has produced a pass and a fail, the code did not cause the failure, because nothing Nx tracks was different between the two runs. Nx Cloud therefore classifies the task as flaky rather than broken.
sources ─┐
deps ─┼─► hash 7f3a… ─┬─► run #1 (CI agent A): PASS
runtime ─┤ └─► run #2 (CI agent B): FAIL
env ─┘ │
▼
same inputs, different outcome → FLAKY
Detecting this requires history from many runs. A local Nx cache stores outputs from one machine's past runs. It cannot tell that the same hash failed on a CI agent yesterday. Nx Cloud collects remote cache entries and run results across the team and CI, so it can see both outcomes on one hash. Without Nx Cloud, you would need to build this tracking yourself, for example by recording each task's hash and exit status from CI logs and flagging any hash that has both outcomes.
Task identity: project + target + configuration (e.g. api:test:ci)
+----------------------+
| Source files |--+
| (inputs/namedInputs) | |
+----------------------+ |
| Dependencies |--+
| (upstream, packages) | | +------------------+
+----------------------+ +----->| Task hash 7f3a |
| Runtime inputs |--+ +--------+---------+
| (command output) | | |
+----------------------+ | +-------+--------+
| Environment inputs |--+ | |
| (declared env vars) | v v
+----------------------+ +-------------+ +-------------+
| Run #1 | | Run #2 |
| CI agent A | | CI agent B |
| PASS | | FAIL |
+------+------+ +------+------+
| |
v v
+-----------------------------------+
| Shared run history (Nx Cloud) |
| hash 7f3a: {PASS, FAIL} |
+-----------------+-----------------+
|
v
+-----------------------------------+
| Same inputs, different outcome |
| => classified FLAKY, not broken |
+-----------------------------------+
Declared inputs produce one task hash; when shared run history records both a pass and a fail for that hash, the task is classified as flaky rather than broken.
Why do under-declared inputs produce false flaky signals?
That detection rests on one assumption: the hash captures everything that can change the result. When a test reads something Nx does not hash, two runs share a hash while running against different conditions. A pass followed by a fail on that hash looks like flakiness, but the test may be deterministic and the hash incomplete.
The usual culprits are an environment variable such as DATABASE_URL or a feature flag, the system clock, a shared staging service, a fixture file under {workspaceRoot} that the project's inputs don't cover, and the Node or tool version on the runner. The reverse failure is worse. If a fixture changes but isn't an input, the hash stays the same and Nx replays a cached pass, so a real regression never runs.
To audit a project, list every external thing its tests read, then compare that list with the resolved inputs for the test target. nx show project <name> prints them. Tighten the gaps in namedInputs in nx.json and in the target's inputs in project.json:
- Add file globs for shared fixtures.
- Add
enventries for variables that change behavior. - Add
runtimeentries for tool versions. - Remove dependencies on the clock and the network from the tests themselves with fake timers and local stubs. Do not try to hash them.
This is a trade-off. Inputs that are too narrow produce wrong cache hits and mislabeled flakes. Inputs that are too broad, such as hashing every environment variable or the whole workspace root, cause cache misses on unrelated changes. More tests then rerun, and real flakes take longer to surface. Input syntax has changed between Nx releases, so check it against the docs for your installed version.
{
"namedInputs": {
"default": ["{projectRoot}/**/*", "sharedGlobals"],
"sharedGlobals": [{ "runtime": "node --version" }],
"testFixtures": ["{workspaceRoot}/fixtures/shared/**/*"],
"production": ["default", "!{projectRoot}/**/*.spec.ts"]
},
"targetDefaults": {
"test": {
"cache": true,
"inputs": [
"default",
"^production",
"testFixtures",
{ "env": "DATABASE_URL" },
{ "env": "FEATURE_NEW_CHECKOUT" }
]
}
}
}
An nx.json sketch that adds shared fixture globs, the Node version and behavior-changing env vars to the test hash. Run nx show project to confirm the resolved inputs, and check the syntax against your installed Nx version.
How do Nx Cloud automatic retries compare with test-runner retry flags?
Nx Cloud retries are selective. A task is marked flaky only after Nx Cloud has recorded both a pass and a failure for the same task hash, meaning identical inputs gave different outcomes. When a task with that history fails on a distributed agent, Nx Cloud retries it. A task with no flaky history fails on the first attempt. A deterministic regression, such as a broken assertion or a type error, therefore still fails the pipeline.
Runner-level retries work differently. Jest's retryTimes, Playwright's retries setting and Cypress's retries option apply to every test in scope, whatever its history. A test broken by a new commit is retried along with a test that has been unstable for months. If the broken test passes on a later attempt, the run reports green, and the failure may only show up in a log line that nobody reads.
Both approaches contain flakiness. Neither fixes it. Retries keep CI moving while someone finds the cause. Be careful with tests that fail because of a race condition in production code, such as an unawaited write or a cache populated out of order. These look exactly like test flakiness. Once retries make them pass reliably, the pressure to investigate disappears, and the bug ships.
| Nx Cloud automatic retries | Test-runner retry flags | |
|---|---|---|
| Scope | Whole task, only if already marked flaky | Individual tests, all of them |
| Failure masking | Low: new failures fail fast | High: real regressions can pass on retry |
| Cost | Reruns the entire task | Reruns only the failing test |
| Run history | Required. A new flake must fail first | None |
How much do flaky tests cost a pipeline? An illustrative estimate
Illustrative estimate (assumed numbers, not a benchmark). Assume each task flakes independently with probability p, and a pipeline runs n affected tasks. The chance that at least one task flakes is:
P(at least one flake) = 1 − (1 − p)ⁿ
The expected number of retries per pipeline is n × p.
| Scenario (assumed) | p | n | P(≥1 flake) | Expected retries |
|---|---|---|---|---|
| Small PR, unit tests | 0.5% | 20 | ~9.5% | 0.10 |
| Large PR, unit tests | 0.5% | 60 | ~26% | 0.30 |
| Change to a shared lib | 0.5% | 120 | ~45% | 0.60 |
| E2E targets | 2% | 8 | ~15% | 0.16 |
A 0.5% flake rate looks harmless for a single task. Once a change to a widely imported library pulls 120 tasks into the affected graph, nearly half of those pipelines would go red without retries. That is why retries feel mandatory: without them, a shared-library change cannot merge.
Retries change who pays and how, but the cost remains. Consider an e2e target that takes an assumed 12 minutes. Each retry adds 12 minutes of compute. If the target is on the critical path, it also adds 12 minutes of wall-clock time. At 400 pipelines a week, 400 × 0.16 × 12 ≈ 770 compute-minutes, or about 13 hours a week, are spent rerunning e2e targets alone.
The engineering cost is larger and harder to see:
- Developers wait on slower pipelines.
- Someone eventually has to investigate the flake.
- A retry that passes can hide a real intermittent bug, such as a race condition that also occurs in production.
Retries keep the pipeline moving. They do not reduce p, and p is the only input in the formula that removes cost.
When should you split test targets to make retries cheaper?
The other lever is the size of the unit you retry. When an e2e target runs 40 spec files as one task, a single flaky spec fails the whole task. The retry then reruns all 40, and the failure log names the target rather than the file. Each retry costs the full suite runtime, and your flake data only says "the e2e target is unstable."
Nx Cloud task splitting solves both problems by turning a large test or e2e target into one task per file, plus a parent task that aggregates them. A flaky spec now fails only its own task. The retry costs one file's runtime, and the task name points directly at the unstable file. That is also the granularity you need for a quarantine list.
Costs of splitting
Splitting has overhead. Every task has to be scheduled, hashed and cached, and has its outputs stored. Every task also pays its own setup: runner startup, transpilation, and environment bootstrap such as launching a browser, starting a dev server or seeding a database.
When to split and when not to
Split when individual files are slow relative to that setup cost and the suite fails often enough that full reruns are visible in pipeline time. Long e2e suites usually meet both conditions. Avoid splitting small suites, where scheduling overhead outweighs any retry savings. Also avoid suites whose shared setup dominates runtime, because paying that setup once per file can cost more than an occasional full rerun.
Support differs by test runner, plugin and Nx version. Check your Nx documentation to confirm which runners and plugins can split targets and which configuration keys your version uses.
How do you implement a test quarantine in an Nx workspace?
Nx has no built-in test quarantine. Retries hide flakiness, but they do not isolate it, so you have to build quarantine from targets, test-runner features and CI configuration.
The most visible option is a dedicated quarantine target. Move known-flaky specs into a separate directory or naming pattern, such as *.quarantine.spec.ts. Configure the normal test target to ignore that pattern and add a test-quarantine target that runs only those files. The required CI path keeps running nx affected -t test. A separate step runs nx affected -t test-quarantine and is marked non-blocking, for example with continue-on-error in GitHub Actions or an allowed-failure setting in your CI system. Because the quarantined tests still run, you keep their pass/fail history. That history tells you when a fix has worked.
A lighter option is a skip annotation or runner tag, such as it.skip or a tag-based filter, with a comment that links to a tracked ticket. This takes less setup, but skipped tests produce no signal. Without the ticket link, they tend to stay skipped indefinitely.
| Approach | Setup effort | Still produces signal | Visibility of debt |
|---|---|---|---|
| Separate quarantine target, non-blocking CI step | Moderate | Yes | High: listed in project config |
| Runner tags filtered out of required run | Low to moderate | Only if a tagged run exists | Medium |
| Skip annotation with ticket link | Low | No | Low unless tickets are audited |
Whichever option you use, quarantine reduces effective coverage while it is active. Code paths covered only by quarantined tests can regress without blocking a merge.
# Project config (apps/web/project.json) referenced by the CI steps below:
# "test": {
# "executor": "nx:run-commands",
# "options": {
# "cwd": "apps/web",
# "command": "jest --testPathIgnorePatterns='\\.quarantine\\.spec\\.ts$'"
# }
# },
# "test-quarantine": {
# "executor": "nx:run-commands",
# "options": {
# "cwd": "apps/web",
# "command": "jest --passWithNoTests '\\.quarantine\\.spec\\.ts$'"
# }
# }
# Every *.quarantine.spec.ts file needs a comment linking its ticket, e.g. WEB-1234.
name: ci
on: pull_request
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # affected needs history to compare base and head
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
- run: npm ci
- uses: nrwl/nx-set-shas@v4 # sets NX_BASE / NX_HEAD for nx affected
# Required path: a failure here blocks the merge.
- name: Tests (required)
run: npx nx affected -t test
# Quarantined specs still run for pass/fail history, but cannot fail the job.
- name: Quarantined tests (non-blocking)
continue-on-error: true
run: npx nx affected -t test-quarantine
A quarantine setup in which the test target ignores *.quarantine.spec.ts files and a separate test-quarantine target runs only those files, with GitHub Actions running the quarantine step under continue-on-error so the specs keep producing results without blocking merges.
Who owns quarantined tests, and when should they leave quarantine?
A non-blocking target gets ignored unless a specific person is responsible for it. Each quarantine entry should record three things: a named owner (a person, not a team alias), a review or expiry date, and agreed exit criteria. Without them, quarantine turns into a slow deletion that nobody signed off on.
A common exit criterion is N consecutive clean runs on the same hash, for example 50 runs with no retry. Running against the same hash matters because the inputs stay fixed, so a clean streak reflects the test's behaviour rather than a changing codebase. The limit is that a test can stop flaking because something outside it changed, such as a faster runner pool, a new container image or lower parallelism. Before releasing a test, the owner should name the defect that was fixed. "It went green" is not a root cause.
Use the flaky task reporting in Nx Cloud as the backlog input. Rank entries by flake frequency and by how many pipelines each one blocked or retried. A test that flakes rarely but sits on the critical path of every pull request costs more than a noisy test in a rarely touched library.
Most fixes fall into a short list of causes:
- Shared state between tests, such as module-level singletons, databases or temp directories.
- Ordering dependencies that only surface when runners shuffle or shard tests.
- Port collisions under parallel execution.
- Unawaited promises or background work that outlives the test.
- Real clocks and timers instead of controllable fakes.
- Calls to external services.
Configuration keys for retries and flaky task handling change between Nx and Nx Cloud releases, so check the documentation for the version you run.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nx flaky tests, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.