What is a flaky task in Nx, and why is the hash a stronger signal?
In Nx, every task (for example api:test) gets a computation hash before it runs. The hash is built from the things that can change the task's result: the project's source files, the files of the projects it depends on, external dependency versions from the lockfile, the target's configuration and command-line arguments, and any extra inputs you declare, such as environment variables or runtime values. Nx uses this hash as the key for its cache. If two runs share a hash, Nx treats them as the same computation.
A flaky task is one where a single hash has produced both a pass and a fail. Every input Nx tracks was identical between those runs, so the different outcome cannot come from a code or config change. It comes from something outside the hash: test ordering, timing, a shared database or port, network calls, the system clock, or leftover state from an earlier run. That is nondeterminism, and the hash gives you evidence of it.
Pipeline-level detection works differently. It counts "failed, then passed on rerun" at the job level, and that pattern is ambiguous. Someone may have pushed a fix between attempts. The base branch may have moved. A dependency may have resolved to a different version. A job-level counter records all of these as flakes, which hides real regressions and inflates the flaky list with false positives.
Hash-based detection pays off most for backend and platform teams running large monorepos with many CI runs per day. At that volume, identical hashes recur often enough to separate real flakes from changed inputs with confidence.
Who should use hash-based flaky detection and quarantine?
This approach fits teams where flaky failures are a shared problem rather than one person's annoyance:
- Teams running a large Nx monorepo with frequent CI runs. Many projects and many pipelines per day mean the same task hashes recur often. Flake evidence builds up quickly, and a single unstable task can block many unrelated pull requests.
- Platform and developer-experience teams who own CI. They need a list of unstable tasks ranked by evidence, not a collection of anecdotes from chat threads, and they need a policy for retries and quarantine that applies across every project.
- Backend teams with integration and e2e suites. These suites depend on databases, ports, networks, and clocks, which are the most common sources of nondeterminism. They are also the slowest to rerun, so each flake costs more.
- Teams using a merge queue or a protected main branch. When a red check blocks a merge for everyone behind it, telling a flake apart from a regression quickly matters most.
It is less useful in some situations. A small repository with a few CI runs per day collects repeated hashes slowly, so detection takes a long time to say anything. A workspace whose task inputs are poorly declared will produce misleading signals until the inputs are fixed (see the diagnostic checklist below). Teams that don't use Nx Cloud can still apply the retry-layer, quarantine, and cost-ranking practices in this article, because those rely on tags, target configurations, and CI job settings rather than on the hosted service.
How does it work end to end?
The full workflow combines Nx's hashing, the remote cache, a retry policy, and quarantine. In order:
- Hash. Before a task runs, Nx computes its hash from the declared inputs.
- Cache lookup. If a successful result for that hash exists in the remote cache, Nx replays it and the task does not execute.
- Execute and record. On a cache miss, the task runs. The outcome is recorded against the hash. Successes are cached and failures are not.
- Detect. When one hash has both a recorded failure and a recorded success, the task is flagged as flaky. Nothing tracked changed between those runs, so the instability is real.
- Retry at one layer. A failed task can be retried at the orchestration layer so the pipeline can proceed. The retry is recorded, so it stays visible instead of disappearing into a green build.
- Quarantine. Tasks or specs that keep flaking move out of the blocking job using a
quarantinetag or aquarantinetarget configuration. They still run on every pipeline in a non-blocking job, and the manifest check makes sure each one has an owner and an issue. - Rank and fix. Flaky tasks are ordered by what they cost, and owners fix, delete, or rewrite them. The quarantine marker is removed once a fix has shown it is stable across repeated runs.
The sections that follow cover each step in detail: detection and its trade-offs, where retries belong, how to quarantine, how to get tests back out, what usually causes the flakiness, and how to prioritise fixes.
How does Nx Cloud flaky detection work with remote caching?
Nx computes a hash for every task from its inputs: source files, dependency outputs, configuration, environment variables you declare, and the runtime. With remote caching enabled, a task that succeeds has its terminal output and artifacts stored under that hash. Any later run on any agent or branch that produces the same hash replays the stored result instead of executing. Failed results are not cached, so a failing task always runs again on the next attempt.
That asymmetry is what makes detection possible. The service records the outcome of each execution against its hash. If one hash has both a failure and a success, the inputs were identical and only the run differed. Nx Cloud can then flag the task as flaky with reasonable confidence, which is a stronger basis than guessing from a test name that failed twice this week.
The design has trade-offs:
- Evidence requires repetition. A task needs several executions of the same hash before a pattern shows up. Tasks that run rarely, or whose hash changes on almost every commit, collect evidence slowly.
- A retried success hides the flake. If a retry passes, that success is cached. Later runs get a cache hit and never execute the task, so no new outcomes are recorded until an input changes the hash.
- CI policy depends on a hosted service. Find out what your pipeline does when the service is unreachable or disabled. Check whether tasks fall back to local execution, whether retries still apply, and whether builds fail or slow down.
Feature availability, setting names, and UI locations depend on your plan and workspace configuration. Check them in your own account rather than assuming they match another team's setup.
+-------------------------+ hash(inputs) +------------------------+
| Task: test my-lib |----------------------->| Nx Cloud |
| inputs: src, deps, | | |
| config, env, runtime |<-----------------------| Remote cache |
+-------------------------+ hit: replay output | (successes only) |
| | |
| miss: execute | Outcome history |
v | per input hash |
+-------------------------+ record outcome | |
| Run task |----------------------->| a1f3: FAIL, PASS |
| pass -> store result | | => flag as flaky |
| fail -> not cached | | b72c: PASS, PASS |
+-------------------------+ | => no signal |
| +------------------------+
| fail, then retry passes
v
Retry pass is cached; later runs hit the cache and record nothing new
until an input changes the hash. Rare tasks gather evidence slowly.
Service unreachable or disabled? Verify: local fallback, retries, fail?
Only successful task results go into the remote cache, and the outcome history recorded for each input hash is what lets Nx Cloud flag a task as flaky; this also explains why a cached retry success hides the flake until an input changes.
What do flaky task retries actually rerun, and where should they live?
An Nx retry reruns the entire task, meaning one project's target such as api:test or web-e2e:e2e, not the single test case that failed. If an e2e target takes 25 minutes and one spec flakes, the retry costs another 25 minutes. The result also tells you only that the target was unstable, not which spec caused it.
Splitting a large e2e suite into smaller targets fixes both problems. If each target runs one area of the suite, such as auth, billing, or admin, a retry reruns only that slice, and the flaky signal points to a specific target. The trade-off is overhead. Each target repeats its setup, such as starting the app, seeding a database, or launching a browser. Each target is also one more thing to configure, cache, and distribute across agents. Split along stable functional boundaries, and keep each slice large enough that setup is a small share of its runtime.
| Retry layer | What reruns | Visibility of flakiness | Main risk |
|---|---|---|---|
| Test runner | Single failing test | Low unless the runner reports retries | Hides flakes inside a passing task |
| Nx / CI orchestration | Whole target | Recorded per task | Slow on large targets |
| CI job | Entire pipeline | Coarse, per job | Most expensive |
Choose one primary layer. Retries at different layers multiply. Illustrative estimate: 2 runner retries × 1 orchestration retry × 1 job rerun allows up to 3 × 2 × 2 = 12 executions of the same test. That inflates runtime, and the real failure rate becomes impossible to read. For most workspaces, the orchestration layer is the better primary choice, because it records instability per task where you can track it.
{
"name": "web-e2e",
"$schema": "../../node_modules/nx/schemas/project-schema.json",
"projectType": "application",
"targets": {
"e2e-auth": {
"executor": "nx:run-commands",
"options": {
"cwd": "apps/web-e2e",
"command": "npx playwright test src/auth --retries=0"
}
},
"e2e-billing": {
"executor": "nx:run-commands",
"options": {
"cwd": "apps/web-e2e",
"command": "npx playwright test src/billing --retries=0"
}
},
"e2e-admin": {
"executor": "nx:run-commands",
"options": {
"cwd": "apps/web-e2e",
"command": "npx playwright test src/admin --retries=0"
}
},
"e2e": {
"executor": "nx:noop",
"dependsOn": ["e2e-auth", "e2e-billing", "e2e-admin"]
}
}
}
A web-e2e project.json that splits one large Playwright suite into auth, billing, and admin targets, with an aggregate e2e target that depends on all three. A retry reruns only the failing slice, and runner retries are disabled with --retries=0 so orchestration remains the single retry layer.
When should you avoid relying on retries?
Every retry trades signal for throughput. When a task fails and then passes on the second attempt, the pipeline turns green, but the failure still happened. Sometimes the test is the problem: a hardcoded timeout or a shared fixture. Sometimes the test is accurate and the system under test is what's unreliable. A race condition in a connection pool, or a missing lock around a read-modify-write, will fail intermittently in CI for the same reason it will fail intermittently in production. Retry it into green and you have shipped the bug with a passing build attached.
Turn retries off, or cap them at one attempt and alert on every retry, in these cases:
- Concurrency-sensitive code. Queue consumers, schedulers, caches with eviction, and anything that coordinates goroutines, threads, or workers. Intermittent failure is the expected symptom of a real defect here, so a retry hides the evidence you need.
- Newly added tests. A new test has no history, so you can't tell whether it is flaky or catching a real problem. Let it fail loudly for its first stretch of runs so the author has to look at it.
- Security and data-integrity paths. Authorization checks, migrations, idempotency guarantees, and ledger or balance logic. A failure that shows up one time in twenty can still corrupt data or let an unauthorized request through.
Retries buy time to diagnose a flake while the rest of the team keeps merging. They are not the fix. Every retried pass should leave a record that someone is expected to act on.
How do you implement CI test quarantine in an Nx workspace?
Quarantine works at two granularities, and the right one depends on how much of a project is unstable.
Project-level. If a whole project is unreliable, add a tag such as quarantine to its tags array in project.json. Exclude it from the blocking job with nx affected -t test --exclude=tag:quarantine, and run it separately with nx run-many -t test --projects=tag:quarantine. Tag patterns in --projects are long established. Tag patterns in --exclude arrived later, so check that your Nx version accepts them before you depend on them. Tags work well here because they live in the project graph, which means affected filtering keeps working without a hand-maintained list of project names.
Spec-level. When only a few specs are flaky, quarantining the whole project would discard too much coverage. Add a second configuration to the existing test target instead. The default configuration ignores the quarantined paths through your test runner's ignore option. A quarantine configuration runs only those paths, invoked as nx test api --configuration=quarantine.
Run the quarantined set in a CI job marked as allowed to fail. It should still execute on every pipeline, because you need its pass/fail history to decide when a test can return to the blocking job.
Every quarantined item needs a named owner and a linked issue. JSON files can't hold comments, so keep a small manifest file next to the config. Add a CI check that fails when a quarantined path or tag has no matching manifest entry. Without that check, quarantine becomes a place where tests are forgotten.
Quarantine has a real cost. The code those tests covered can now regress without blocking a merge. Treat each entry as a temporary coverage gap, not a resolution.
#!/usr/bin/env bash
set -euo pipefail
case "${1:-blocking}" in
blocking)
# Blocking job: affected tests, minus projects tagged "quarantine" in project.json.
# VERIFY: older Nx versions may not accept tag patterns in --exclude.
npx nx affected -t test --exclude=tag:quarantine
;;
quarantine)
# Non-blocking job: mark it allowed-to-fail in CI, but run it on every pipeline
# so its pass/fail history shows when a test can return to the blocking job.
npx nx run-many -t test --projects=tag:quarantine
# Spec-level quarantine: the "quarantine" configuration runs only the flaky specs.
npx nx test api --configuration=quarantine
;;
check)
# Fail if any quarantined project lacks an owner and issue in the manifest.
# quarantine-manifest.json example:
# [{"project": "api", "owner": "jane.doe", "issue": "TICKET-123"}]
find . -name project.json -not -path './node_modules/*' -print0 |
xargs -0 jq -r 'select(any((.tags // [])[]; . == "quarantine")) | .name' |
while read -r name; do
jq -e --arg n "$name" \
'any(.[]; .project == $n and (.owner // "") != "" and (.issue // "") != "")' \
quarantine-manifest.json > /dev/null ||
{ echo "Quarantined project '$name' has no owner/issue entry" >&2; exit 1; }
done
;;
*)
echo "unknown mode: $1" >&2; exit 2
;;
esac
A CI script that keeps tag-quarantined projects out of the blocking test run, still runs them (and spec-level quarantined tests) in a non-blocking job, and fails the build when a quarantined project has no owner and linked issue in the manifest.
How do you get tests out of quarantine, and what are the alternatives?
Each quarantined test should leave by one of three exits:
- Fix and return. Find the cause, fix it, and remove the quarantine marker in the same pull request. Before merging, run the test repeatedly in CI to show it is stable. A single green run proves little for a test that already passes most of the time.
- Delete. If the behaviour is covered elsewhere, or the test asserts something nobody depends on anymore, remove it. A deleted test is honest. A test that stays quarantined forever is a skip that nobody reviews.
- Rewrite. If the scenario matters but the test design is the problem, such as real network calls, shared database state, or wall-clock assertions, replace it with a deterministic version at a lower level.
Limit quarantine by time and by size. Give each entry an owner and a deadline, for example two weeks. Cap the list at a fixed number per project. When the cap is reached, no new test can enter until one leaves, which forces the team to fix tests instead of hiding them. The owning team should review the list every week, and expired entries should go to someone with authority to choose between deleting and fixing.
| Approach | Visibility | Coverage | Cost |
|---|---|---|---|
| Quarantine | High if it is a tracked list | Temporarily lost for listed tests | Review overhead |
| Plain retries | Low; flakes look green | Kept, but weakened | Extra CI time per retry |
| Skip or disable | Very low | Lost, often permanently | Near zero, until a regression ships |
| Fix immediately | High | Kept | Engineer time up front |
| Pipeline rerun counting | Medium; tracks how often pipelines are rerun | Kept | Full reruns cost the most compute |
What causes nx flaky tests in backend suites? A diagnostic checklist
Work through these in order. The first five are real nondeterminism. The last is usually a configuration bug that only looks like flakiness.
- Shared databases or ports across parallel tasks. Nx runs independent tasks concurrently, so two projects' integration suites can hit the same schema or bind the same port. Give each task its own database name or schema and let the OS assign ports, rather than hardcoding
5432or3000. - Test order dependence. A test that passes only after another one has seeded data will fail when the runner shuffles or shards. Run the suite in random order locally to expose these tests.
- Real clocks and timezones. Assertions near midnight, month boundaries, or DST changes fail on agents set to UTC. Inject a clock or use fake timers, and pin
TZexplicitly. - Unmocked network calls. Third-party APIs and DNS add latency and outages you don't control. Stub them at the HTTP client boundary.
- Resource limits on CI agents. Tight timeouts that pass on a laptop fail on a throttled container. If failures cluster under high parallelism, lower parallelism before raising timeouts.
- Inputs missing from the hash. If a test reads
DATABASE_URLor a gitignored generated client, Nx can't see changes to it. The same hash then produces different results, which gets flagged as flaky, or a stale result gets replayed from cache. Declare environment variables with{ "env": "DATABASE_URL" }entries in a target'sinputs, or in a sharednamedInputsset innx.json. For generated files, produce them in a target that the test target lists underdependsOn, and declare that generator's own sources as its inputs.
{
"namedInputs": {
"default": ["{projectRoot}/**/*", "sharedGlobals"],
"sharedGlobals": [{ "env": "TZ" }],
"testEnv": [{ "env": "DATABASE_URL" }]
},
"targetDefaults": {
"generate-client": {
"inputs": ["{projectRoot}/schema/**/*"],
"outputs": ["{projectRoot}/src/generated"],
"cache": true
},
"test": {
"dependsOn": ["generate-client"],
"inputs": ["default", "testEnv", "^default"],
"cache": true
}
}
}
An nx.json that adds TZ and DATABASE_URL to the test hash through named inputs, and makes test depend on a cached generate-client target whose inputs are its schema sources.
How can it save engineering time?
Flaky tasks cost engineers time in a few predictable ways. A developer reruns a pipeline by hand and waits. A teammate investigates a red check that has nothing to do with their change. A merge queue stalls behind a failure. Someone triages a "flaky" report that was really a regression. The practices in this article address each of these:
- Less time spent triaging. A task flagged because the same hash both passed and failed is already known to be nondeterministic. Nobody has to rule out "did my change cause this?" before moving on, and real regressions aren't mislabelled as flakes, because a changed input produces a different hash.
- Cheaper reruns. The remote cache replays successful tasks, so a retry or a rerun pipeline executes only what failed or changed. Splitting large e2e targets makes each retry rerun one slice instead of the whole suite.
- Fewer blocked merges. Quarantining a known-flaky task removes it from the blocking job, so unrelated pull requests stop failing because of it. The quarantine job keeps running, so the signal isn't lost.
- No retry multiplication. Keeping a single retry layer stops one unstable test from quietly running many times per pipeline and stretching every build.
- Effort goes where it pays. Ranking flaky tasks by cost, as described in the next section, points engineers at the fixes that recover the most waiting time, instead of the noisiest entries in a failure list.
- Less ongoing maintenance. Tag-based quarantine and the manifest check run automatically in CI. Nobody has to maintain a hand-written list of excluded projects or chase down who owns a skipped test.
How much time this recovers depends on your run volume, flake rates, and how long developers wait on each retry. The next section shows how to estimate it for your own workspace with explicit assumed inputs.
How much do flaky tasks cost, and which should you fix first?
Rank flaky tasks by the time and money they waste, not by how often they appear in a failure list. A task that flakes often but retries in seconds can cost less than one that rarely flakes but forces a twelve-minute rerun while someone waits on the PR.
The formula per task:
daily cost = runs/day × flake rate × retry minutes × (agent cost/min + developer cost/min × share of retries a developer waits on)
The developer term usually dominates. Agent minutes are cheap compared with an engineer blocked on a red check.
Illustrative estimate (assumed numbers, not a benchmark). Assume agents cost $0.05/min, developer time costs $1.50/min, and a developer waits on every retry. Each retry minute then costs $1.55.
| Task | Runs/day | Flake rate | Retry min | Flakes/day | Daily cost |
|---|---|---|---|---|---|
api:integration |
200 | 3% | 6 | 6 | 6 × 6 × 1.55 = $55.80 |
billing:e2e |
40 | 5% | 12 | 2 | 2 × 12 × 1.55 = $37.20 |
utils:unit |
200 | 10% | 0.5 | 20 | 20 × 0.5 × 1.55 = $15.50 |
utils:unit has the highest flake rate but comes last. api:integration costs about $55.80 × 250 working days ≈ $13,950 a year. If a fix takes two engineer-days (about 16 hours × $90 = $1,440), it pays back in roughly 26 working days.
Run the same calculation with your own run counts and rates. Fix tasks from the top of the list down, and stop when the yearly saving drops below the cost of the fix.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nx flaky tests, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.