How does Nx detect flaky tests?
In Nx, a task is flaky when the same computation hash has produced both a successful and a failed result. The hash is derived from the task's declared inputs: the relevant source files, the configuration that affects the task, the outputs of the tasks it depends on, and any runtime values or environment variables you have declared as inputs. Two runs with the same hash are, as far as Nx can tell, the same computation.
That is a stronger signal than "the job failed, then passed when someone clicked rerun." A CI job rerun can pick up a new commit, a refreshed lockfile resolution, or a different base image, so a green second attempt proves little. A matching hash rules out changes to code and declared dependencies. If the result still varies, the cause is something Nx was never told about, such as timing, test ordering, shared state, network calls, or the machine the task landed on.
Detection depends on history. A task has to run more than once with the same hash before it can be flagged. Projects that rarely run in CI, or whose inputs change on almost every commit, may never repeat a hash often enough to build that history. Those projects can be unstable without ever appearing on the list.
Flaky detection and automatic task retries are Nx Cloud features. They are not part of the open-source task runner alone. Defaults and configuration options can change, so check the current Nx Cloud documentation before you rely on specific behaviour.
+---------------------------------------------+
| Declared inputs |
| source files, task config, dep outputs, |
| declared runtime values / env vars |
+---------------------------------------------+
|
v
+-------------------+
| computation hash |
| H1 |
+-------------------+
|
+-----------+-----------+
v v
+-------------+ +-------------+ +------------------+
| run A (H1) | | run B (H1) |<---| Not in the hash: |
| PASS | | FAIL | | timing, order, |
+-------------+ +-------------+ | shared state, |
| | | network, machine |
+-----------+-----------+ +------------------+
v
+---------------------------------------+
| Nx Cloud: results recorded per hash |
| H1 -> {PASS, FAIL} => flaky task |
| needs 2+ runs with the same hash |
+---------------------------------------+
Nx Cloud flags a task as flaky when runs with the same computation hash both pass and fail, so the variance must come from factors outside the declared inputs.
Why do accurate task inputs matter for flaky detection?
Flaky detection rests on one assumption: two runs with the same hash ran the same work. Nx Cloud labels a task flaky when the same computation hash yields both a pass and a failure. If the hash leaves out something that changes the result, that label is wrong. Suppose a test reads DATABASE_URL, a fixture outside the project root, or a feature flag from the environment, and none of these is declared. A real behaviour change then shares a hash with the old passing run. It looks like flakiness, and a retry may pass and hide it. The reverse also happens: a cache hit replays a green result for code that would now fail.
Auditing test inputs
Start with the test target's inputs in project.json and the namedInputs in nx.json. Go through the gaps one by one:
- Environment variables. Declare every variable that changes test behaviour as an
{ "env": "NAME" }input. - Runtime state. Add
{ "runtime": "..." }entries for things like the Node version or a database image tag. - Shared files. Add files outside the project, such as fixtures, migrations, or root-level test config. Reference them with
{workspaceRoot}paths. - Irrelevant files. Exclude them with negated patterns, for example docs or
.mdfiles under{projectRoot}.
Then check what Nx actually resolves. nx show project <name> and the task details in the project graph both list the inputs each target hashes.
The trade-off
Under-declared inputs give you false flaky labels and incorrect cache hits. Over-declared inputs cause the opposite problem. Including volatile values, such as a timestamp variable or the whole workspace, makes hashes change on almost every run. Cache hit rates drop. Identical hashes also recur less often, so flaky detection has fewer same-hash pairs to compare and takes longer to spot real instability.
// nx.json: test-target inputs that hash what changes the result and nothing else
{
"namedInputs": {
"default": ["{projectRoot}/**/*", "sharedGlobals"],
"sharedGlobals": [],
"production": ["default", "!{projectRoot}/**/*.spec.ts"],
// Values outside the file tree that change test outcomes
"testEnv": [
// Feature flag read by the code under test
{ "env": "FEATURE_NEW_CHECKOUT" },
// DB image tag, not the connection string, so no secret goes into the hash
{ "env": "TEST_DB_IMAGE_TAG" },
// A Node upgrade should produce a new hash, not a false flaky label
{ "runtime": "node --version" }
]
},
"targetDefaults": {
"test": {
"inputs": [
"default",
"^production",
"testEnv",
// Shared files outside the project that tests depend on
"{workspaceRoot}/jest.preset.js",
"{workspaceRoot}/test/fixtures/**/*",
// Docs never change a test result, so keep them out of the hash
"!{projectRoot}/**/*.md"
]
}
}
}
// Check what Nx resolved for the target: nx show project <name>
An nx.json fragment that declares env vars, Node runtime, and shared fixtures as test inputs and excludes Markdown files, so identical hashes reflect identical test conditions.
Task-level retries vs CI job reruns: what are the alternatives?
Each option trades compute, wall-clock time, and how much it hides from you.
| Option | What reruns | Cost | What it hides |
|---|---|---|---|
| Rerun CI job or pipeline step | Everything in the job, including setup | High: install, checkout, queue time again | Any failure that passes on the second run, flaky or not |
| Test-framework retries | Individual test cases | Low | Every intermittent test, with no record outside the test log |
| Retry Nx Cloud–flagged tasks | Only tasks with a recorded same-hash pass/fail | Low: one task's duration | Only known flaky tasks |
| Quarantine | Nothing, because the task is excluded from the gate | None | Real regressions in that task until it is fixed |
| Do nothing | Nothing | Developer time spent on manual reruns | Nothing, but people start ignoring red |
Retrying only flagged tasks is cheaper because the retry costs one task's runtime, not a full job. It also hides fewer problems. A task that fails for the first time is not retried, so a new regression still turns the pipeline red.
Illustrative estimate. Assume 10 flagged tasks, each failing independently in 3% of runs.
- No retries: 1 − 0.97¹⁰ ≈ 26% of pipelines are red.
- One task-level retry: a task fails only if both attempts fail (0.03² = 0.0009), so 1 − 0.9991¹⁰ ≈ 0.9% are red.
- One full-job rerun: the pipeline stays red only if both runs are red, so 0.26² ≈ 7%, at roughly double the job cost.
None of this helps when the flakiness comes from shared infrastructure, such as a saturated runner or an overloaded shared database. Retrying adds load to the resource that is already struggling. The fix there is isolation or more capacity.
What causes same-hash flakiness in backend test suites?
An Nx task hash covers the inputs you declare: source files, configuration, dependency versions, and listed environment variables. It records nothing about the database the test connects to, the system clock, the network, or what the previous task left on disk. Same-hash flakiness comes from those undeclared factors.
| Root cause | Typical symptom | Typical fix |
|---|---|---|
| Shared database or ports between parallel tasks | Unique-constraint violations, "address already in use", rows from another suite | Give each task its own schema or database; bind to port 0 and read back the assigned port |
| Test ordering dependencies | Passes in isolation, fails in the full run or after reordering | Reset state in setup rather than teardown; randomise order locally to expose hidden coupling |
| Time and timezone assumptions | Fails near midnight, month end, or DST changes, or only on UTC runners | Inject a clock; set the timezone explicitly in the test environment |
| Unmocked network calls | Timeouts and rate-limit errors when a third-party service is slow | Stub at the HTTP client boundary; block outbound traffic in CI so leaks fail loudly |
| Resource contention on CI runners | Timeouts that cluster when many tasks run at once | Wait on conditions instead of fixed sleeps; reduce parallelism on small runners |
| Leftover state in containers or temp directories | Fails only on reused runners or on a second run | Use a unique temp directory per task; start from clean containers |
Several of these causes are real defects, not test noise. A race between two writers to the same row and a connection pool that runs out of connections under parallel load are failures production will also hit, usually at peak traffic. An ordering bug often points to module-level caches or singletons that leak state between requests. When a retry passes, it shows that the code is nondeterministic. It does not show that the code is correct.
// test-isolation.ts: helpers that remove undeclared shared state between parallel tasks
import { createServer, RequestListener, Server } from "node:http";
import { AddressInfo } from "node:net";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
// Parallel tasks run in separate processes, so pid + worker id gives each one
// its own schema instead of sharing tables in one database.
const workerId = process.env.JEST_WORKER_ID ?? "1";
export const schemaName = `test_${process.pid}_${workerId}`;
// Port 0 asks the OS for a free port. No hard-coded 3000 means no "address already in use".
export async function startTestServer(handler: RequestListener): Promise<{ server: Server; baseUrl: string }> {
const server = createServer(handler);
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const { port } = server.address() as AddressInfo;
return { server, baseUrl: ` };
}
// A fresh directory per task, so files left on a reused runner are never read back.
export function makeTempDir(): { dir: string; cleanup: () => void } {
const dir = mkdtempSync(join(tmpdir(), `api-test-${process.pid}-`));
return { dir, cleanup: () => rmSync(dir, { recursive: true, force: true }) };
}
// Inject the clock instead of calling new Date() inside business logic.
export type Clock = () => Date;
export function isExpired(expiresAt: Date, now: Clock = () => new Date()): boolean {
return expiresAt.getTime() <= now().getTime();
}
// In a test, pin time to a known edge case, such as the last second of a month in UTC:
// const fixedNow: Clock = () => new Date("2024-01-31T23:59:59Z");
// expect(isExpired(new Date("2024-02-01T00:00:00Z"), fixedNow)).toBe(false);
Shared test helpers that fix several same-hash flakiness causes: a per-process database schema name, OS-assigned ports via port 0, a unique temp directory per task, and an injected clock for time-dependent logic.
How should you triage and fix the flaky task list?
Treat the flagged list as a queue with a fixed review slot, weekly or once per sprint, not as a dashboard someone checks when CI feels slow. In each review, give every flagged task a named owner, usually the team that owns the project. Without an owner, the task stays in the queue because each engineer who sees it assumes someone else has it.
The owner starts by reproducing the failure. Run the task in isolation many times with nx run api:test --skip-nx-cache in a shell loop. Skipping the cache matters because a cache hit replays an earlier result instead of running the tests, so a loop of cached runs proves nothing.
If the task passes reliably in isolation, run it under load with nx run-many -t test --parallel=8 --skip-nx-cache. Parallel runs bring back the conditions that isolation removes, such as port collisions, shared database schemas, temp-directory clashes and CPU starvation that stretches timeouts. Whether the task fails alone or only alongside other tasks tells you which category of cause to look at first.
Once you understand the cause, the task can leave the queue in one of two ways:
- Fix it. Then confirm the fix with the same repeated runs that reproduced the failure.
- Quarantine it explicitly. Disable or isolate the test, open a tracked issue that links the evidence, and set a deadline for re-enabling or deleting it.
A quarantine list without owners and deadlines tends to only grow. Each quarantined test is behaviour that CI no longer checks, so coverage erodes while the build stays green.
#!/usr/bin/env bash
# Reproduce a flaky Nx task: first in isolation, then under parallel load.
# Usage: ./repro-flaky.sh api:test 50
set -u
TASK="${1:-api:test}" # project:target flagged in the review
RUNS="${2:-50}" # how many repetitions per phase
LOG_DIR="flaky-logs"
mkdir -p "$LOG_DIR"
# Phase 1: isolation. --skip-nx-cache forces a real run every time;
# a cache hit would replay an old result and prove nothing.
fail_iso=0
for i in $(seq 1 "$RUNS"); do
if ! npx nx run "$TASK" --skip-nx-cache > "$LOG_DIR/iso-$i.log" 2>&1; then
fail_iso=$((fail_iso + 1))
fi
done
echo "isolation: $fail_iso/$RUNS failed"
# Phase 2: parallel load. Running every test target at once brings back
# port collisions, shared DB schemas, temp-dir clashes and CPU contention.
fail_par=0
for i in $(seq 1 "$RUNS"); do
if ! npx nx run-many -t test --parallel=8 --skip-nx-cache > "$LOG_DIR/par-$i.log" 2>&1; then
fail_par=$((fail_par + 1))
fi
done
echo "parallel: $fail_par/$RUNS failed"
# Fails alone -> look inside the test (timing, ordering, randomness).
# Fails only in parallel -> look at shared resources between tasks.
# Attach both counts and the logs in $LOG_DIR to the tracked issue.
A shell script that reruns a flagged Nx task with caching skipped, first in isolation and then under parallel load, and records failure counts and logs to attach as evidence to the tracking issue.
When should you avoid automatic retries?
A retry is only safe when running the task twice gives the same result as running it once. Test, lint, and type-check targets that read the workspace and write only to their own output directories usually meet that bar. Deployment, database migration, package publish, and any target that sends notifications or changes shared infrastructure do not. If a deploy fails halfway and is then retried, it can push a partial release a second time. A migration can apply against a schema that is already half-changed. A publish can fail on a version that already reached the registry. None of these should ever be eligible for automatic retry, so keep retry configuration scoped to an explicit allowlist of read-only targets rather than applying it across the workspace.
Retries also have a cost on targets where they are safe. Each attempt adds wall-clock time to the pipeline and compute to the bill. A task that fails every time but has been mislabeled as flaky is the worst case: it uses up its full retry budget and then fails the build anyway, so developers wait longer for the same red result. If a task's retries almost never succeed, treat it as broken, not flaky.
Flaky detection and task-level retries rely on Nx Cloud, which is an external service. Decide in advance how CI behaves when it is unreachable or disabled. Pipelines should still run tasks locally and report real failures, and the team should know that retries will not happen in that mode. Before enabling it, review the pricing model and what task metadata and logs leave your network.
How do you keep flaky CI visible and measure the payoff?
A retry that turns red into green hides information, so make that information land somewhere people look. The minimum is a line in the CI summary or PR check output listing every task that passed only after a retry, with its project, target, and attempt count. Reviewers then see "green, but two tasks needed a second attempt" instead of an unqualified pass. Nx Cloud records retried and flaky tasks on the run itself. Linking that run from the PR keeps the detail one click away rather than buried in logs.
Next, treat the flaky count as a repo health metric alongside build duration and cache hit rate. Track two numbers per week: how many distinct tasks were flagged flaky, and how many runs needed at least one retry. A rising count is an early signal of a real intermittent bug, such as a race in connection pooling or test ordering that leaks state. A falling count shows the triage work from earlier is paying off. Assign the list an owner. Without one, it only grows.
Platform teams running large Nx monorepos with parallel backend tests gain the most. More concurrent tasks mean more chances for one to fail spuriously and sink the whole pipeline. The payoff follows from that, as reasoning rather than measurement:
- Fewer whole-pipeline reruns: Retrying one task is cheaper than repeating the entire pipeline.
- Less engineer time: Developers spend less time watching and re-triggering jobs.
- Earlier detection: Intermittent defects surface as a visible trend before they reach production.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nx flaky tests, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.