How Nx detects flaky tasks from task hashes

Nx computes a hash for every task from its declared inputs: the project's source files, the outputs of dependency tasks, relevant configuration, the resolved external dependency versions, and any environment variables or runtime values you list as inputs. If two runs produce the same hash, Nx treats them as the same piece of work. That is what makes caching safe, and it also makes flakiness detectable.

The reasoning is simple. If the inputs are identical, a deterministic task should always produce the same result. When one hash has a recorded success in one run and a failure in another, the result depends on something the hash does not capture. Common causes are timing, ordering, shared state, or network access. Nx Cloud keeps the history of results per hash and flags tasks where both outcomes occur.

 inputs (src, deps, config, env)
              │
              ▼
      hash: 7f3a…c91e
              │
   ┌──────────┼──────────┐
 run 1      run 2      run 3
  PASS       FAIL       PASS
              │
              ▼
   same hash, mixed results → flaky

Detection only works after Nx Cloud has seen the same hash run more than once. A new task, or one whose inputs change on every commit, has no repeat history yet. During that cold-start period, a failure looks like an ordinary failure.

Flaky detection and automatic retries are Nx Cloud features. Open-source Nx alone does not provide them. Their thresholds and behavior can change between releases, so check the current Nx Cloud documentation before you rely on specifics.

┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐
│ source     │ │ dep task   │ │ project /  │ │ external   │ │ env vars / │
│ files      │ │ outputs    │ │ nx config  │ │ dep vers.  │ │ runtime    │
└─────┬──────┘ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘
      └──────────────┴──────────────┼──────────────┴──────────────┘
                                    ▼
                         ┌─────────────────────┐
                         │ Nx task hasher      │
                         └──────────┬──────────┘
                                    ▼
                         ┌─────────────────────┐
                         │ hash: 7f3a…c91e     │
                         └──────────┬──────────┘
                ┌───────────────────┼───────────────────┐
                ▼                   ▼                   ▼
          ┌───────────┐       ┌───────────┐       ┌───────────┐
          │ CI run 1  │       │ CI run 2  │       │ CI run 3  │
          │ PASS      │       │ FAIL      │       │ PASS      │
          └─────┬─────┘       └─────┬─────┘       └─────┬─────┘
                └───────────────────┼───────────────────┘
                                    ▼
                  ┌───────────────────────────────────┐
                  │ Nx Cloud: result history per hash │
                  │ 7f3a…c91e: PASS, FAIL, PASS       │
                  │ cold start: 1 run = no verdict    │
                  └─────────────────┬─────────────────┘
                                    ▼
                  ┌───────────────────────────────────┐
                  │ same hash, mixed results → FLAKY  │
                  └───────────────────────────────────┘

Identical declared inputs produce one task hash, and when Nx Cloud records both a pass and a fail for that hash across runs, it flags the task as flaky (a hash seen only once has no verdict yet).

Getting inputs and namedInputs right first

Flaky detection assumes that two runs with the same hash ran the same work. If the hash misses something that affects the result, a real behavior change shows up as "same inputs, different outcome," and the task gets flagged as flaky. Common causes are an undeclared environment variable such as DATABASE_URL, a generated client or schema file outside the project's file patterns, and a test that calls a live external service. Retrying these tasks hides a real signal.

The opposite mistake has a different cost. If inputs covers the whole workspace, or includes files that change on every commit such as build stamps, logs or coverage output, the hash changes on almost every run. Repeat executions with matching hashes then rarely happen, so there is little evidence to detect flakiness from, and caching stops working too.

The example that follows defines a production named input that excludes spec files and test configuration. It also adds { "env": "..." } entries for the variables the tests read, and uses !-prefixed patterns to keep generated artifacts and reports out of the hash. Put shared definitions in nx.json and override them per project only where a project actually differs.

Auditing a test target's inputs

  • Search the test code and its setup files for every environment variable they read, and declare each one.
  • Find files produced by code generation and decide whether to hash them or their source.
  • Check that dependency projects contribute their production inputs, not their test files.
  • Exclude outputs, caches and reports.
  • Look for network calls to services the hash cannot represent, and mock them or pin their version.
{
  "namedInputs": {
    "default": [
      "{projectRoot}/**/*",
      "!{projectRoot}/src/generated/**",
      "!{projectRoot}/coverage/**",
      "!{projectRoot}/reports/**",
      "!{projectRoot}/**/*.log",
      "sharedGlobals"
    ],
    "sharedGlobals": [
      "{workspaceRoot}/tsconfig.base.json",
      { "runtime": "node --version" }
    ],
    "production": [
      "default",
      "!{projectRoot}/**/*.spec.ts",
      "!{projectRoot}/jest.config.ts",
      "!{projectRoot}/src/test-setup.ts"
    ]
  },
  "targetDefaults": {
    "test": {
      "cache": true,
      "inputs": [
        "default",
        "^production",
        "{workspaceRoot}/jest.preset.js",
        "{workspaceRoot}/schema/api.graphql",
        { "env": "DATABASE_URL" },
        { "env": "API_BASE_URL" },
        { "env": "TZ" }
      ],
      "outputs": ["{projectRoot}/coverage"]
    }
  }
}

An nx.json that keeps generated code, coverage, reports and logs out of the hash, hashes the schema that generated code comes from, has dependencies contribute only their production files, and declares the environment variables and Node version the tests depend on.

Who should use automatic retries, and when to avoid them

A retry does not fix a flaky task. It contains it, so one unstable test doesn't block every pull request that touches its project. Treat it as a temporary measure for a problem someone is already working on.

Who benefits

Retries pay off most in large monorepos with shared CI, where a single flaky test in a widely depended-on library can fail dozens of unrelated pipelines a day. Platform teams responsible for merge throughput gain the most. A retry turns a blocked merge queue into a few extra seconds of compute, and the flake still shows up in run history for whoever owns it.

Retry a task only when all three of these are true:

  • It is already flagged as flaky.
  • It is idempotent, meaning a second run starts from the same state as the first.
  • It has an open ticket with an owner.

Without the ticket, a retry just hides the problem.

When to avoid them

Skip retries for non-idempotent tasks. If a test leaves behind database rows, a bound port, a populated temp directory or a running container, the second attempt runs against different conditions than the first. It may fail for reasons unrelated to the original flake. Worse, it may pass only because the first attempt did setup work.

Be equally cautious with tests that exercise concurrency, locking, retry logic or other race-prone code. In those tests, intermittent failure is often the signal itself. A retry that goes green hides a real defect that will show up in production under load, where nothing retries for you.

Choosing a retry layer and task granularity

Each layer you can retry at sees a different unit of failure, and that determines what it costs and what it hides.

Retry layer Cost of one retry Visibility of the flake Blast radius
Test runner (per test case) Re-runs a single test Often only in runner output, easy to miss Smallest; can mask a real defect inside one test
Nx task Re-runs the whole target, including setup Recorded against the task hash, so it can be flagged One project's target
CI job Re-runs every task in the job, minus cache hits Coarse; you see that the job failed, not which test caused it Entire pipeline stage

Pick one primary layer. Stacking all three multiplies the worst case. With two runner retries, two task retries and one job retry, a persistently failing test can execute 3 × 3 × 2 = 18 times before anyone sees red. That is an illustrative estimate based on assumed retry counts, not a measurement.

Granularity matters as much as the layer. A single e2e target that runs 200 spec files gives a task-level retry two bad options. It can re-run everything, or it can report "something in here is flaky." This is why Nx can split e2e suites into one task per file. Each file gets its own hash, so a flake is attributed to a specific spec, and a retry re-runs only that spec.

Splitting has costs. Each task has orchestration overhead. Each task also repeats setup and teardown, such as booting a server, seeding a database or launching a browser. The task graph and cache also grow, with many more entries to store and evict.

Teams without Nx Cloud can enable runner-level retries and record their own results. For each run, store the task hash, the test identifier and whether the test passed on the first or a later attempt. A test that both passes and fails under the same hash is flaky by the same definition Nx uses.

Reproducing and fixing a flagged task

Treat each flagged task as a ticket with an owner, the task hash, and the failing test identifier attached. The hash matters because it tells you the inputs were identical across the passing and failing runs, so the cause is in the test or its environment, not in a code change.

Start by reproducing locally. Run the task in a shell loop with --skip-nx-cache so every iteration actually executes instead of replaying a cached result, and count failures across a few dozen runs. If it never fails serially, rerun the loop with a higher --parallel value alongside sibling projects. Many backend flakes only appear under contention, and CI agents usually run more work concurrently than a laptop does.

Once you can trigger the failure, check these causes in rough order of frequency:

  • Shared state between tests. Module-level singletons, in-memory caches, or database rows that one test writes and another reads. Reset state in setup rather than relying on teardown, because teardown is skipped when a test crashes.
  • Test order dependence. Run the file with randomized order if your runner supports it. A test that passes only after another one has run is depending on a side effect.
  • Real timers and sleeps. A fixed sleep assumes a speed that a loaded CI agent may not have. Use fake timers, or poll for the condition with a timeout.
  • Unmocked network calls. External services add latency and failures you don't control.
  • Port collisions. Hardcoded ports clash when several projects run in parallel. Bind to port 0 and read back the port the OS assigned.
  • Time zone and clock assumptions. Tests that format dates or cross midnight break on agents set to UTC. Pin the time zone and inject a clock.
#!/usr/bin/env bash
# Reproduce a flaky Nx task by running it repeatedly with the cache bypassed.
# Usage: ./repro-flake.sh <project> [runs] [parallel] [sibling-projects]
# Example: ./repro-flake.sh api 30 6 api,worker,billing
set -u

PROJECT="${1:?project name required}"
RUNS="${2:-30}"
PARALLEL="${3:-4}"
SIBLINGS="${4:-$PROJECT}"
LOG_DIR="flake-logs/$PROJECT"
mkdir -p "$LOG_DIR"

run_loop() {
  local label="$1"; shift
  local failures=0
  for i in $(seq 1 "$RUNS"); do
    # --skip-nx-cache forces real execution; a cache hit would replay the old result.
    if ! "$@" > "$LOG_DIR/$label-$i.log" 2>&1; then
      failures=$((failures + 1))
      echo "[$label] run $i FAILED (log: $LOG_DIR/$label-$i.log)"
    fi
  done
  echo "[$label] $failures/$RUNS runs failed"
}

# Phase 1: the task on its own, one at a time.
run_loop serial npx nx run "$PROJECT:test" --skip-nx-cache

# Phase 2: add contention by running sibling projects concurrently,
# closer to how a busy CI agent schedules work.
run_loop parallel npx nx run-many -t test -p "$SIBLINGS" \
  --parallel="$PARALLEL" --skip-nx-cache

# Phase 3: rule out time zone assumptions by forcing a non-local zone.
TZ=UTC run_loop tz-utc npx nx run "$PROJECT:test" --skip-nx-cache

A shell loop that reruns an Nx test target with --skip-nx-cache, first on its own, then under higher --parallel contention with sibling projects, then with TZ forced to UTC. It counts failures and keeps a log for each run.

Backend-specific flakiness and mitigations

Integration targets fail in their own ways. These failures look random but have a single cause: tests assume an external process is ready, sized correctly, or exclusively theirs.

Cause Why it flakes Fix
Containerized dependency not ready A container that has started is not the same as a service that accepts queries. Fast agents win the race and slow agents lose it. Gate tests on a readiness check, such as a health check or a polling loop, rather than on the container starting.
Migrations racing tests Tests query tables that a concurrent migration has not created yet. Make migration a separate target and list it in the test target's dependsOn.
Connection pool exhaustion Nx runs several projects at once. Each test worker opens its own pool, and together they exceed the server's connection limit. Set an explicit parallelism limit for integration targets that is lower than the limit for unit tests.
Shared database Two tasks truncate or seed the same tables. The outcome depends on which task runs first. Give each task its own isolated schema or database, named after the project.
Leaked async handles Open sockets, timers, or pools keep the process alive or fail the next suite. Enforce teardown in the test lifecycle and run with open-handle detection, for example Jest's --detectOpenHandles.

For readiness, the startup script should poll until the service answers before it exits successfully. With PostgreSQL, loop on pg_isready with a deadline. With Docker Compose, declare a healthcheck and use depends_on with condition: service_healthy. Fail loudly when the deadline passes, so a slow container shows up as an infrastructure error and not as a flaky test.

For parallelism, run integration targets as their own invocation with a lower limit, such as nx affected -t integration --parallel=2. Size that number from your database's connection limit divided by the pool size per worker, rather than from the number of CPU cores on the agent.

#!/usr/bin/env bash
# CI step: start the database, wait until it accepts connections,
# then run integration targets with a lower parallelism limit than unit tests.
set -euo pipefail

DB_HOST=localhost
DB_PORT=5432
DB_USER=app_test          # placeholder test role, not a real credential
READY_TIMEOUT_SECONDS=60

# A running container does not mean the server is ready.
docker compose up -d postgres

# Poll until PostgreSQL accepts connections, with a hard deadline.
deadline=$((SECONDS + READY_TIMEOUT_SECONDS))
until pg_isready -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -q; do
  if (( SECONDS >= deadline )); then
    # Fail as an infrastructure error, not as a flaky test.
    echo "ERROR: postgres not ready after ${READY_TIMEOUT_SECONDS}s" >&2
    docker compose logs postgres >&2
    exit 1
  fi
  sleep 1
done
echo "postgres is ready"

# Unit tests have no shared external state, so they can use more parallel tasks.
npx nx run-many -t test --parallel=6

# Each integration task opens its own connection pool. A low limit keeps
# the combined total below the server's max_connections.
# Migrations run first if 'integration' lists 'migrate' in its dependsOn.
npx nx run-many -t integration --parallel=2

A CI script that waits on pg_isready with a deadline before any tests run, then runs integration targets with a lower --parallel limit than unit tests so the combined connection pools stay within the database's connection limit.

Quarantine policy and impact on cost and engineering time

Quarantine policy

A quarantined test keeps running on every pipeline, but its result no longer blocks a merge. Keeping it running matters because the pass/fail history is what tells you whether a fix worked. Deleting or skipping the test removes that evidence. Every quarantine entry needs three fields: an owner (a person or team, not "platform"), the date it entered quarantine, and an exit criterion such as "zero failures across 200 consecutive runs" or "root cause fixed and merged." Entries that reach a fixed age, for example 30 days, should come up for review, and that review ends in either a fix or a deletion.

Quarantine saves time because engineers stop rerunning pipelines and investigating failures that have nothing to do with their change. Unowned quarantine has the opposite effect. The list grows, real regressions get hidden among known-flaky tests, and developers learn to ignore red builds.

Track these metrics weekly:

  • The number of tasks flagged as flaky.
  • The retry rate per target.
  • The CI time spent on retries.

Illustrative estimate (assumed numbers, not a benchmark)

Daily retry cost = runs per day × flaky failure rate × duration of the retried task

Assume 200 runs per day and a 5% flaky failure rate.

  • Coarse target (20 minutes): 200 × 0.05 × 20 = 200 CI minutes per day
  • Split target (flaky test isolated in a 2-minute target): 200 × 0.05 × 2 = 20 CI minutes per day

The failure rate is identical in both cases. Splitting the target cuts the retry cost by 90% because a failure only reruns the small target that contains the flaky test.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx flaky tests, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.