Nx Flaky Tests: Retry, Detect, and Quarantine Unstable Backend Tests

How Nx detects flaky tests from task hashes and the cache

In Nx terms, a flaky task is one that has produced both a success and a failure for the same task hash. Nx computes that hash from the task's declared inputs: source files matched by its input patterns, dependency outputs, relevant config, runtime values you list, and the command itself. If two runs share a hash, Nx considers them the same work on the same inputs. A pass and a fail under one hash therefore cannot be explained by a code change. Something outside the declared inputs varied, such as timing, ordering, shared state, or the network. This is the signal Nx Cloud uses to flag a flaky task.

Copyhash a91f… (inputs unchanged)
  run 1 ──> FAIL ──> not cached ──> rerun
  run 2 ──> PASS ──> written to cache
  run 3 ──> cache hit ──> PASS replayed, test never executes

Caching hides this. Failures are not cached, but a lucky pass is, and later runs replay it without executing the test. A flaky suite can look stable until its inputs change and it runs for real again. For that reason, read the cache hit rate and the flaky rate together. A high hit rate with a low flaky rate may only mean the flaky tests are rarely executed.

The method is only as good as your input definitions in project.json and nx.json:

• Inputs too broad: if they include something that varies between machines, runs get different hashes and the signal is lost.
• Inputs too narrow: genuinely different code shares a hash, so a real regression can look like a flake, or a stale pass gets replayed.
• Undeclared environment drift: a base image update or an external service change is not part of the hash. It shows up as a plain failure on a new run, not as a flake, unless the same hash already has a pass recorded.

Copy Declared inputs (project.json / nx.json)
 ┌────────────────────────────────────────────────┐
 │ source files · dep outputs · config ·          │
 │ runtime values · command                       │
 └─────────────────┬──────────────────────────────┘
                   │ hashed
            ┌──────▼──────┐   Not in hash: timing, ordering,
            │ hash a91f…  │   shared state, network, base image
            └──────┬──────┘
        ┌──────────┼──────────────────────────┐
        ▼          ▼                          ▼
   ┌─────────┐┌─────────┐              ┌─────────────┐
   │ run 1   ││ run 2   │              │ run 3       │
   │ FAIL    ││ PASS    │              │ cache hit   │
   └────┬────┘└──┬───┬──┘              └──────▲──────┘
        │        │   │ write                  │ replay PASS
        │        │   ▼                        │ (test not run)
        │        │ ┌───────────┐              │
        │        │ │   cache   ├──────────────┘
        │        │ └───────────┘
        ▼        ▼
   ┌────────────────────────────────────────┐
   │ Nx Cloud: same hash has PASS and FAIL  │
   │ → task flagged as flaky                │
   └────────────────────────────────────────┘
   FAIL: not cached, rerun.  PASS: cached, later runs replay it.

Runs that share one input hash but produce both a pass and a fail are flagged as flaky, while the cached pass is replayed on later runs and hides the instability until the declared inputs change.

Where retries happen and what each layer hides

Retries can happen in three places, and each layer hides something different.

Test-level retries are the cheapest, but they hide the most. The runner absorbs the failure inside one process, so the task-level detection described above never sees it. Task-level retries cost a full target run, but they produce exactly the fail-then-pass signal on an identical hash that detection needs. Job reruns cost the most and leave the least structure behind.

The trade-off is direct. Automatic retries keep the merge queue moving, and they also make flakiness invisible. A race condition in a connection pool that fails one run in twenty will pass on retry nearly every time. Without an alert on tasks that were retried and then passed, that bug ships to production.

If you enable runner retries, call jest.retryTimes(1) in a setup file, or set test.retry: 1 in the Vitest config. Keep the count at one or two. Each extra attempt lowers the chance that a genuine intermittent failure surfaces, and it inflates the runtime of tests that are truly broken.

Copy// jest.config.ts (in the project root, e.g. apps/api/jest.config.ts)
import type { Config } from 'jest';

const config: Config = {
  testEnvironment: 'node',
  // Runs after the test framework is installed, so the `jest` global
  // and jest.retryTimes are available.
  setupFilesAfterEnv: ['<rootDir>/jest.setup.ts'],
};

export default config;

// ---------------------------------------------------------------
// jest.setup.ts
// Retry only in CI. Locally a flaky test should fail loudly.
if (process.env.CI) {
  // One retry. Each extra attempt makes a real intermittent failure
  // less likely to surface and slows down tests that are truly broken.
  jest.retryTimes(1, {
    // Print the first failure before retrying so the evidence is
    // kept in the CI log instead of being replaced by the passing run.
    logErrorsBeforeRetry: true,
  });
}

// ---------------------------------------------------------------
// Vitest equivalent (vitest.config.ts):
// import { defineConfig } from 'vitest/config';
// export default defineConfig({
//   test: { retry: process.env.CI ? 1 : 0 },
// });
//
// With either runner the Nx task still exits 0 after a retried pass,
// so Nx's task-level flaky detection never sees these retries.

A Jest setup that allows a single CI-only retry and logs the first failure, with the equivalent Vitest retry setting. Because the Nx task exits 0, Nx cannot see these retries.

Who should use retries plus quarantine, and when to avoid it

Retries plus quarantine pays off when a flaky test has real costs. The typical case is a backend or platform team in an Nx monorepo with a large integration suite, shared CI agents, and a merge queue. In that setup, one intermittent failure in a database or message-broker test stops every pull request behind it. The cost scales with the number of engineers waiting, not with the size of the bug. A single retry absorbs the noise, and quarantine moves the offender out of the blocking path so the queue keeps moving while someone investigates.

The approach is the wrong tool in several situations:

• New suites. If a suite is weeks old and already flaky, the problem is usually shared state, fixed ports, or ordering assumptions. Fix isolation now, before the pattern spreads.
• Critical-path tests. Tests that guard payments, auth, or data migrations should not be parked. A quarantined test protects nothing, so rewrite it until it is deterministic.
• Environment drift. When failures come from mismatched container images, runtime versions, or agent configuration, retries hide an infrastructure fault that will eventually break everything.
• Thin run history. An entry rule such as "failed and passed on the same commit N times" needs enough runs to evaluate. Without that history, quarantine decisions are guesses.

Nx Cloud's flaky task detection is a useful safety net, but relying on it ties your process to that service. If you don't use it, collect run history yourself, for example by archiving JUnit-style test reports from CI. Check the current Nx docs for configuration details.

Implementing a quarantine target in an Nx workspace

Start with a naming convention, because it is the cheapest signal both humans and tools can read. Rename a known-flaky file from orders.spec.ts to orders.quarantine.spec.ts. A tag inside the test name also works, but a file suffix keeps quarantine visible in code review and in git log.

Split the targets

In the project's project.json, keep the existing test target and add a pattern that ignores *.quarantine.spec.ts. With Jest, testPathIgnorePatterns does this. Then add a test-quarantine target that uses the same executor and config but matches only quarantine files, for example through testMatch or a testPathPattern option.

Make the inputs correct

Give test-quarantine the same inputs as test, typically the default and ^production named inputs. If the inputs are too narrow, Nx can return a cached result after a dependency changed. A cached replay is not a new observation, so it misleads the pass/fail history you are trying to build. For this reason, many teams also run the quarantine job with --skip-nx-cache so every CI run executes the tests.

Wire up CI

The blocking job runs nx affected -t test, which never selects quarantine files. A separate job runs nx run-many -t test-quarantine (or nx affected -t test-quarantine to save compute) and is marked as allowed to fail. Archive its JUnit reports on every run. Running it on every merge to the main branch, and not only on pull requests, gives a steadier stream of results for judging whether a test has stabilized.

Copy# apps/orders-api/project.json: both targets run Jest with the same config.
# Only the file filter differs: quarantine files end in .quarantine.spec.ts.
cat <<'EOF' > apps/orders-api/project.json
{
  "name": "orders-api",
  "targets": {
    "test": {
      "executor": "nx:run-commands",
      "cache": true,
      "inputs": ["default", "^production"],
      "options": {
        "command": "npx jest --ci --config apps/orders-api/jest.config.ts --testPathIgnorePatterns '\\.quarantine\\.spec\\.ts$'"
      }
    },
    "test-quarantine": {
      "executor": "nx:run-commands",
      "cache": true,
      "inputs": ["default", "^production"],
      "options": {
        "command": "npx jest --ci --config apps/orders-api/jest.config.ts '\\.quarantine\\.spec\\.ts$'"
      }
    }
  }
}
EOF

# Blocking CI job: a failure here fails the pipeline.
# Quarantined files are never selected by the test target.
npx nx affected -t test

# Non-blocking CI job: skip the cache so every run is a fresh observation.
# '|| true' keeps the job green. Archive the test reports from every run.
npx nx run-many -t test-quarantine --skip-nx-cache || true

*The example shows a project.json with a blocking test target that excludes .quarantine.spec.ts files and a test-quarantine target that runs only those files, both using the same inputs, followed by the nx commands for the blocking CI job and the non-blocking, uncached quarantine job.

Quarantine policy: entry criteria, ownership, deadlines, and exit

A quarantined test still runs, but it no longer protects anything. A regression in the code it covers will show up as a failure in a job everyone has agreed to ignore. Every day a test spends in quarantine is a day that code path ships unguarded. That is why the policy needs hard deadlines that are enforced automatically, not left to good intentions.

The deletion rule sounds harsh, but it is honest. A test that has failed its time box is no longer doing its job. Deleting it shows that gap in coverage plainly, where keeping it would hide the gap.

Finding and fixing root causes of backend flakiness

Most backend flakiness comes from a handful of shared-state problems. Nx's --parallel option and distributed execution make them visible: two projects that passed for months when run one after another start colliding once they share a CI agent, a database, or a port range at the same time. The coupling was already there. Parallelism just changed the timing enough to expose it.

For per-worker databases, derive the name from the worker ID, for example app_test_${JEST_WORKER_ID}. Each worker then owns its data and cannot race with another worker.

To reproduce a failure locally, run the single target in a loop with --skip-nx-cache. Without that flag, a cache hit replays the stored result and the tests never run again. Match the CI parallelism, CPU limits, and memory limits as closely as you can, because many races only show up under contention. Log the test order and seed on every run so you can replay a failing sequence exactly.

Strong isolation has a cost. Per-worker databases and stricter teardown add wall-clock time and consume more agent resources.

Copy#!/usr/bin/env bash
# Rerun one Nx test target until it fails, recording the seed of each run.
# Assumes a Jest-based target named api:test (Jest 29.2+ for --randomize/--seed).
set -u

RUNS=50
LOG_DIR=./flaky-runs
mkdir -p "$LOG_DIR"

# Fix the timezone so clock-dependent failures are not caused by the laptop's TZ.
export TZ=UTC

for i in $(seq 1 "$RUNS"); do
  SEED=$RANDOM
  echo "run $i seed=$SEED" | tee -a "$LOG_DIR/seeds.log"

  # --skip-nx-cache forces a real run; a cache hit would replay the stored result.
  # --maxWorkers=2 mirrors the CI agent's worker count so contention matches.
  # Inside the tests, each worker derives its own database name, for example
  # app_test_${JEST_WORKER_ID}, so parallel workers never share data.
  if ! npx nx run api:test --skip-nx-cache -- \
      --randomize --seed="$SEED" --maxWorkers=2 \
      > "$LOG_DIR/run-$i.log" 2>&1; then
    echo "FAILED on run $i with seed $SEED; see $LOG_DIR/run-$i.log"
    echo "Replay: npx nx run api:test --skip-nx-cache -- --randomize --seed=$SEED --maxWorkers=2"
    exit 1
  fi
done

echo "No failure in $RUNS runs"

A shell loop that reruns a single Nx test target with --skip-nx-cache, a fixed timezone, CI-matched worker count and a logged random seed, stopping on the first failure and printing the command to replay it.

Alternatives, metrics, and impact on cost and engineering time

That isolation cost is one input. Compare it with the cost of the alternatives:

Track four metrics:

• Flaky rate per project: the share of runs that fail and then pass on the same hash.
• Extra CI minutes caused by retries.
• Quarantine size and the age of its oldest entry.
• Fix rate: the share of quarantined tests that are fixed rather than deleted. Deletion hides lost coverage.

Illustrative estimate (assumed numbers, not a benchmark). Formula:

extra minutes/day = runs/day × flaky rate × retries × rerun duration

Assumed inputs:

• Target duration: 10 minutes
• Runs per day: 200
• Flaky rate: 5% of runs
• Retry count: 1
• Single-test rerun: 0.5 minutes

Task-level retry: 200 × 0.05 × 1 × 10 = 100 minutes/day.

Test-level retry: 200 × 0.05 × 1 × 0.5 = 5 minutes/day.

The larger engineering-time gain comes from a different source than CI minutes. With the policy in place, a flaky test no longer blocks a merge or triggers a manual rerun. Engineers also stop debugging failures that are not real. Those interruptions cost more than compute, because each one breaks someone's focus.

The upfront cost is real. Per-worker databases, deterministic clocks, and rewrites of order-dependent suites take days of work. Start with the projects that have the highest flaky rate, so the isolation work pays back first where failures are most frequent.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx flaky tests, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.