Nx Flaky Tests: Retry, Quarantine and Track Backend Test Failures

How Nx identifies a flaky task

In Nx, every task (for example, api:test) gets a computation hash. The hash is derived from the task's declared inputs: source files, the project's dependency graph, the command and its flags, configured environment variables, and runtime values. Nx uses that hash to decide whether a cached result can be reused. Nx Cloud also uses it for flaky task detection. A task is considered flaky when the same hash has produced both a passing and a failing result across runs.

This is a stronger signal than "the test failed once on main." A failure on main could come from a real regression, a bad merge, or a dependency bump. Matching hashes rule those out. Nothing Nx was told to watch changed, yet the outcome did. The cause is therefore something outside the declared inputs, such as timing, shared state, test ordering, network calls, or the clock.

For backend and platform teams running Jest across dozens of services in one monorepo, this distinction matters. It separates "someone broke the payments service" from "the payments integration suite is nondeterministic." Those two problems need different owners and different urgency.

The signal is only as good as the inputs behind the hash. If a task reads a file or environment variable that isn't declared, real changes can hide behind an unchanged hash. Genuine regressions then get labelled as flakiness. Section 2 covers how to audit this.

This guide takes one position throughout. Retries are a triage tool that keeps CI moving. Fixing the test is separate work, with its own owner and deadline.

Copy Declared inputs
+------------------------------+
| source files                 |
| project dependency graph     |    +-----------------------+
| command + flags              |--->| computation hash      |
| configured env vars          |    | api:test -> hash H    |
| runtime values               |    +-----------------------+
+------------------------------+                |
                                     +----------+------------+
+--------------------------+         v                       v
| Outside the hash:        |  +-------------+         +-------------+
| timing, shared state,    |  | CI run 1    |         | CI run 2    |
| order, network, clock,   |  | hash: H     |         | hash: H     |
| undeclared files/env     |  | PASS        |         | FAIL        |
+--------------------------+  +-------------+         +-------------+
                                     |                       |
                                     +----------+------------+
                                                v
                              +-----------------------------------+
                              | Nx Cloud: same hash, both results |
                              | => task marked FLAKY              |
                              +-----------------------------------+
                                                |
                                 +--------------+----------------+
                                 v                               v
                        +-----------------+             +-----------------+
                        | Triage: retry   |             | Fix: separate   |
                        | keeps CI moving |             | job with owner  |
                        | (not the fix)   |             | and deadline    |
                        +-----------------+             +-----------------+

Nx Cloud marks a task flaky when the same computation hash, derived from declared inputs, produces both a pass and a fail, so the cause lies outside the hash; retries handle triage, and fixing the test is separate owned work.

Auditing inputs so detection is accurate

Nx Cloud decides a task is flaky when the same computation hash produces both a pass and a fail. That rule is only as good as the hash. If something that affects the test's outcome sits outside the task's inputs, two runs can share a hash while actually testing different things. A real regression then looks like nondeterminism.

The reverse failure is quieter. A test that would fail intermittently keeps getting a replayed cached success because its hash never changes. Nx never sees the failure, so it never classifies it.

What to check for backend test targets

Start with targetDefaults in nx.json, then each project.json that overrides inputs for test or e2e. For each target, ask what can change the result and confirm it is hashed:

• Environment variables. Values such as DATABASE_URL, feature flags or TZ need env inputs. Otherwise a CI runner with a different value reuses the same hash.
• Fixtures and seed data stored outside {projectRoot}, for example a shared test-data directory.
• Migrations and schema files. A new migration changes behaviour without touching any test file.
• Generated code. Generated clients, ORM types and protobuf output should come in through dependentTasksOutputFiles or an explicit file set.

The trade-off

Broad inputs, such as hashing the whole workspace or every env var, are safe but costly. They cut cache hits, and they produce pass/fail pairs across hashes that are genuinely different, which buries the real flakes in noise. Narrow inputs keep the cache fast but give you wrong cache hits and misclassified regressions. Aim for the smallest set that captures every outcome-relevant dependency. When a test is flagged as flaky, check its inputs before you add a retry.

Copy{
  "namedInputs": {
    "default": ["{projectRoot}/**/*", "sharedGlobals"],
    "production": ["default", "!{projectRoot}/**/*.spec.ts"],
    "sharedGlobals": ["{workspaceRoot}/tsconfig.base.json"],
    "backendTestEnv": [
      { "env": "DATABASE_URL" },
      { "env": "FEATURE_FLAGS" },
      { "env": "TZ" }
    ],
    "backendTestData": [
      "{workspaceRoot}/test-data/**/*",
      "{workspaceRoot}/db/migrations/**/*",
      "{workspaceRoot}/db/schema.prisma"
    ]
  },
  "targetDefaults": {
    "test": {
      "dependsOn": ["^build"],
      "cache": true,
      "inputs": [
        "default",
        "^production",
        "backendTestEnv",
        "backendTestData",
        { "dependentTasksOutputFiles": "**/*.d.ts", "transitive": true }
      ]
    }
  }
}

An nx.json fragment that hashes env vars, shared fixtures, migrations, schema and generated type outputs for backend test targets, so a changed dependency yields a new hash instead of a false flake or a stale cache hit.

Retry layers: Nx Cloud task retries vs Jest retryTimes

Once the inputs are trustworthy, decide where retries should live. There are two layers, and they behave very differently.

Nx Cloud flaky task retry works at the task level. Nx Cloud marks a task as flaky when the same computation hash has both passed and failed. When such a task fails again, Nx Cloud re-runs the whole task and records the occurrence, so you build a history instead of a silent pass. Check the current Nx Cloud documentation for the exact settings and retry limits.

jest.retryTimes works at the test level and needs the jest-circus runner. You call it inside a test file, and it re-runs only the failing tests in that file. It works anywhere Jest runs, including laptops and self-managed CI, with no hosted service. The trade-off is that nothing tracks the retries for you unless you add that yourself.

Avoid retries on release-critical paths, where a pass on the third try is not evidence of correctness. Also avoid them for tests that mutate shared state without idempotent setup. In that case, a retry runs against state the failed attempt left behind.

Don't stack the two layers. Retries multiply, and the final report shows only the attempt that passed, which hides the failure pattern you need for debugging.

Illustrative estimate (assumed numbers, not a benchmark): worst-case executions = (1 + task retries) × (1 + test retries). With 2 task retries and 3 test retries, that is 3 × 4 = 12 runs of one broken test.

Copy// orders.integration.spec.ts
// jest.retryTimes needs the jest-circus runner (the default runner since Jest 27).
// It applies to every test in THIS file, so keep release-critical tests in a
// separate file that never calls it.
import { createTestDb, TestDb } from './test-utils/db'; // placeholder helper
import { OrderService } from '../src/order.service';

// Retry failing tests in this file up to 2 more times (3 attempts total).
// logErrorsBeforeRetry prints each failed attempt's error, so the report
// does not show only the passing attempt.
jest.retryTimes(2, { logErrorsBeforeRetry: true });

let db: TestDb;
let service: OrderService;

beforeAll(async () => {
  db = await createTestDb('postgres://test:test@localhost:5432/orders_test');
});

// Idempotent setup. beforeEach runs before every attempt, including retries,
// so a retry starts from a clean state and not from rows the failed
// attempt left behind.
beforeEach(async () => {
  await db.truncate(['orders', 'order_items']);
  service = new OrderService(db);
});

afterAll(async () => {
  await db.close();
});

test('creates an order and reserves stock', async () => {
  const order = await service.create({ sku: 'SKU-123', qty: 2 });
  expect(order.status).toBe('reserved');
});

A Jest test file that uses jest.retryTimes with logged failed attempts and an idempotent beforeEach reset, so each retry runs against clean state and keeps release-critical tests out of the retrying file.

Common root causes of flaky backend tests and their fixes

Retries hide these failures. They do not remove them. Most flaky backend tests fall into a small set of causes:

For isolation, JEST_WORKER_ID is a practical key for naming databases or schemas per worker. Combine it with the project name so parallel Nx tasks never share one.

Starvation needs the most judgment. Lowering parallelism costs almost nothing today, but every pipeline gets slower, and contention comes back as the test suite grows. Per-task isolation needs work up front: provisioning scripts, per-worker databases and dynamic ports. In return, you can raise parallelism safely as you add agents. Reduce parallelism as a stopgap, and track the isolation work as the real fix.

Copy// test/isolation.ts: list in jest.config `setupFilesAfterEnv` so each test file gets its own schema.
import { Client } from 'pg';
import type { Server } from 'node:http';
import type { AddressInfo } from 'node:net';

// Nx sets NX_TASK_TARGET_PROJECT for each task. Jest sets JEST_WORKER_ID per worker.
// Combining both means parallel Nx tasks and parallel Jest workers never share a schema.
const project = (process.env.NX_TASK_TARGET_PROJECT ?? 'local').replace(/[^a-z0-9]/gi, '_').toLowerCase();
const worker = process.env.JEST_WORKER_ID ?? '1';
export const schema = `test_${project}_w${worker}`;

export const db = new Client({ connectionString: process.env.TEST_DATABASE_URL }); // e.g. postgres://test:test@localhost:5432/app_test

beforeAll(async () => {
  await db.connect();
  await db.query(`CREATE SCHEMA IF NOT EXISTS "${schema}"`);
  await db.query(`SET search_path TO "${schema}"`); // unqualified table names now resolve to this worker's schema
  // Run migrations against the schema here.
});

// Clear rows before every test so no test depends on data left by an earlier one.
beforeEach(async () => {
  const { rows } = await db.query('SELECT tablename FROM pg_tables WHERE schemaname = $1', [schema]);
  if (rows.length > 0) {
    const tables = rows.map((r) => `"${schema}"."${r.tablename}"`).join(', ');
    await db.query(`TRUNCATE ${tables} RESTART IDENTITY CASCADE`);
  }
});

// Close the connection so Jest does not report an open handle or hang after the run.
afterAll(async () => {
  await db.end();
});

// Port 0 lets the OS choose a free port, so parallel tasks never collide on a fixed one.
export async function listenOnFreePort(server: Server): Promise<number> {
  await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
  return (server.address() as AddressInfo).port;
}

This Jest setup file gives each Nx project and Jest worker its own Postgres schema, truncates tables before every test, closes the connection after the file runs, and binds test servers to OS-assigned ports.

Reproducing flakiness locally

CI history tells you a test is flaky. A local loop tells you whether your fix worked. Before trusting a change, reproduce the failure on demand.

Loop a single target with the cache skipped

Run one target, for example nx run orders-api:test, repeatedly with --skip-nx-cache. Without that flag, Nx replays the cached result after the first success, and you learn nothing. Twenty to fifty iterations is a reasonable starting point. A test that fails once in fifty runs locally is likely to fail regularly across many CI agents. Record how many runs failed, both before and after your fix. A drop from several failures to zero across the same number of runs is evidence you can put in the pull request.

Compare serial and parallel execution

Run the same loop twice: once with Jest's --runInBand and once with several workers through --maxWorkers. Each result points to a different cause:

• Fails only in parallel: workers are competing for something shared, such as a database schema, a fixed port, a temp directory or a cache key.
• Fails only in band: state is leaking between test files that share a single process, such as module-level singletons, unclosed connections or mocks that were never restored.

Shake out order dependencies

Hidden ordering assumptions surface when the order changes. Recent Jest versions support --randomize and print a seed. Rerun with --seed to reproduce a failing order exactly.

To check whether a test depends on earlier tests, run it on its own with --testNamePattern or by passing its file path. If it fails alone, it relies on setup that another test performs. If it passes alone but fails in the full suite, another test is polluting shared state.

A test quarantine CI workflow with ownership and exit rules

Once you know a test is flaky but can't fix it today, move it out of the blocking path without letting it go silent. In Nx, the cleanest way is a second target, such as test-quarantine, alongside test. Keep a quarantine list of file paths in the project. The main Jest config excludes those files through testPathIgnorePatterns, and the quarantine config includes only them through testMatch. CI runs both targets. Only test is a required check, while test-quarantine reports its results without failing the merge. Because the quarantined tests still execute on every run, you keep collecting evidence about whether they fail.

Each entry in the quarantine list needs three fields: an owner (a person, not a team alias), a tracking issue, and the date it was added. Have a CI step reject any entry that is missing one of them. Without an owner, nobody fixes the test. Without a date, nobody notices that it has been quarantined for months.

Define the exit rule up front. For example, a test returns to test after 20 consecutive green runs in quarantine following a fix. Add a ceiling as well: after 30 days, the owner must either fix the test or delete it.

Quarantine reduces the pressure to fix, and that is the main risk. Compare it with the alternatives:

• Fixing immediately is best when the cause is known.
• Skipping with .skip hides the test entirely, so you stop learning anything from it.
• Deleting is honest if the test's coverage is duplicated elsewhere or low-value.

Quarantine fits between these options. It is a time-boxed holding area for tests that matter but can't be fixed right away.

Tracking cost, prioritizing fixes and keeping retries temporary

Rank flaky tests by what they cost, not by how annoying they feel. Two data sources cover this. Nx Cloud's flaky-task history shows which task flips between pass and fail for the same inputs. CI logs or Jest JUnit-style reports show which test case inside that task failed and how long the retry took.

Join the two on task name and commit. Then sort by flake frequency × retry duration.

Illustrative estimate (assumed numbers, not a benchmark). Assume 400 CI runs per week, a 5% retry rate on one task, 6 minutes per retry, and a developer blocked waiting on half of the affected runs.

• Lost CI minutes = runs × retry rate × retry duration = 400 × 0.05 × 6 = 120 min/week
• Blocked engineer time = runs × retry rate × blocked fraction × retry duration = 400 × 0.05 × 0.5 × 6 = 60 min/week

Put your own values into the formulas. Even small rates add up across a monorepo with many tasks.

Guardrails

• Alert on rising flaky rates per task, not on absolute counts. A busy project will always have more raw flakes than a quiet one.
• Review quarantine size on a fixed schedule. Sort entries by age. Any test past its exit date gets fixed or deleted.
• Report CI duration with and without retries. Retries inflate p95 durations and can hide real regressions in build time.
• Treat pass-on-retry as a failure on release-critical paths, such as payments, auth and migrations. On those paths, a retry that turns red into green is a bug you have not found yet.

How the workflow works end to end

The pieces above form one loop. Each step feeds the next:

1. Hash. Nx computes a hash for each task from its declared inputs. The input audit makes sure everything that can change a test's outcome is part of that hash.
2. Detect. Nx Cloud compares results for the same hash across runs. A pass and a fail for one hash marks the task as flaky, which means the cause lies outside the declared inputs.
3. Contain. A single retry layer, either Nx Cloud task retries or jest.retryTimes in specific files, keeps CI moving while the failure is recorded rather than hidden.
4. Reproduce. Locally, you loop the target with --skip-nx-cache, compare --runInBand with multiple workers, and randomize order to narrow the cause to shared resources, leaked state or ordering.
5. Isolate or quarantine. If the fix is quick, you fix it. If not, the test moves to test-quarantine with an owner, a tracking issue, a date and an exit rule. It keeps running but no longer blocks merges.
6. Measure and exit. Flake history joined with per-test reports ranks fixes by cost. Tests leave quarantine once they meet the exit rule, and retries are removed once the cause is fixed.

The important property is that nothing leaves the loop silently. A retried pass is still recorded, a quarantined test still runs, and every quarantine entry has someone accountable for it.

Who should use this approach

This workflow is aimed at teams that run backend tests in an Nx monorepo and already feel the cost of nondeterministic failures:

• Platform and developer-experience engineers who own nx.json, CI pipelines and shared test infrastructure. They set up input auditing, retry policy, the quarantine target and the CI check that validates quarantine entries.
• Backend service teams whose Jest suites talk to databases, queues, HTTP services or the clock. They own the root-cause fixes, the per-worker isolation and the quarantined tests assigned to them.
• Tech leads and engineering managers who need to decide which flaky tests to fix first. The cost formulas and the quarantine age review give them something concrete to prioritize against.

Teams that don't use Nx Cloud can still apply most of it. Input auditing, jest.retryTimes, local reproduction, per-worker isolation and the quarantine target all work with Nx and Jest alone. What you lose is the central, hash-based flaky-task history. You would need to build an equivalent from CI logs and JUnit-style reports.

A small repository with a handful of fast, purely unit-level tests probably doesn't need the full setup. When flakes are rare and the cause is obvious, fixing them directly is cheaper than maintaining a quarantine process.

When to use it

Use each part of the workflow when its trigger appears, not all at once:

• Audit inputs when you first enable caching or flaky-task detection for test targets, and again whenever a test is flagged as flaky. Do this before adding any retry, because a misclassified regression should not be retried.
• Add retries when a flaky test is blocking unrelated merges and the cause is not yet known. Do not use them on release-critical paths or on tests whose setup is not idempotent.
• Reproduce locally as soon as a task shows up in flaky history, and again after any proposed fix, so the pull request includes before-and-after failure counts from the same number of runs.
• Quarantine when the cause is known or suspected but the fix will take longer than the team can tolerate the blocking failure. Skip quarantine when the fix is small enough to land right away.
• Isolate resources when reproduction shows failures only under parallel execution, or when you plan to raise Nx --parallel or Jest --maxWorkers on larger agents.
• Track cost continuously once more than a few tasks are flaky, so prioritization is based on data rather than on whichever failure was most recent.

Revisit the setup when the monorepo changes shape. New services, a new database, or new shared fixtures are all common points where undeclared inputs and shared-resource contention appear.

How it saves engineering time

The time savings come from removing repeated manual work and from directing fix effort at the tests that cost the most:

• Less manual re-running. Without a policy, engineers re-run failed pipelines by hand and wait for them. A controlled retry layer does that automatically, and Nx Cloud records each occurrence, so nobody has to remember which failures were "probably flaky."
• Faster triage. Same-hash detection separates nondeterminism from real regressions. Engineers don't spend time bisecting commits for a failure that no input change caused, and real regressions aren't dismissed as flakiness when the inputs are audited.
• Shorter debugging. The local loop with --skip-nx-cache, serial versus parallel runs, and randomized order narrows the cause to one category quickly, instead of relying on guesswork from a single CI log.
• Unblocked merges. Quarantine removes a known-flaky test from the required check, so unrelated pull requests stop waiting on it. The test keeps running, so its owner still gets evidence.
• Better use of fix time. Ranking by flake frequency × retry duration points effort at the tests with the highest combined CI and waiting cost, rather than the most recently noticed one.
• Fewer recurring flakes. Per-worker schemas, dynamic ports and closed handles remove whole classes of failures, which also makes it safe to raise parallelism and shorten pipelines later.

To check the effect in your own repository, apply the formulas from the tracking section before and after the workflow is in place, using your measured run counts, retry rates and retry durations. Any figures you derive before measuring should be treated as illustrative estimates based on assumed inputs.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx flaky tests, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.