Nx E2E Docker Compose Targets for Affected Test Dependencies

What a per-project compose target is and why a global stack breaks down

A per-project compose target treats the Docker Compose stack an E2E suite needs as a normal Nx target on that project. If orders-api-e2e needs Postgres and a Kafka broker, it gets its own compose-up target that starts those services. Its e2e target declares that target in dependsOn. When Nx decides the project is affected, the services start. When it isn't, nothing starts.

The common alternative is one global docker compose up in the CI script, followed by nx affected -t e2e. That setup has three problems:

• Every job pays the startup cost. A pull request that only touches a frontend library still starts every database, broker and mock server in the repository. It also waits for their health checks, even if no affected suite uses them.
• Dependencies are hidden. The link between a suite and its services lives in a shell script outside the project graph. Nx can't see it, reviewers can't trace it, and removing a service doesn't show which suites will break.
• Ports collide under parallelism. When Nx runs several E2E tasks at once against one shared stack, they compete for the same fixed host ports and the same database state. A suite that resets tables can break another suite halfway through its run.

The pattern pays off for backend and platform engineers who run several E2E suites backed by real services, where each suite uses a different subset of those services. It fits best when suites can run independently and the full stack is slow or heavy to start. It adds little when there is a single suite or every suite needs the whole stack.

How it works: dependsOn, run-commands and the affected graph

Three Nx mechanisms combine to make this work. Each handles one concern.

Selection. nx affected -t e2e compares the current commit against a base and walks the project graph. An E2E project is selected if its own files changed or if any project it depends on changed. Unaffected suites never start containers.

Ordering. The e2e target declares dependsOn on a compose-up target in the same project. Nx always runs compose-up first and starts e2e only if compose-up succeeds. Ordering is resolved per project, so one project's compose-up does not wait on another project's tests.

Execution. compose-up uses the nx:run-commands executor with cwd set to the project root. It runs docker compose -f <project file> -p <unique name> up -d --wait. Each part of the command has a specific job:

• -p gives each project its own compose project name. Containers, the default network and named volumes are scoped to that name, so stacks for different projects do not collide.
• --wait blocks until services are running, or healthy if they define a healthcheck. A Postgres container without a healthcheck counts as "running" before it accepts connections. Every service the tests hit needs a real healthcheck, or --wait gives a false signal.

Copynx affected -t e2e
        │
   ┌────┴─────────────────────┐
   ▼                          ▼
orders-e2e                 billing-e2e
compose-up (-p orders-e2e) compose-up (-p billing-e2e)
 [postgres, redis]          [postgres, stripe-mock]
   │ healthy                  │ healthy
   ▼                          ▼
 e2e                         e2e
        same Docker host

dependsOn only defines what runs before a target. Nx has no matching "after" hook. If e2e fails, nothing stops the stack, so teardown has to be built separately.

Copy ┌──────────────────────────────────────────────────────────────────────┐
 │ nx affected -t e2e   (diff vs base, walk the project graph)          │
 └───────────┬───────────────────────────┬──────────────────────┬───────┘
             │ affected                  │ affected             │ unaffected
             ▼                           ▼                      ▼
 ┌───────────────────────┐   ┌───────────────────────┐   ┌──────────────┐
 │ orders-e2e            │   │ billing-e2e           │   │ search-e2e   │
 │ compose-up            │   │ compose-up            │   │ not run, no  │
 │ nx:run-commands       │   │ nx:run-commands       │   │ containers   │
 │ -p orders-e2e         │   │ -p billing-e2e        │   └──────────────┘
 │ up -d --wait          │   │ up -d --wait          │
 └───────────┬───────────┘   └───────────┬───────────┘
             ▼                           ▼
 ┌───────────────────────┐   ┌───────────────────────┐
 │ stack: orders-e2e     │   │ stack: billing-e2e    │
 │ postgres    [healthy] │   │ postgres    [healthy] │
 │ redis       [healthy] │   │ stripe-mock [healthy] │
 │ net + volumes scoped  │   │ net + volumes scoped  │
 └───────────┬───────────┘   └───────────┬───────────┘
             │ --wait returns            │ --wait returns
             ▼ dependsOn met             ▼ dependsOn met
 ┌───────────────────────┐   ┌───────────────────────┐
 │ e2e (tests)           │   │ e2e (tests)           │
 └───────────┬───────────┘   └───────────┬───────────┘
             ▼                           ▼
      no "after" hook:             no "after" hook:
      stack stays up               stack stays up
 ════════════════════════ same Docker host ══════════════════════════════

Each project selected by nx affected starts its own compose stack under its own -p name. --wait holds e2e until the healthchecks pass, and because Nx has no "after" hook, the stack keeps running when the tests finish.

Implementing compose-up, e2e and guaranteed teardown

to live somewhere that runs regardless of the test exit code.

Step 1: the naive targets

Start with two nx:run-commands targets in the app's project.json. compose-up runs docker compose up -d --wait, and compose-down runs docker compose down -v. Set cwd to the project root on both so each project resolves its own docker-compose.yml. Then give e2e a dependsOn entry pointing at compose-up.

This works on a green run. A trailing compose-down does not. Nx stops a task chain at the first non-zero exit, so if e2e fails, any target sequenced after it is skipped. The failing runs are exactly the ones that leave orphaned containers and volumes behind.

Step 2: one wrapper with a trap

Replace the chain with a single target that calls a small script, such as tools/e2e-with-stack.sh. The script does three things:

1. Registers trap cleanup EXIT INT TERM. cleanup runs docker compose down -v whether the tests pass, fail or are cancelled with Ctrl-C or a CI job abort.
2. Brings the stack up with --wait.
3. Runs the test command and preserves its exit code so Nx still reports the failure.

Inside cleanup, check an opt-in variable such as E2E_KEEP_STACK=1. When it is set, skip teardown and print the project name, so you can inspect the containers after a failure.

Connection details

Do not hard-code localhost:5432. Load host and port values through the target's envFile. If you let Compose assign ports, resolve them in the script with docker compose port and export them before the tests start.

Copy#!/usr/bin/env bash
# tools/e2e-with-stack.sh <compose-dir> <test command...>
# Called from one nx:run-commands target, e.g.:
#   "command": "tools/e2e-with-stack.sh apps/api npx playwright test"
set -euo pipefail

cd "$1"; shift
# Unique project name so parallel affected runs don't share containers.
export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-e2e-$(basename "$PWD")-$$}"

cleanup() {
  local rc=$?            # exit code of the tests (or 130/143 on a signal)
  trap - EXIT            # never run teardown twice
  if [[ "${E2E_KEEP_STACK:-}" == "1" ]]; then
    echo "E2E_KEEP_STACK=1: leaving '$COMPOSE_PROJECT_NAME' running" >&2
    echo "Remove later: docker compose -p $COMPOSE_PROJECT_NAME down -v" >&2
  else
    docker compose down -v --remove-orphans || true
  fi
  exit "$rc"             # Nx still sees the real failure
}
# EXIT covers pass and fail; INT/TERM (Ctrl-C, CI abort) exit and so hit EXIT.
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

docker compose up -d --wait

# Resolve the host port Compose assigned instead of assuming localhost:5432.
db_addr="$(docker compose port db 5432)"   # e.g. 0.0.0.0:49153
export DB_HOST=127.0.0.1
export DB_PORT="${db_addr##*:}"
export DATABASE_URL="postgres://e2e:e2e-password@${DB_HOST}:${DB_PORT}/app_test"

# Run the tests; under set -e a failure triggers the EXIT trap with its code.
"$@"

A wrapper script that starts the Compose stack, exports dynamically assigned ports, runs the tests and always runs docker compose down -v via an EXIT trap unless E2E_KEEP_STACK=1 is set.

Isolation, affected inputs and caching rules

Isolation starts with the Compose project name. Compose derives container, network and volume names from it, and by default it uses the directory name. Two projects with similarly named folders, or two CI jobs sharing a Docker host, can then attach to each other's Postgres or tear it down partway through a run. Pass -p (or set COMPOSE_PROJECT_NAME) using the Nx project name, and append a job or pipeline identifier when runners share a daemon. The same rule applies to host ports. Either let Compose assign ephemeral ports, or allocate them through environment variables that the test runner reads. A fixed 5432:5432 mapping fails as soon as two stacks start in parallel.

Inputs that mark the right projects as affected

The affected graph is only as accurate as the inputs you declare. List the compose file, seed SQL, init scripts and any env file that pins image tags in the e2e target's inputs, using {projectRoot}/... paths. That way, bumping postgres:16 to a new tag counts as a real change.

Some files are shared across projects, such as a common seed fixture or a base compose file under {workspaceRoot}/tools/compose. Define these once in a namedInputs entry in nx.json, for example composeShared. Then reference it from each consuming target's inputs. An edit to a shared file then affects exactly the projects that declare it, not the whole workspace and not none of it.

No caching for side effects

Set "cache": false on compose-up, compose-down and e2e, ideally in targetDefaults so new projects inherit it. A cache hit replays stored output without running the target. Nx would report a passing e2e run while no containers ever started. A target whose purpose is a side effect on the host has no output worth restoring, so it should never be cached.

Copy{
  "namedInputs": {
    "default": ["{projectRoot}/**/*"],
    "composeShared": [
      "{workspaceRoot}/tools/compose/base.compose.yml",
      "{workspaceRoot}/tools/compose/seed/common-fixtures.sql"
    ],
    "composeLocal": [
      "{projectRoot}/docker-compose.e2e.yml",
      "{projectRoot}/e2e/seed/**/*.sql",
      "{projectRoot}/e2e/init/**/*",
      "{projectRoot}/.env.e2e"
    ]
  },
  "targetDefaults": {
    "compose-up": {
      "cache": false,
      "inputs": ["composeLocal", "composeShared"]
    },
    "compose-down": {
      "cache": false
    },
    "e2e": {
      "cache": false,
      "dependsOn": ["compose-up"],
      "inputs": ["default", "composeLocal", "composeShared", "^default"]
    }
  }
}

An nx.json that declares compose files, seed SQL and the image-tag env file as named inputs, so edits mark the right projects affected, and disables caching for compose-up, compose-down and e2e in targetDefaults.

Running it in CI and debugging failures

The CI sequence is the same on any provider. Check out the repository with enough git history to reach the base commit. A shallow single-commit clone makes the comparison fail or treat everything as affected. Set the base to the merge base with your main branch (or the last successful main build) and the head to the current commit. Then run nx affected -t e2e --base=<sha> --head=<sha> --parallel=<n>. Pass --parallel explicitly so the runner's memory sets the limit, not the default.

Illustrative estimate. These inputs are assumptions, not measurements. Take a 16 GB runner with 3 GB reserved for the OS, the CI agent and Nx. Each e2e task starts a database (about 0.5 GB), a cache (about 0.1 GB), the service under test (about 1 GB) and a test runner (about 1.5 GB), for roughly 3 GB per task. That allows (16 − 3) / 3 ≈ 4 concurrent tasks. Use --parallel=3 so a memory spike causes slower tests instead of OOM-killed containers.

When a run fails, set the keep-up opt-in variable your teardown checks so the stack survives the run. Then read docker compose -p <name> logs using the project name derived for that task. A CI job killed by a timeout or cancellation never runs its teardown. On self-hosted or persistent runners, docker compose ls lists those orphaned stacks so you can remove them before they hold ports and memory.

Remote or distributed execution changes one requirement: the compose-up step and the tests must run in the same task on the same host. If an agent executes the e2e target on a machine where the dependencies were never started, the tests have nothing to connect to.

Copy#!/usr/bin/env bash
# Vendor-neutral CI script: run affected e2e targets with bounded parallelism.
set -euo pipefail

# Full history so the merge base with main is reachable (no shallow clone).
git fetch --no-tags --prune origin '+refs/heads/main:refs/remotes/origin/main'
git fetch --unshallow 2>/dev/null || true

BASE=$(git merge-base origin/main HEAD)
HEAD_SHA=$(git rev-parse HEAD)

# Illustrative estimate: (16 GB - 3 GB reserved) / ~3 GB per task ~= 4; keep headroom.
PARALLEL=3

# Opt-in: set E2E_KEEP_UP=1 so the teardown step leaves stacks running for debugging.
export E2E_KEEP_UP="${E2E_KEEP_UP:-0}"

if ! npx nx affected -t e2e --base="$BASE" --head="$HEAD_SHA" --parallel="$PARALLEL"; then
  echo "e2e failed; compose stacks still present on this host:"
  docker compose ls
  # Read logs for one task's stack using the project name your target derives, e.g.:
  #   docker compose -p e2e-orders-api logs --no-color --tail=200
  exit 1
fi

# On persistent runners, list stacks left by killed jobs so they can be removed:
#   docker compose ls --all
#   docker compose -p <name> down -v

A provider-agnostic CI script that computes base and head, runs nx affected -t e2e with an explicit --parallel sized from an illustrative memory estimate, and uses docker compose ls and logs to debug failures.

When to avoid it and the alternatives

Per-project stacks are not free, and several costs show up only at scale.

Memory multiplies. If five affected projects each start Postgres and Redis in parallel, the runner holds five copies. Use --parallel to cap concurrency, or the job runs out of memory before any test fails.

Boot time can dominate. A suite that finishes in ten seconds gains little from isolation if container start and healthcheck polling add thirty. For fast suites, the overhead is most of the run.

The graph hides the dependency. Nx sees a run-commands wrapper, not the services behind it. nx graph will not show that two projects share a database image, so reviewers have to read project.json to find out.

Healthy does not mean ready. A compose healthcheck confirms that Postgres accepts connections. It cannot tell whether your migrations have finished. Tests need their own readiness check, such as polling an application endpoint that returns success only after the schema is in place.

CI runners vary. Some runners have no Docker daemon. Others need privileged mode or Docker-in-Docker, which raises security and configuration concerns. Without a registry mirror or pull-through cache, every job pulls images from scratch.

Impact on development cost and engineering time

The cost shift is easiest to see in CI minutes. A global stack charges every pipeline for every service. Per-project compose targets charge only for what the affected projects declare.

Illustrative estimate (assumed numbers, not a benchmark)

• Formula: minutes saved per pipeline = T_global − Σ T_deps(affected project)
• Assume a full stack takes 6 minutes to start and become healthy.
• Assume a typical PR affects 2 e2e projects, and each needs about 2 minutes of dependency startup.
• Saving per pipeline: 6 − (2 × 2) = 2 minutes.
• At an assumed 200 pipelines per week, that is about 400 runner-minutes saved weekly.

Put in your own numbers. If Σ T_deps often exceeds T_global, the pattern costs you money instead of saving it.

Engineering time usually matters more than runner time. A new engineer can open one project's compose file and see exactly which services its tests need, instead of reverse-engineering a shared stack. Local and CI runs read the same environment-based configuration, so "passes on my machine" stops being a separate investigation. Guaranteed teardown and isolated project names remove a class of flakes that used to eat hours: leaked containers, stale volumes and port collisions.

The cost is upkeep. Each compose file, health check and inputs declaration is code that drifts. An undeclared input means a broken test gets skipped, and nothing warns you.

Signals to watch

• It is paying off when most PRs start only a fraction of the services, and dependency-related flakes trend toward zero.
• A shared stack fits better when most changes touch most projects, when compose files are near-duplicates of each other, or when the team keeps missing inputs in review.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx e2e docker compose, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.