Nx GitLab CI Affected Pipelines: SHAs, Caching and Parallel Jobs

How nx affected works and why GitLab has no nx-set-shas

nx affected answers one question: which projects could have changed between two commits? Nx runs a git diff between a base SHA and a head SHA (more precisely, from their merge base to head), collects the changed files, and maps each file to the project that owns it in the project graph. It then walks the graph to add every project that depends on a changed one. Only those projects get their test, lint or build targets run.

The result depends entirely on the two SHAs. If the base is too recent, a change that was never verified falls out of the diff and ships untested. If the base is too old, the diff grows until almost everything is affected and the optimisation does nothing. Choosing the base is the hard part.

On GitHub, the nx-set-shas Action handles this by finding the commit of the last successful workflow run on the main branch. GitLab has no drop-in equivalent. You compute the SHAs yourself from GitLab's predefined CI variables, then either export them as NX_BASE and NX_HEAD or pass them as --base and --head.

This setup pays off for monorepos with enough projects that running every target on every push wastes noticeable runner time. In a small repository where a full run finishes in a few minutes, the extra pipeline logic costs more to maintain than it saves.

Affected detection also only sees what the project graph knows. Changes to root configuration, .gitlab-ci.yml, shared scripts or any input a project reads without declaring it may not mark the right projects as affected. Handle these explicitly, either by declaring them as project inputs or by forcing a full run when they change.

Copy+------------------------------+     +--------------------------------+
| GitLab predefined CI vars    |     | Not seen by project graph:     |
| (you compute SHAs yourself;  |     | root config, .gitlab-ci.yml,   |
|  no nx-set-shas on GitLab)   |     | shared scripts, undeclared     |
+--------------+---------------+     | inputs                         |
               |                     +---------------+----------------+
               v                                     |
+------------------------------+                     | declare as inputs
| NX_BASE / NX_HEAD            |                     | or force full run
| or --base / --head           |                     |
+--------------+---------------+                     |
               |                                     |
               v                                     |
+------------------------------+                     |
| git diff: merge-base -> head |                     |
| = list of changed files      |                     |
+--------------+---------------+                     |
               |                                     |
               v                                     |
+------------------------------+                     |
| Project graph                |<--------------------+
| file -> owning project       |
| + all dependent projects     |
+--------------+---------------+
               |
               v
+------------------------------+
| Run test / lint / build      |
| on affected projects only    |
+------------------------------+

On GitLab you derive the base and head SHAs from predefined CI variables, Nx diffs from their merge base to head, and maps the changed files onto the project graph to pick affected projects; inputs the graph cannot see must be declared or must force a full run.

Who should use affected pipelines on GitLab

This setup is for teams that keep an Nx workspace on GitLab and run its checks in GitLab CI. It fits best when:

• The workspace has many projects with clear boundaries. Affected detection only saves work when a typical change touches a small part of the graph. If most libraries depend on one shared core that changes constantly, nearly every pipeline is a full run anyway.
• Someone owns the CI configuration. The SHA script, the workflow:rules and the cache layout are code that needs a maintainer. That person should be comfortable reading git behaviour around shallow clones and merge bases, because that is where most failures come from.
• Project inputs are declared accurately. Teams that already keep inputs and outputs in their Nx configuration up to date get correct affected sets and safe cache replays. Teams that rely on undeclared files or environment variables should fix that first, or they will trade runner time for silent misses.

It is a poor fit for single-application repositories, for small workspaces where a full run is already fast, and for teams that cannot spare time to debug CI logic. Teams whose process requires every target to run on every merge can still use the pattern for merge request feedback while keeping full runs on the default branch.

When to use it

Adopt affected pipelines when full pipelines have become a bottleneck: merge request feedback takes long enough that developers context-switch while waiting, runner queues back up at busy times, or runner-minute spend is a line item someone asks about. Check first that the problem is the number of targets being run, not one slow test suite. Affected detection cannot shorten a single project's tests.

Within a working setup, use each mode for a specific pipeline type:

• Merge request pipelines: run affected targets against the MR diff base. This is where the fast feedback matters most.
• Default branch pipelines: run affected targets against the last successful pipeline, so nothing a failed pipeline skipped stays untested.
• Scheduled pipelines and root-file changes: run every target. This is the safety net for anything the project graph cannot see.

Hold off when the workspace is still being restructured and project boundaries move every week, or when you cannot yet log and inspect the resolved SHAs. Without that visibility, a wrong base looks exactly like a green build.

How the pipeline works end to end

The rest of this article builds the pieces in this order. At runtime they fit together like this:

1. GitLab decides the pipeline type. workflow:rules allow either a merge request pipeline or a branch pipeline for a given commit, never both.
2. A shared setup step resolves the SHAs. Every Nx job extends a hidden job whose before_script sources a shell script. The script picks NX_BASE from GitLab's predefined variables (or the nx-last-green tag on the default branch), sets NX_HEAD to CI_COMMIT_SHA, fetches the base if the shallow clone lacks it, and fails loudly if the commit still cannot be found.
3. Caches are restored. The lockfile-keyed dependency cache and the Nx computation cache are pulled separately, with exactly one job allowed to write each.
4. Nx computes the affected set. It diffs from the merge base of NX_BASE and NX_HEAD to head, maps changed files to projects, and adds their dependents.
5. Work is distributed. Either a single job runs nx affected, parallel: N shards split a sorted project list, or a generator job writes a child pipeline with one job per project.
6. Results are recorded. Each job logs the base, head and affected list. On success, a final job on the default branch moves the nx-last-green tag, which becomes the next pipeline's base.

Choosing base and head SHAs per pipeline type

The first thing to settle is what Nx compares against: NX_BASE and NX_HEAD decide which projects count as affected.

In merge request pipelines, CI_MERGE_REQUEST_DIFF_BASE_SHA is the commit that GitLab's diff view uses. Nx then sees the same change set a reviewer sees.

On the default branch, CI_COMMIT_BEFORE_SHA is easy to use but only looks back one push. Suppose commit A breaks api and commit B fixes it. B's pipeline compares against A, so projects that A affected and B didn't touch are never tested green. Comparing against the last successful pipeline closes that gap. You can get that SHA in two ways:

• Pipelines API. Query ref=<default branch> with status=success. First confirm your instance's job token permissions allow this endpoint. If they don't, use a project access token with read scope.
• Moving tag. Push a tag such as nx-last-green in a final job that runs only on success.

CI_COMMIT_BEFORE_SHA is forty zeros on new branches and first pushes. Treat that value as "no base": either fall back to the default branch tip or run all targets.

Shallow clones are fast, but if the base falls outside GIT_DEPTH, Nx fails. Raise the depth, or run an explicit git fetch of the base SHA. The script below resolves the base this way, exports both variables, and exits non-zero if git cat-file -e cannot find the base. A silent fallback to HEAD~1 would hide the problem.

Copy#!/usr/bin/env bash
set -euo pipefail
ZERO_SHA="0000000000000000000000000000000000000000"
export NX_HEAD="$CI_COMMIT_SHA" NX_RUN_ALL=false NX_BASE=""

ensure_commit() {
  # A shallow clone may not contain the base; fetch it with some history so merge-base works.
  git cat-file -e "$1^{commit}" 2>/dev/null || git fetch --no-tags --depth=200 origin "$1" || true
  git cat-file -e "$1^{commit}" 2>/dev/null || {
    echo "ERROR: base commit $1 not in clone; raise GIT_DEPTH or check the ref" >&2; exit 1; }
}

if [ "${CI_PIPELINE_SOURCE:-}" = "merge_request_event" ]; then
  # Same base the MR diff view uses, so Nx sees what reviewers see.
  NX_BASE="$CI_MERGE_REQUEST_DIFF_BASE_SHA"
elif [ "${CI_COMMIT_BRANCH:-}" = "$CI_DEFAULT_BRANCH" ] &&
     git fetch --no-tags origin "+refs/tags/nx-last-green:refs/tags/nx-last-green" 2>/dev/null; then
  # Tag moved by a success-only job: covers commits after a failed pipeline.
  NX_BASE="$(git rev-parse 'nx-last-green^{commit}')"
elif [ "${CI_COMMIT_BEFORE_SHA:-$ZERO_SHA}" != "$ZERO_SHA" ]; then
  echo "WARN: no nx-last-green tag; using CI_COMMIT_BEFORE_SHA (may skip work after a red pipeline)" >&2
  NX_BASE="$CI_COMMIT_BEFORE_SHA"
elif [ "${CI_COMMIT_BRANCH:-}" != "$CI_DEFAULT_BRANCH" ]; then
  # New branch: all-zeros before SHA, so compare against the default branch tip.
  git fetch --no-tags --depth=200 origin "$CI_DEFAULT_BRANCH"
  NX_BASE="$(git rev-parse FETCH_HEAD)"
else
  # First push to the default branch with no known-good base: test everything.
  echo "WARN: no usable base; running all targets" >&2
  NX_RUN_ALL=true
fi

if [ "$NX_RUN_ALL" = "false" ]; then ensure_commit "$NX_BASE"; fi
export NX_BASE
echo "NX_BASE=${NX_BASE:-<none>} NX_HEAD=$NX_HEAD NX_RUN_ALL=$NX_RUN_ALL"

A sourced shell script that picks NX_BASE per pipeline type (MR diff base, last-green tag, CI_COMMIT_BEFORE_SHA, or default-branch tip), handles the all-zeros SHA, fetches a base missing from a shallow clone, and exits with an error if the commit still cannot be found.

Separating merge request and branch pipelines with rules

Before any job runs, decide which pipeline type a commit gets. By default, pushing to a branch with an open merge request can create two pipelines: a branch pipeline and a merge request pipeline. This happens when some jobs have merge_request_event rules and others do not. Each pipeline resolves its base differently. The MR pipeline uses CI_MERGE_REQUEST_DIFF_BASE_SHA. The branch pipeline uses the previous push or the default branch tip. They can compute different affected sets for the same commit.

Nothing errors when this happens. You pay for two runs. Worse, the branch pipeline may compare against CI_COMMIT_BEFORE_SHA, which covers only the latest push. It can then report green while skipping projects that earlier commits in the MR touched. If reviewers look at the wrong pipeline, untested code merges.

The fix belongs in workflow:rules, evaluated top to bottom:

1. Run when CI_PIPELINE_SOURCE is merge_request_event.
2. Set when: never when CI_COMMIT_BRANCH is set and CI_OPEN_MERGE_REQUESTS is non-empty. This suppresses the duplicate branch pipeline.
3. Run for any other CI_COMMIT_BRANCH, which covers the default branch and branches without an MR.

Tags and schedules need their own explicit rules if you want them.

Job-level rules should only narrow this further, for example limiting deploy jobs to the default branch. They should never reintroduce a pipeline source that the workflow excluded.

The SHA script then runs from a hidden .nx-base job. Its before_script sources the script, so NX_BASE and NX_HEAD stay set for the job's script, because both run in the same shell. Every Nx job inherits it through extends. As a result, no job computes its own base.

Copyworkflow:
  rules:
    # 1. MR pipelines: base comes from CI_MERGE_REQUEST_DIFF_BASE_SHA.
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    # 2. Branch push with an open MR: suppress the duplicate branch pipeline.
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    # 3. Default branch and branches without an MR.
    - if: $CI_COMMIT_BRANCH
    # Tags and schedules are not listed, so they do not run unless you add rules.

.nx-base:
  image: node:20
  before_script:
    - npm ci
    - . ./scripts/nx-set-shas.sh   # exports NX_BASE and NX_HEAD

lint:
  extends: .nx-base
  script:
    - npx nx affected -t lint --base="$NX_BASE" --head="$NX_HEAD"

test:
  extends: .nx-base
  script:
    - npx nx affected -t test --base="$NX_BASE" --head="$NX_HEAD"

deploy:
  extends: .nx-base
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - npx nx affected -t deploy --base="$NX_BASE" --head="$NX_HEAD"

workflow:rules allow either an MR pipeline or a branch pipeline for a commit, never both, and every Nx job extends a hidden .nx-base job that sources the SHA script once in before_script.

Caching dependencies and the Nx cache on GitLab

Two caches matter here, and they should stay separate. The first holds package-manager dependencies. Key it with cache:key:files pointing at your lockfile, so the key changes only when dependencies change. The second is the Nx computation cache. Give it its own cache entry with its own key, because it changes on almost every commit and should not invalidate or bloat the dependency archive. Check which directory your Nx configuration actually writes to before you list it under paths. The default location has changed across Nx releases, and cacheDirectory can override it. If the path is wrong, the cache archives nothing and you won't see an error.

Set policy: pull on most jobs and pull-push on a single job, usually the install job on the default branch. Concurrent jobs that all push the same key overwrite each other, and the last one to finish wins. With one designated writer, the stored contents are predictable.

GitLab's cache guarantees less than it might seem. It lives on each runner unless a distributed cache is configured. It is best-effort, so any job can start cold. It is also keyed coarsely: one key maps to one archive. On ephemeral runners, every job downloads and extracts the whole archive. A large node_modules or Nx cache can take longer to restore than the tasks it saves.

A restored Nx cache is only correct if each target declares accurate inputs and outputs. If a target omits an input, such as an environment variable or a config file, Nx will replay an old successful result for code that would now fail.

Copy.deps-cache: &deps-cache
  key:
    files:
      - package-lock.json      # key changes only when dependencies change
  paths:
    - .npm/                    # npm download cache, not node_modules (npm ci deletes it)
  policy: pull

.nx-cache: &nx-cache
  key: nx-$CI_COMMIT_REF_SLUG  # own key, so frequent changes don't touch the deps archive
  paths:
    - .nx/cache/               # verify against your Nx config before relying on it
  policy: pull

install:
  stage: .pre
  script:
    - npm ci --cache .npm --prefer-offline
  cache:
    - <<: *deps-cache
      policy: pull-push        # the only job that writes the deps cache

test:
  stage: test
  script:
    - npm ci --cache .npm --prefer-offline
    - npx nx affected -t test
  cache:
    - *deps-cache
    - *nx-cache                # read-only: parallel jobs never overwrite each other

build:
  stage: build
  script:
    - npm ci --cache .npm --prefer-offline
    - npx nx affected -t build
  cache:
    - *deps-cache
    - <<: *nx-cache
      policy: pull-push        # single designated writer for the Nx cache

A GitLab CI config that keeps a lockfile-keyed npm cache separate from the Nx cache and gives each cache exactly one pull-push job, so concurrent jobs cannot overwrite each other's archives.

Splitting affected work across parallel jobs

GitLab's parallel: N keyword starts N copies of the same job and gives each one CI_NODE_INDEX (1-based) and CI_NODE_TOTAL. Nx knows nothing about these variables, so each job has to work out its own slice of the affected set.

The shard script does this in four steps:

1. Run nx show projects --affected --withTarget=test to list only the projects that have the target.
2. Pipe the list through sort. Every job must compute an identical list, otherwise two shards can claim the same project while another project is skipped.
3. Keep every line where (line_number - 1) % CI_NODE_TOTAL == CI_NODE_INDEX - 1.
4. Pass the result to nx run-many -t test --projects=<comma-separated list>.

If a shard receives no projects, the script should print a message and exit 0. Calling run-many with an empty project list will either fail or run more than intended. The job definition sets parallel, uses the same base and head SHAs as the other jobs, and calls the script.

Job-level parallelism and Nx's --parallel flag operate at different levels:

• parallel: N spreads work across separate runners.
• --parallel sets how many tasks Nx runs at once inside a single job.

Combine them when each runner has several cores and the affected set is large. Use a few shards, each with --parallel matched to its CPU count.

Round-robin sharding balances the number of projects, not runtime. If one project's tests take most of the total time, the shard that gets it sets the pipeline's duration no matter what N is.

Small affected sets cause the opposite problem. Shards with little or no work still pay for runner startup, image pull and dependency install, so a high N can cost more runner minutes than it saves in wall-clock time.

Static sharding vs dynamic child pipelines and other alternatives

When shard sizes vary this much, a fixed parallel: N is the wrong shape. The alternative is a dynamic child pipeline. A generator job lists the affected projects, writes a YAML file with one job per project or target, and stores it as an artifact. A trigger job then starts that file with trigger:include using artifact: and job:. The generator snippet follows this section.

The generator is a full job of its own. It needs a runner, a checkout and a dependency install before the child pipeline can start. The artifact handoff adds another point where the pipeline can fail. The pipeline view nests the child pipeline under the trigger job, so a reviewer has to expand it to find the failing test. Merge request status checks only follow the child pipeline if the trigger job uses strategy: depend. Without it, the parent pipeline can pass while the child pipeline is still running or has already failed.

You can also skip sharding entirely. One job can run all targets against a shared remote Nx cache, and cache hits make the unchanged projects fast. That cache can be Nx's remote cache or a cache you build from GitLab's cache and artifacts. GitLab's cache is scoped by key and by runner, so hit rates are usually lower.

Verifying the setup and its impact on cost and engineering time

Print three things at the start of every job: the resolved base SHA, the head SHA, and the affected project list. When a pipeline runs too much or too little, these lines are your only record of why. Add a scheduled pipeline on the default branch that runs every target. It catches whatever affected detection missed. Also force a full run when root files change: nx.json, the lockfile, the base tsconfig, .gitlab-ci.yml and your CI scripts. Nx does not treat CI scripts as part of the project graph, so a change to them would otherwise go untested.

Illustrative estimate (assumed numbers, not a benchmark). Assume 40 projects, a full run of 60 runner-minutes, 6 affected projects per merge request, and 200 MR pipelines per week. Assume cost is spread evenly across projects.

Saved per week = full × (1 − affected ÷ total) × pipelines = 60 × (1 − 6/40) × 200 = 10,200 runner-minutes

Subtract the ongoing costs from that figure:

• Maintaining the SHA scripts.
• Rotating a project-scoped, read-only API token.
• Debugging silent misses. A single regression that reaches the default branch unnoticed can cost more engineering time than the compute you saved that week.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx gitlab ci affected, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.