What is an Nx self-hosted remote cache and how does it work?

Every Nx task gets a hash before it runs. Nx computes that hash from the task's declared inputs: the project's source files, files from the projects it depends on, relevant configuration such as nx.json and project settings, external dependency versions from the lockfile, and any environment variables or runtime commands you list as inputs. If the hash matches a previous run, Nx replays the stored outputs and terminal log instead of executing the task.

A remote cache is a shared key-value store keyed by that hash. One machine computes an artifact, and every other machine with the same hash can reuse it. Teams usually let CI write and let everyone else read:

  CI (main branch) ──write──┐
                            ▼
                    ┌───────────────┐
                    │ S3/GCS bucket │  key: task hash
                    │               │  value: outputs + logs
                    └───────────────┘
                     ▲             ▲
          read-only  │             │  read-only
       developer laptops        PR pipelines

There are two ways to self-host it:

  • Official storage plugins. Nx publishes plugins that read from and write to S3-compatible storage or GCS directly.
  • A custom server. You build a service against Nx's published remote cache API and put your own storage and authentication behind it.

Package names, licensing terms and config keys for both routes have changed over time. Check the current Nx docs before you copy any configuration.

When the cache returns a wrong result, the storage layer is rarely the cause. In most cases a task read a file, an environment variable or a tool version that was not declared as an input. Two runs that should differ then produce the same hash, and the cache returns the stored result exactly as designed.

 ┌───────────────────────────────────────────────────────────┐
 │ task hash = f(source files, dependency projects' files,   │
 │              nx.json + project config, lockfile versions, │
 │              declared env vars / runtime commands)        │
 └─────────────────────────────┬─────────────────────────────┘
                               │ computed on every machine
                               ▼
 ┌──────────────────┐  write  ┌───────────────────────────┐
 │ CI (main branch) │────────▶│      S3 / GCS bucket      │
 │ runs task on miss│         │  key:   task hash         │
 └──────────────────┘         │  value: outputs + logs    │
                              └───────────────────────────┘
                                 ▲                   ▲
                       read-only │                   │ read-only
                    ┌────────────┴─────┐    ┌────────┴─────────┐
                    │ developer laptops│    │   PR pipelines   │
                    │ replay on hit    │    │  replay on hit   │
                    └──────────────────┘    └──────────────────┘

 Undeclared input (file, env var, tool version) ⇒ same hash for
 different runs ⇒ wrong "hit" returned. Fix the inputs, not storage.

Every machine derives the same task hash from declared inputs. CI on the main branch writes outputs and logs to the bucket under that hash, and developer laptops and PR pipelines only read from it.

Who should self-host the Nx cache, and when should you avoid it?

Self-hosting is an artifact-store design problem: you own a bucket, an access policy, and a set of identities that read and write to it. That suits backend and platform teams that already run cloud storage and IAM, because the cache becomes one more bucket with lifecycle rules and audit logging.

Why teams adopt it

  • Data residency. Cached outputs can include compiled code, generated clients, and test reports. Keeping them in a bucket in a region you choose helps you meet residency or contractual requirements.
  • Artifacts stay in your account. Access logs, encryption keys, and retention follow the same policies as the rest of your infrastructure. You do not need a separate vendor review.
  • Less duplicated CI work. When one pipeline has already built or tested a project at a given hash, other pipelines and developers can reuse that result. The saving is engineer time spent waiting on CI as much as compute.

When to avoid it

  • Nobody owns it. Someone has to manage the bucket, rotate credentials, respond when the bucket is unavailable, and investigate unexpected misses. Without a clear owner, the cache usually decays into something people disable.
  • Runners are far from the bucket. If CI runs in a different region or cloud, each cache read adds latency and possibly egress charges. For fast tasks like linting or small unit test suites, rebuilding can be cheaper than downloading.
  • Most tasks are non-hermetic. If many tasks depend on undeclared inputs, sharing the cache spreads wrong results to more machines.

Self-hosting also makes you dependent on the Nx cache plugin for your storage backend. Review its current licensing terms before you commit, and recheck them when you upgrade.

How do you provision an S3 bucket and IAM policy for the Nx cache?

Start with a dedicated bucket, or at minimum a dedicated prefix such as nx-cache/ in an existing bucket. Isolating the cache keeps the IAM policy small and lets you apply a lifecycle rule that expires old artifacts without touching anything else. Turn on all four S3 Block Public Access settings at the bucket level. Cache artifacts can contain build output, bundled environment values, or source maps, and a single misconfigured bucket policy should not be able to expose them.

New S3 buckets encrypt objects with SSE-S3 by default. If your organization requires SSE-KMS, the cache principals also need kms:Decrypt and kms:GenerateDataKey on that key. Missing KMS permissions are a common reason that uploads fail while reads still appear to work.

Scope the read/write policy to the bucket ARN and the cache prefix. Typically that means s3:GetObject and s3:PutObject on arn:aws:s3:::your-bucket/nx-cache/*. If the plugin checks whether an object exists before fetching it, it may also need s3:ListBucket on the bucket ARN, restricted with an s3:prefix condition. The read-only variant drops s3:PutObject. Do not grant s3:DeleteObject; expiry belongs to the lifecycle rule. Read the plugin's documentation for the exact action list instead of guessing. Some plugins issue HeadObject or multipart calls that need permissions you would not expect.

For S3-compatible stores such as MinIO or Ceph, you set a custom endpoint, and you usually need path-style addressing (endpoint/bucket/key). Many self-hosted deployments lack the wildcard DNS and TLS certificates that virtual-hosted-style requests (bucket.endpoint/key) depend on.

#!/usr/bin/env bash
set -euo pipefail
BUCKET=acme-nx-cache
REGION=eu-west-1
PREFIX=nx-cache

# Dedicated bucket (LocationConstraint is required outside us-east-1)
aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" \
  --create-bucket-configuration LocationConstraint="$REGION"

# Turn on all four Block Public Access settings so a bad policy cannot expose artifacts
aws s3api put-public-access-block --bucket "$BUCKET" \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

# Set SSE-S3 explicitly. If you switch to aws:kms, grant kms:Decrypt and kms:GenerateDataKey on the key
aws s3api put-bucket-encryption --bucket "$BUCKET" \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

# Read/write policy: objects under the cache prefix only, and no s3:DeleteObject (expiry is the lifecycle rule's job).
# Check your plugin's docs for the exact action list before applying this policy.
cat > nx-cache-rw.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "CacheObjects", "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::${BUCKET}/${PREFIX}/*" },
    { "Sid": "ListCachePrefixOnly", "Effect": "Allow", "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::${BUCKET}",
      "Condition": { "StringLike": { "s3:prefix": "${PREFIX}/*" } } }
  ]
}
EOF

# Read-only variant: same policy with s3:PutObject removed (for example, for untrusted PR builds)
jq '.Statement[0].Action = ["s3:GetObject"]' nx-cache-rw.json > nx-cache-ro.json

aws iam create-policy --policy-name nx-cache-rw --policy-document file://nx-cache-rw.json
aws iam create-policy --policy-name nx-cache-ro --policy-document file://nx-cache-ro.json

This AWS CLI script creates a dedicated cache bucket with Block Public Access and default encryption turned on, then creates a read/write IAM policy scoped to the cache prefix and a read-only variant derived from it.

How do you set up a GCS cache bucket with Workload Identity Federation?

On GCS, start by creating the bucket with uniform bucket-level access enabled (gcloud storage buckets create with --uniform-bucket-level-access). This turns off per-object ACLs, so IAM becomes the only access control. Without it, an object written with an unexpected ACL can quietly become readable or unreadable, and debugging that across thousands of cache artifacts is painful.

Next, create a dedicated service account with gcloud iam service-accounts create, for example nx-cache-writer. Grant it roles on the bucket with gcloud storage buckets add-iam-policy-binding, not on the project. A project-level storage role applies to every bucket in the project, which turns a cache credential into a path to your other data.

Read-only versus read/write bindings

  • Read-only principals get roles/storage.objectViewer. This covers developer groups and untrusted CI jobs such as fork pull requests.
  • Writers get roles/storage.objectViewer plus roles/storage.objectCreator. Creator can add objects but cannot delete them, so a writer cannot overwrite an existing cache entry. That suits content-addressed artifacts.
  • Reserve roles/storage.objectUser, which includes delete, for whatever handles cleanup. In most setups that is a lifecycle rule, which needs no principal at all.

Why Workload Identity Federation for CI

A JSON key is a long-lived bearer secret. It gets copied into CI variables, outlives the people who created it, and rarely gets rotated. With Workload Identity Federation, the CI system presents its own short-lived OIDC token, and Google exchanges it for temporary credentials. Set this up by creating a pool and an OIDC provider (gcloud iam workload-identity-pools create, then providers create-oidc). Restrict the provider with an attribute condition on repository and branch. Then grant roles/iam.workloadIdentityUser on the service account to only the matching principal set. This way, only protected-branch pipelines can impersonate the writer. Fall back to JSON keys only when your CI system cannot issue OIDC tokens.

PROJECT_ID="example-project"
PROJECT_NUMBER="123456789012"
BUCKET="gs://example-nx-cache"
SA="nx-cache-writer@${PROJECT_ID}.iam.gserviceaccount.com"
ISSUER_URI="<your CI system's OIDC issuer URL>"

# Uniform access: IAM is the only access control, no per-object ACLs
gcloud storage buckets create "$BUCKET" --project="$PROJECT_ID" \
  --location=us-central1 --uniform-bucket-level-access

gcloud iam service-accounts create nx-cache-writer --project="$PROJECT_ID" \
  --display-name="Nx remote cache writer"

# Bind roles on the bucket, not the project. Writer: read + create, no delete
for ROLE in roles/storage.objectViewer roles/storage.objectCreator; do
  gcloud storage buckets add-iam-policy-binding "$BUCKET" \
    --member="serviceAccount:${SA}" --role="$ROLE"
done

# Developers (and untrusted jobs) get read-only access
gcloud storage buckets add-iam-policy-binding "$BUCKET" \
  --member="group:nx-devs@example.com" --role=roles/storage.objectViewer

# Workload Identity Federation: CI exchanges its OIDC token, no JSON key
gcloud iam workload-identity-pools create ci-pool --project="$PROJECT_ID" \
  --location=global --display-name="CI pool"

# Claim names vary by CI system; restrict to the trusted repository
gcloud iam workload-identity-pools providers create-oidc ci-oidc \
  --project="$PROJECT_ID" --location=global --workload-identity-pool=ci-pool \
  --issuer-uri="$ISSUER_URI" \
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository" \
  --attribute-condition="assertion.repository == 'example-org/monorepo'"

# Only identities from that repository may impersonate the writer SA
gcloud iam service-accounts add-iam-policy-binding "$SA" --project="$PROJECT_ID" \
  --role=roles/iam.workloadIdentityUser \
  --member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/ci-pool/attribute.repository/example-org/monorepo"

Creates a uniform-access GCS cache bucket, binds read-only and read/create roles at the bucket level, and lets a specific CI repository impersonate the writer service account through Workload Identity Federation instead of a JSON key.

How should CI and developer machines authenticate, and who gets write access?

Whatever the provider, CI should get credentials that expire within the hour and are scoped to a single job. Both clouds accept GitHub Actions, GitLab, and similar OIDC tokens, so no long-lived key needs to sit in CI settings. On laptops, rely on the standard credential chains: aws sso login or a named profile for S3, and gcloud auth application-default login for GCS. The Nx cache client reads these the same way any SDK does. Keep bucket names in nx.json if you like, but never put keys there or anywhere else in the repository. Anything committed is readable by every clone and every fork.

The trust split

The cache stores outputs keyed by a hash of the inputs. Anyone who can write an entry can plant a malicious bundle under a hash that a release pipeline will later fetch and ship. This is cache poisoning, and the hash cannot detect it because the hash covers the inputs, not the stored output. So only CI on protected branches writes. Everyone else reads.

Identity S3 permissions GCS role Writes?
CI, protected branches s3:GetObject, s3:PutObject, s3:ListBucket roles/storage.objectCreator + objectViewer Yes
CI, PRs from branches s3:GetObject, s3:ListBucket roles/storage.objectViewer No
Fork PRs None, or read-only via a separate role None, or objectViewer No
Developer laptops s3:GetObject, s3:ListBucket roles/storage.objectViewer No

Letting developers write raises the hit rate, because their local builds warm the cache before CI runs. The cost is that one compromised laptop can poison builds for the whole team. Most teams should accept the lower hit rate.

Cached artifacts and replayed terminal logs can contain environment values and tokens. Apply the same encryption, retention, and access logging to the bucket that you apply to other build artifacts.

# .github/workflows/ci.yml
# CI gets short-lived AWS credentials via OIDC. No access keys are stored in the repo or in CI settings.
name: ci
on:
  push:
    branches: [main]
  pull_request:

permissions:
  id-token: write   # allow the job to request a GitHub OIDC token
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Pushes to protected main assume the writer role. Everything else assumes the reader role.
      # The IAM trust policy for each role must also check the token's 'sub' claim
      # (for example repo:acme/platform:ref:refs/heads/main for the writer).
      # The workflow expression alone does not enforce the split.
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'arn:aws:iam::123456789012:role/nx-cache-writer' || 'arn:aws:iam::123456789012:role/nx-cache-reader' }}
          aws-region: eu-west-1
          role-duration-seconds: 3600   # credentials expire within the hour

      - run: npm ci
      # The Nx cache client picks up credentials from the standard AWS environment.
      # On the reader role, uploads fail with AccessDenied, which is the intended behavior.
      - run: npx nx affected -t build test

# Fork PRs: GitHub does not issue an OIDC token to pull_request runs from forks,
# so the configure step fails and those builds get no cache credentials at all.
# To give forks read access, use a separate, narrower reader role.

A GitHub Actions workflow that uses OIDC to assume a write-capable IAM role only for pushes to main and a read-only role for every other build, so no long-lived keys are stored in the repository or CI settings.

How do you keep a shared Nx cache correct and verify it works?

A local cache confines a hashing mistake to one laptop. A remote cache serves that mistake to every CI runner and developer who computes the same hash. If a task's output depends on something Nx does not hash, the first machine to run it decides the result for everyone else.

Declare everything that affects output

In each target's inputs, list the source globs, then add the hidden dependencies:

  • { "env": "API_BASE_URL" } for every environment variable that changes the build.
  • { "runtime": "node --version" } for compiler or runtime versions.
  • { "externalDependencies": [...] } for tools whose version matters.

Declare outputs explicitly too. Nx can only restore paths it knows about, so a missing output turns a cache hit into a build that silently lacks artifacts.

Exclude non-hermetic tasks

Set "cache": false on any target that reads the clock, calls the network, or shells out to an unpinned global tool. These tasks produce different results from identical inputs, so a cache hit replays one arbitrary run as if it were correct for every later one.

Debug a suspected bad hit

Pass --skip-nx-cache, or set NX_SKIP_NX_CACHE=true, to force execution. Then compare the fresh output with the cached one. A difference means some input is not declared.

Verify the setup

  1. Run the target in CI and confirm the run uploads to the bucket.
  2. On a clean machine, run nx reset, run the same target, and check that the terminal output reports a remote cache hit.
  3. List the bucket and confirm there is an object keyed by that task hash.
  4. Change one declared input, such as an environment variable, and confirm the next run misses the cache.

What does a self-hosted Nx cache cost, and how does S3 compare to GCS?

Both providers support age-based expiration: S3 through lifecycle rules, GCS through Object Lifecycle Management. Age counts from object creation, not last read. A cache entry that a long-lived release branch hits every day will still expire, and the next run rebuilds it. If release branches rebuild rarely, set retention longer than their typical gap between runs. Otherwise those branches get no hits at all. To catch runaway growth, track bucket size with each provider's daily storage metrics. Track GET and PUT counts with request metrics, which may need enabling.

Illustrative estimate (assumed numbers, not a benchmark). The assumed inputs are 5 MB per task artifact, 40 tasks per run, 200 runs per day, a 50% miss rate and 14-day retention.

  • Writes per day = tasks × runs × miss rate = 4,000 PUTs
  • Storage ≈ writes × size × retention = 280 GB. This is an upper bound, because identical hashes overwrite each other.
  • Reads per day = tasks × runs = 8,000 GETs
  • Download volume = hits × size = 20 GB/day

Multiply each line by your provider's current rates. Then compare the total with the CI minutes and engineer wait time that cache hits save.

Option Fits when Egress exposure Ops burden
S3 Runners in AWS Low in-region, high cross-cloud Low
GCS Runners in GCP Low in-region, high cross-cloud Low
Custom cache server On-prem or mixed runners You control network path High
Local-only Small repo, few CI runs None None

The 20 GB/day of downloads usually decides the choice. Put the cache in the same cloud and region as your runners.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx self-hosted remote cache, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.