Nx Self-Hosted Remote Cache: S3 vs GCS vs Azure Blob Setup

What is an Nx self-hosted remote cache and how does it work?

Before running a task, Nx computes a hash from everything that can affect its result: the relevant source files, the task's configuration, declared environment and runtime inputs, and the hashes of the tasks it depends on. That hash is the cache key. When the task finishes, Nx stores the declared outputs and the terminal log under that key. The terminal log matters because a cache hit replays the original output, so a restored build looks the same as a fresh one.

Nx checks its local cache first and then the remote cache. On a remote hit, it downloads the artifact, restores the outputs and skips the task. On a miss, it runs the task and uploads the result. Because the remote cache is a key-value lookup on content hashes, any object store can back it: S3, GCS or Azure Blob. Something only has to translate Nx's cache reads and writes into bucket operations.

There are two ways to do that:

• A cache adapter configured in nx.json. Nx talks to the bucket directly, using credentials from the environment.
• A small HTTP cache server in front of the bucket. Nx calls the server, and the server holds the bucket credentials and enforces access rules.

The supported mechanism and package names have changed across Nx releases. Check the current Nx documentation for your version before choosing one.

Copy Dev machines          CI runners
      |  ^                 |  ^
      |  | read            |  | read
      |  |       write     |  |
      |  |   +-------------+  |
      v  |   v                |
 +-----------------------------+
 |  Adapter or HTTP cache svr  |
 +-----------------------------+
      |  GET (read)   ^
      |  PUT (write)  | artifact
      v               |
 +-----------------------------+
 |   S3 / GCS / Azure Blob     |
 +-----------------------------+

Copy+------------------+              +------------------+
|  Dev machines    |              |   CI runners     |
|  (local cache)   |              |  (local cache)   |
+------------------+              +------------------+
   |          ^                      |          ^
   | write    | read                 | write    | read
   | (upload  | (hit: restore        | (upload  | (hit: restore
   |  on miss)|  outputs + log)      |  on miss)|  outputs + log)
   v          |                      v          |
+-----------------------------------------------------------+
|      Cache adapter (nx.json)  or  HTTP cache server       |
|             key = task hash (computed by Nx)              |
+-----------------------------------------------------------+
   |                    ^                    |
   | GET <hash>         | artifact           | PUT <hash>
   | (read path)        | (outputs + log)    | (write path)
   v                    |                    v
+-----------------------------------------------------------+
|                    Object store bucket                    |
|                 S3  /  GCS  /  Azure Blob                 |
+-----------------------------------------------------------+

Developer machines and CI runners check their local cache, then read cached artifacts by task hash through an adapter or HTTP cache server backed by a bucket, and upload new results along a separate write path after a cache miss.

Who should self-host remote caching, and when should you avoid it?

A self-hosted cache pays off when three conditions line up. First, the monorepo has tasks that are expensive to recompute: TypeScript builds of large libraries, integration test suites, or lint runs across hundreds of projects. Second, CI already runs inside one cloud, so runners can reach a bucket in the same region over short, cheap network paths. Third, there is a policy or contractual requirement to keep build artifacts in an account you control. That third point often decides the question on its own.

When to avoid it

• Mostly small, fast tasks. A cache hit still needs a hash computation, a network round trip and an archive extraction. If a task finishes locally in under a second, a remote fetch can be slower than simply running it again.
• No one to own security and IAM. The cache is a write path into every developer machine and CI job. It needs someone to rotate trust policies, review permissions and respond if artifacts are tampered with. If nobody has that capacity, don't run it.
• CI and bucket in different clouds. Every cache read becomes cross-cloud egress. You pay for it on every pipeline, and the extra latency reduces the speedup.
• Unmaintained community adapters. If the integration you depend on stops being updated and has no fallback to local execution, a broken cache can block builds instead of only slowing them.

The benefit is concentrated in a small number of costly tasks. Measure cache hit rates and task durations on your own pipelines before you assume a gain.

How do you set up an Nx S3 cache with OIDC instead of access keys?

Start with a bucket used only for the Nx cache. Enable Block Public Access at the bucket level, turn on default server-side encryption (SSE-S3 is enough unless policy requires KMS), and create it in the same region as your CI runners. Same-region placement lowers latency on every cache lookup and avoids cross-region transfer charges on artifacts that are read far more often than they are written.

Scope the role narrowly

An illustrative permission policy grants s3:GetObject and s3:PutObject on arn:aws:s3:::your-nx-cache/nx/*, and s3:ListBucket on the bucket ARN with an s3:prefix condition restricted to nx/. It grants no delete, no ACL changes and no access to other prefixes.

The trust policy carries most of the security weight. With GitHub Actions OIDC, require aud to equal sts.amazonaws.com. Also require sub to match exactly, for example repo:your-org/your-repo:ref:refs/heads/main. Use StringEquals rather than a wildcard.

Why not access keys

Long-lived keys stored as CI secrets do not expire on their own. They are copied into forks and local shells, and they are rarely rotated. OIDC credentials last only as long as the job, and the trust policy, not secret hygiene, decides who can get them.

How misconfiguration shows up

If the sub condition is too strict, for example because it names the wrong branch or the repository was renamed, role assumption fails. The cache client then typically logs a warning and runs every task locally. Builds pass but get slower. If the condition is too loose, such as repo:your-org/*, any branch in any repository in the organization can write artifacts into the cache.

Copy#!/usr/bin/env bash
set -euo pipefail
BUCKET=your-nx-cache; REGION=eu-west-1   # same region as the CI runners

# Dedicated bucket, no public access, default SSE-S3 encryption.
aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" \
  --create-bucket-configuration LocationConstraint="$REGION"
aws s3api put-public-access-block --bucket "$BUCKET" --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
aws s3api put-bucket-encryption --bucket "$BUCKET" --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

# Trust policy: only GitHub OIDC tokens from main in one repo can assume the role.
# StringEquals on sub means no wildcard. A wrong branch or repo name causes silent cache misses.
cat > trust.json <<'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
  "Principal":{"Federated":"arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"},
  "Action":"sts:AssumeRoleWithWebIdentity",
  "Condition":{"StringEquals":{
    "token.actions.githubusercontent.com:aud":"sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub":"repo:your-org/your-repo:ref:refs/heads/main"}}}]}
EOF

# Permissions: read/write objects under nx/ only. No delete, no ACL changes, no other prefixes.
cat > cache-policy.json <<'EOF'
{"Version":"2012-10-17","Statement":[
  {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],
   "Resource":"arn:aws:s3:::your-nx-cache/nx/*"},
  {"Effect":"Allow","Action":"s3:ListBucket","Resource":"arn:aws:s3:::your-nx-cache",
   "Condition":{"StringLike":{"s3:prefix":["nx/*"]}}}]}
EOF

aws iam create-role --role-name nx-cache-ci --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name nx-cache-ci --policy-name nx-cache-rw \
  --policy-document file://cache-policy.json

Illustrative AWS CLI setup that creates a private, encrypted cache bucket. It also adds a role that only GitHub Actions OIDC tokens from the main branch of one repository can assume, with read and write access limited to the nx/ prefix.

How do you set up an Nx GCS cache with Workload Identity Federation?

Create a dedicated bucket for the cache rather than reusing one that holds artifacts or logs. That way an IAM binding on the bucket grants access to cache entries and nothing else. Enable uniform bucket-level access, for example with gcloud storage buckets create gs://your-nx-cache --location=us-central1 --uniform-bucket-level-access, so per-object ACLs cannot quietly widen access later. Put the bucket in the region where your runners execute. Same-region reads are faster, and you avoid inter-region egress charges on every cache hit.

Create a service account for CI and grant it roles on the bucket, not the project. A project-level roles/storage.objectAdmin binding would reach every bucket in the project. Bind with gcloud storage buckets add-iam-policy-binding gs://your-nx-cache and pass --member=serviceAccount:... and --role=.... Use roles/storage.objectViewer for read-only pipelines. Use roles/storage.objectCreator for writers: it can add objects but cannot overwrite or delete existing ones, which limits the damage a compromised job can do.

Next, create a workload identity pool and an OIDC provider for your CI system (gcloud iam workload-identity-pools create and gcloud iam workload-identity-pools providers create-oidc). Map token claims with --attribute-mapping and restrict them with --attribute-condition. For GitHub Actions, a condition such as assertion.repository == 'your-org/your-repo' && assertion.ref == 'refs/heads/main' limits which tokens are accepted. Finally, grant roles/iam.workloadIdentityUser on the service account to the matching principalSet:// member. CI exchanges its token for short-lived credentials, so there are no JSON keys to leak or rotate.

These commands are illustrative. Check exact flags and claim names against current Google Cloud documentation.

Copy#!/usr/bin/env bash
# Illustrative only: verify flags and claim names against current Google Cloud docs.
set -euo pipefail
PROJECT_ID="my-ci-project"; PROJECT_NUMBER="123456789012"
BUCKET="gs://acme-nx-cache"; POOL="ci-pool"; PROVIDER="github"
SA="nx-cache-ci@${PROJECT_ID}.iam.gserviceaccount.com"
ISSUER_URI="<your CI system's OIDC issuer URL>"

# Dedicated cache bucket in the runners' region; uniform access disables per-object ACLs.
gcloud storage buckets create "$BUCKET" --project="$PROJECT_ID" \
  --location=us-central1 --uniform-bucket-level-access

# CI identity. Roles are bound on the bucket, never on the project.
gcloud iam service-accounts create nx-cache-ci --project="$PROJECT_ID" --display-name="Nx cache CI"
# Writers need to read hits (objectViewer) and add new entries (objectCreator, no overwrite/delete).
# Read-only pipelines should get a separate service account bound to objectViewer only.
for ROLE in roles/storage.objectViewer roles/storage.objectCreator; do
  gcloud storage buckets add-iam-policy-binding "$BUCKET" \
    --member="serviceAccount:${SA}" --role="$ROLE"
done

# Pool + OIDC provider; the condition rejects tokens from other repos or branches.
gcloud iam workload-identity-pools create "$POOL" --project="$PROJECT_ID" --location=global
gcloud iam workload-identity-pools providers create-oidc "$PROVIDER" \
  --project="$PROJECT_ID" --location=global --workload-identity-pool="$POOL" \
  --issuer-uri="$ISSUER_URI" \
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.ref=assertion.ref" \
  --attribute-condition="assertion.repository == 'acme/monorepo' && assertion.ref == 'refs/heads/main'"

# Let only identities from that repository impersonate the service account (no JSON keys).
gcloud iam service-accounts add-iam-policy-binding "$SA" --project="$PROJECT_ID" \
  --role=roles/iam.workloadIdentityUser \
  --member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL}/attribute.repository/acme/monorepo"

Illustrative gcloud commands that create a uniform-access cache bucket and bind a CI service account to object-level roles on that bucket only. They then set up a Workload Identity Federation pool and provider whose attribute condition limits impersonation to one repository's main branch.

How do you set up an Nx cache on Azure Blob Storage with federated credentials?

Create a dedicated storage account and a single container for cache artifacts. Disable anonymous blob access at the account level, not only on the container, so a later misconfiguration cannot expose build outputs publicly. Set allow storage account key access to off if your tooling supports Entra ID authentication, because that removes the most common leak path entirely.

Account keys grant full control over every container in the account and do not expire. Long-lived SAS tokens are scoped more narrowly but are still bearer secrets: anyone who copies one from a CI log or environment dump can use it until it expires or the key that signed it is rotated. Federated credentials avoid both problems, because CI never holds a reusable secret.

For hosted runners, create a user-assigned managed identity or an app registration and add a federated credential. That credential trusts your CI provider's OIDC issuer and matches a specific subject, for example repo:<org>/<repo>:ref:refs/heads/main. Pinning the subject to the main branch means pull request jobs cannot obtain the write identity. Self-hosted runners on Azure VMs can use the VM's managed identity directly.

Assign Storage Blob Data Contributor to that identity at the container scope, not the account or resource group. An illustrative scope is /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account>/blobServices/default/containers/nx-cache, passed to az role assignment create along with --assignee and --role. Give developers Storage Blob Data Reader on the same container through an Entra ID group, so local builds can pull artifacts but cannot write them.

Treat these commands as illustrative. Check the exact flags, subject formats and role names against current Azure documentation before you use them.

Copy# Illustrative only: verify flags, subject formats and role names against current Azure docs.
RG=rg-build-cache
ACCOUNT=stnxcache001            # must be globally unique
SCOPE="/subscriptions/<sub-id>/resourceGroups/$RG/providers/Microsoft.Storage/storageAccounts/$ACCOUNT/blobServices/default/containers/nx-cache"

# Account-level guardrails: no anonymous blob access, no shared-key (account key / key-signed SAS) auth.
az storage account create --name "$ACCOUNT" --resource-group "$RG" --location westeurope \
  --sku Standard_LRS --kind StorageV2 \
  --allow-blob-public-access false --allow-shared-key-access false --min-tls-version TLS1_2

# Create the container with Entra ID auth, since shared keys are disabled.
az storage container create --name nx-cache --account-name "$ACCOUNT" --auth-mode login

# Identity that CI will act as. It holds no secret; CI exchanges its OIDC token for it.
az identity create --name id-nx-cache-writer --resource-group "$RG"

# Trust only main-branch jobs from one repo. PR jobs present a different subject and are rejected.
az identity federated-credential create --name ci-main \
  --identity-name id-nx-cache-writer --resource-group "$RG" \
  --issuer "<ci-provider-oidc-issuer-url>" \
  --subject "repo:<org>/<repo>:ref:refs/heads/main" \
  --audiences "api://AzureADTokenExchange"

# Write access for CI, scoped to the container rather than the account or resource group.
WRITER_ID=$(az identity show --name id-nx-cache-writer --resource-group "$RG" --query principalId -o tsv)
az role assignment create --assignee-object-id "$WRITER_ID" --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" --scope "$SCOPE"

# Read-only access for developers through an Entra ID group, so local builds pull but never push.
READERS_ID=$(az ad group show --group nx-cache-readers --query id -o tsv)
az role assignment create --assignee-object-id "$READERS_ID" --assignee-principal-type Group \
  --role "Storage Blob Data Reader" --scope "$SCOPE"

Illustrative Azure CLI sequence that locks down the storage account, creates a user-assigned identity with a main-branch-only federated credential, and grants container-scoped Contributor to CI and Reader to a developer group.

How should you scope read and write access to prevent cache poisoning?

The cache key is a hash of task inputs, but nothing in that hash proves who produced the artifact stored under it. Anyone with write access can upload a tampered build under a hash that other machines will compute. Those machines then restore it without running the task, and if your deploy pipeline restores from the same cache, the tampered output ships to production. Write access is therefore the trust boundary. Developer laptops and pull requests from forks get read-only identities. Only CI running on protected branches can assume the writer identity, and the OIDC trust condition must match only those branch refs.

Encryption at rest does not help here. It protects stored data from someone who reads the underlying storage directly. An attacker holding a valid write credential uploads through the normal API, and the provider encrypts the poisoned artifact like any other object.

Reading carries a separate risk. Cached outputs and terminal logs are replayed to every reader. A task that bakes an API key into a bundle or prints a token to stdout will hand that secret to every developer who can read the bucket. Disable caching for any task whose outputs depend on secrets or deploy-specific environment, rather than relying on bucket permissions to contain the leak.

S3 vs GCS vs Azure Blob: which remote cache backend should you choose?

None of the three is generally faster for this workload. Nx cache artifacts are compressed archives fetched once per task. Latency depends on how close the bucket is to your runners, not on which provider stores the bytes.

Choose the cloud that your CI runners and IAM already trust. If your runners live in AWS and the cache lives in GCS, you have to maintain a second federation trust and a second set of role bindings. You also pay egress on every cache hit, and cache hits are the traffic you want to maximize.

If a bucket is the wrong fit, there are three alternatives:

• A managed remote cache service. There is no storage or IAM to run yourself, but your artifacts leave your account.
• A self-run HTTP cache server. You get full control, but you now own its uptime, authentication and scaling.
• Local caching only. This option needs no setup. It is adequate for small teams, but it gives no cache reuse across CI runs.

How much does a self-hosted Nx cache cost, and how much engineering time does it save?

Each provider can expire cache objects after N days without custom tooling. In S3, add a lifecycle rule with an expiration action scoped to the cache prefix. In GCS, use Object Lifecycle Management with a Delete action and an age condition. In Azure Blob, add a lifecycle management policy that deletes base blobs when daysAfterModificationGreaterThan is reached. All three count age from when the object was written, not when it was last read. A heavily used artifact therefore expires on schedule and is rebuilt on the next miss.

Pick N to match your branching model. A 7-day window keeps the bill small, but a release branch that gets patched once a month will rebuild from scratch on every patch. A 90-day window keeps those hits. It also keeps stale artifacts around, along with any artifact that accidentally captured a secret.

Illustrative estimate (assumed numbers, not a benchmark). These inputs are assumptions, not quoted provider prices:

• Artifact size: 20 MB
• Tasks per pipeline: 40
• Pipelines per day: 200
• Hit rate: 50%
• Retention: 14 days
• Storage price: $0.02/GB-month
• Internet egress: $0.09/GB

At these inputs, egress is about ten times the storage cost. Where your runners sit relative to the bucket matters more than the headline storage price per GB.

How do you verify your Nx remote cache is working and fails safely?

Run these checks before you rely on the cache, and again after any Nx or adapter upgrade.

Prove a cross-machine hit

1. On machine A (a CI runner with write access), run nx run app:build.
2. On machine B, starting from a fresh clone with an empty local cache, run the same command at the same commit.
3. B's output should report that the result came from the remote cache.

A hit on the same machine only proves the local cache works.

Find unstable hashes

If B misses, the inputs differ. Common causes:

• Build steps that embed timestamps.
• Absolute paths, such as /home/alice/ versus /runner/work/.
• Environment variables that are declared as task inputs but differ per machine.

Run nx run app:build --skip-nx-cache twice on one machine and diff the outputs. If the outputs differ, the build is nondeterministic. If they match but hashes still differ across machines, compare the inputs Nx records for the task on each machine.

Confirm read-only clients fail cleanly

Use a developer credential to run a task that misses. The upload should be rejected, Nx should log a warning, and the task should still succeed. If the build fails, the adapter is treating write errors as fatal. Fix that before you roll it out.

Pin and test upgrades

• Pin the adapter to an exact version in your lockfile.
• Upgrade Nx and the adapter together in one branch, and rerun the two-machine test there.

Cache formats and plugin interfaces can change between releases.

Keep a fallback switch

Use an environment variable that removes the adapter from the configuration. When the bucket or identity provider is down, builds should run uncached and slower, not fail. Test that path on purpose.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx self-hosted remote cache, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.