What is an Nx self-hosted remote cache and how does it work?

Nx computes a hash for every task from its inputs: the source files the task depends on, the resolved versions of external dependencies, the task's configuration, and any declared environment variables or runtime commands. When a task finishes, Nx packages the terminal output and the task's declared outputs (such as dist/ folders or coverage reports) into an archive stored under that hash. The next time any machine computes the same hash, Nx downloads the archive, replays the logs, and restores the files instead of running the task.

A remote cache moves that store off the local disk so CI runners and developers share results. All the cognitive work happens on the client. The backend never interprets a hash or inspects an archive. It only needs to answer two requests: get the blob for this key, and put a blob under this key. That is why a plain object store is enough, and why no database or custom service is required.

There are two ways to connect Nx to a backend:

  • Official adapters. Nx publishes cache plugins for S3 and GCS that call the storage APIs directly using your cloud credentials.
  • Custom HTTP cache servers. Nx can talk to any server that implements its documented remote-cache HTTP API. You can put that server in front of any storage you choose.

Package names, configuration keys, and licensing terms have changed between Nx releases. Confirm all three in the Nx docs for the version you run.

  CI runners (trusted)          Developer laptops
        |                              |
  compute task hash             compute task hash
        |                              |
        +--------> key = hash <--------+
        |                              |
   GET + PUT                        GET only
   (write path)                   (read path)
        |                              |
        v                              v
  +-------------------------------------------+
  |  Object bucket, same region as CI runners |
  +-------------------------------------------+
+------------------------+        +------------------------+
|  CI runners (trusted)  |        |   Developer laptops    |
|  compute task hash     |        |   compute task hash    |
+-----------+------------+        +-----------+------------+
            |                                 |
            v                                 v
+-----------------------------------------------------------+
|  Nx client: hash(inputs) -> cache key                     |
|  inputs = sources, deps, config, env, runtime commands    |
+-----------+---------------------------------+-------------+
            |                                 |
   GET + PUT (write path)            GET only (read path)
   read-write credentials            read-only credentials
            |                                 |
            v                                 v
+-----------------------------------------------------------+
|  Same-region bucket (S3 or GCS)                           |
|                                                           |
|  key = <hash>  -->  archive                               |
|                     (terminal output + declared outputs)  |
|                                                           |
|  Reached via official S3/GCS adapter, or via a custom     |
|  HTTP cache server placed in front of the storage         |
+-----------------------------------------------------------+

Clients hash task inputs into a cache key; trusted CI runners read and write archives in a same-region bucket, while developer laptops only read.

Who should self-host the Nx cache, and when should you avoid it?

The best fit is a team whose CI already runs on AWS or Google Cloud, with the bucket in the same region as the runners. You also need platform engineers who can own the IAM policies, rotate credentials, and watch the storage and request bill. Your tasks should also be reasonably hermetic, so that the hash Nx computes captures everything that changes the output.

The payoff comes from sharing. A build or test target that ran once on any CI job or any laptop becomes a download for everyone else. The saving grows with the number of engineers and pipelines that touch the same projects. Against that, count the ongoing work: bucket policies, lifecycle rules, credential distribution for developers, and debugging the occasional bad artifact. If the time saved each week doesn't clearly exceed the hours someone spends maintaining the cache, the cache is a net loss.

When to avoid it

  • Non-hermetic tasks. A target might read an undeclared environment variable, depend on a globally installed tool version, or consume a generated file that isn't listed as an input. Locally, a stale hit like that confuses one person. A remote cache serves the same wrong output to the whole team and to CI, and it looks like a legitimate result.
  • Runners outside the bucket's cloud. Runners in another cloud or in an on-premises network pay egress on every cache hit. Those transfer costs can erase the savings.
  • No incident capacity. A poisoned or leaked artifact needs someone who can purge entries, revoke credentials, and audit access quickly. If nobody owns that, don't self-host.

S3 vs GCS cache: how do the backends compare?

Both backends store each cache entry as an opaque object keyed by task hash, so read and write behavior is nearly identical. The differences are in identity, policy and operations.

Concern S3 GCS
CI identity IAM role assumed through an OIDC identity provider Service account impersonated through Workload Identity Federation
Permission scoping Bucket policy or role policy scoped to a key prefix Bucket-level objectViewer / objectCreator roles; prefix limits need IAM Conditions
Expiry Lifecycle rules by prefix, tag or age Lifecycle rules by age, prefix or storage class
Encryption SSE-S3 by default; SSE-KMS for customer-managed keys Google-managed by default; CMEK through Cloud KMS
Audit trail CloudTrail data events or server access logs, both opt-in Data Access audit logs, opt-in for Cloud Storage
Laptop access SSO profile plus a read-only role gcloud application-default credentials plus viewer grant

On S3, prefix scoping is the native pattern, so separating per-repo or per-branch writers is straightforward. On GCS, the natural boundary is the bucket. Many teams use one bucket per trust level instead of writing conditions.

For laptop access, both clouds need a developer login flow and a read-only grant. The friction usually comes from the organization's SSO setup more than from the storage service.

For blob get and put, choose based on where your CI runners run, which cloud already federates your CI identity, and which path avoids cross-cloud egress charges.

The choice is sticky. Moving later means rebuilding roles, federation trust, key policies and logging, not just renaming a bucket.

How do you set up read/write separation on S3 and GCS?

Read/write separation is the most important design decision here, and it follows from how cache poisoning works. Nx trusts any artifact stored under a matching hash. Any principal that can write to the cache can plant output that later builds, including release builds on main, will restore without rebuilding. Only trusted CI on main and protected branches should get write access. Pull-request jobs, fork PRs and laptops should get read-only access or no access.

AWS

Create two IAM roles that CI assumes through OIDC federation. The trust policy on the writer role uses a StringEquals condition on token.actions.githubusercontent.com:sub (or your CI provider's equivalent claim) that pins it to refs/heads/main and named protected branches. The reader role can accept a wider StringLike match. The bucket policy grants s3:PutObject on arn:aws:s3:::<bucket>/nx-cache/* only to the writer role, and s3:GetObject on that prefix to both roles. Fork PRs should not receive OIDC credentials at all.

GCP

Bind two service accounts through Workload Identity Federation. Use an attribute condition on assertion.ref so that only protected refs can impersonate the writer. Grant bucket roles with gcloud storage buckets add-iam-policy-binding:

  • roles/storage.objectViewer for the reader.
  • roles/storage.objectCreator for the writer. This role cannot overwrite existing objects, which is a useful property for a content-addressed cache.

Laptops

Developers should authenticate through SSO, such as AWS IAM Identity Center or gcloud auth login, and receive short-lived, read-only credentials. Avoid shared static keys. They leak into dotfiles and old CI variables, they cannot be traced to a person, and rotating one breaks everyone who uses it.

#!/usr/bin/env bash
# Placeholder values: replace with your own project, pool and repo.
set -euo pipefail
PROJECT_NUMBER=123456789012
POOL=ci-pool
BUCKET=gs://acme-nx-cache
READER=nx-cache-reader@acme-ci.iam.gserviceaccount.com
WRITER=nx-cache-writer@acme-ci.iam.gserviceaccount.com
POOL_PATH="projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL}"

# The OIDC provider maps token claims to attributes. Its condition rejects
# tokens from any repository except ours.
gcloud iam workload-identity-pools providers create-oidc github \
  --location=global --workload-identity-pool="${POOL}" \
  --issuer-uri="" \
  --attribute-mapping="google.subject=assertion.sub,attribute.ref=assertion.ref,attribute.repository=assertion.repository" \
  --attribute-condition="assertion.repository == 'acme/monorepo'"

# Any CI job from the repo can impersonate the reader.
gcloud iam service-accounts add-iam-policy-binding "${READER}" \
  --role=roles/iam.workloadIdentityUser \
  --member="principalSet://iam.googleapis.com/${POOL_PATH}/attribute.repository/acme/monorepo"

# Only jobs running on refs/heads/main can impersonate the writer.
# Add one binding per additional protected branch.
gcloud iam service-accounts add-iam-policy-binding "${WRITER}" \
  --role=roles/iam.workloadIdentityUser \
  --member="principalSet://iam.googleapis.com/${POOL_PATH}/attribute.ref/refs/heads/main"

# Bucket roles. The reader can read only.
gcloud storage buckets add-iam-policy-binding "${BUCKET}" \
  --member="serviceAccount:${READER}" --role=roles/storage.objectViewer

# The writer also needs read access, because main builds restore from the cache.
# objectCreator can create objects but cannot overwrite or delete existing ones.
for ROLE in roles/storage.objectViewer roles/storage.objectCreator; do
  gcloud storage buckets add-iam-policy-binding "${BUCKET}" \
    --member="serviceAccount:${WRITER}" --role="${ROLE}"
done

Workload Identity Federation lets any CI job in the repository impersonate a read-only cache service account, while only jobs on refs/heads/main can impersonate the writer, which can add new cache objects but cannot overwrite existing ones.

How much does a self-hosted Nx remote cache cost?

Placement matters more than any other cost decision. Keep the bucket in the same cloud and region as your CI runners. A cache is read-heavy, and when runners pull artifacts across clouds or over the internet, egress is often the largest line on the bill. Same-region reads usually avoid that charge entirely. Request charges matter next, especially in workspaces with many small tasks, where every lookup costs a request even when the artifact is tiny.

Illustrative estimate (assumed numbers, not a benchmark). Assume 200 CI runs per day, 50 cacheable tasks per run, a 5 MB average artifact, a 60% hit rate and 30-day retention.

Component Formula Assumed daily volume
Lookups (GET/HEAD) runs × tasks 10,000 requests
Writes (PUT) runs × tasks × (1 − hit rate) 4,000 requests
Data read runs × tasks × hit rate × size 30 GB
Stored data PUT volume × size × retention days ~600 GB at steady state
Egress data read, if crossing region or cloud 0 GB same-region, 30 GB otherwise

To get a monthly cost, multiply each daily volume by 30. Then multiply by the current unit price on your provider's pricing page. Prices vary by region and storage class and change over time, so these figures are not quoted here. With these inputs, cross-region egress would add 900 GB per month, which is why placement comes first.

Set a budget alert on the project or account that holds the bucket. A misconfigured runner pool in another region can raise costs well before anyone looks at the invoice.

How should you handle retention, encryption and access logging?

Neither S3 nor GCS evicts least-recently-used entries. Lifecycle rules on both clouds expire objects by age since creation, not since last read. An entry that every pipeline hits still gets deleted on schedule, and the next run rebuilds and re-uploads it. Because Nx cache keys are content hashes, that costs one cold task, not a correctness problem.

Retention

On S3, add a lifecycle rule with an expiration action of N days, scoped to the cache prefix. On GCS, add a lifecycle rule with a Delete action and an age condition of N days. To choose N:

  • Branch lifetime: Set N longer than your typical long-lived branch, such as a release or maintenance line. If N is shorter, those branches lose their cache between infrequent builds.
  • Release cadence: Set N longer than the gap between releases. A hotfix to the last release should still hit cache.
  • Upper bound: Beyond those two limits, more days mostly store archives for hashes that will never be requested again.

Storage-class transitions rarely help. Infrequent-access and nearline classes have minimum storage durations and retrieval fees. Moving short-lived, frequently read cache archives into those classes usually raises cost.

Encryption

Both backends encrypt objects at rest by default with provider-managed keys. If policy requires control over key rotation or revocation, use customer-managed keys: SSE-KMS on S3, CMEK via Cloud KMS on GCS. Your CI identities then also need permission to use the key.

Access logging

Enable S3 server access logs or CloudTrail data events. On GCS, enable Data Access audit logs, which are off by default. These logs record which principal wrote each object, so you can trace a poisoned or unexpected cache entry back to the runner or credential that uploaded it.

#!/usr/bin/env bash
# Expire Nx cache archives after N days on both clouds.
# Pick N to exceed your longest-lived branch gap and your release cadence.
set -euo pipefail

RETENTION_DAYS=30            # N: illustrative value, tune per team
S3_BUCKET=example-nx-cache   # placeholder bucket names
GCS_BUCKET=example-nx-cache
PREFIX=nx-cache/             # scope rules to the cache prefix only

# S3: expire objects N days after creation and clean up abandoned multipart uploads.
cat > s3-lifecycle.json <<EOF
{
  "Rules": [{
    "ID": "expire-nx-cache",
    "Filter": { "Prefix": "${PREFIX}" },
    "Status": "Enabled",
    "Expiration": { "Days": ${RETENTION_DAYS} },
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 }
  }]
}
EOF
aws s3api put-bucket-lifecycle-configuration \
  --bucket "$S3_BUCKET" \
  --lifecycle-configuration file://s3-lifecycle.json

# GCS: Delete action with an age condition, limited to the same prefix.
# Age counts from object creation, not last read; there is no LRU eviction.
cat > gcs-lifecycle.json <<EOF
{
  "rule": [{
    "action": { "type": "Delete" },
    "condition": { "age": ${RETENTION_DAYS}, "matchesPrefix": ["${PREFIX}"] }
  }]
}
EOF
gcloud storage buckets update "gs://${GCS_BUCKET}" \
  --lifecycle-file=gcs-lifecycle.json

Applies equivalent age-based expiration rules, scoped to the Nx cache prefix, to an S3 bucket and a GCS bucket so that old cache archives are deleted after N days.

How do you roll out and operate the cache safely?

Roll out in stages

Start with CI as the only writer and developers as read-only consumers. CI runners have a controlled, reproducible environment. Developer machines do not. If a laptop with a stray environment variable or a local toolchain mismatch can write, it can publish an artifact that every other machine then trusts.

Run in this mode for a while and check hit rates in the CI logs. Nx reports which tasks were restored from cache, and a hit rate that stays low usually means your task inputs are too broad or include nondeterministic files. Next, run a cold-cache test: point a pipeline at an empty prefix and confirm it builds cleanly and repopulates the cache. This shows that nothing depends on stale entries. Grant read access to more developers and teams only after those checks pass.

Runbook for a suspected bad artifact

Write this runbook before you need it. It should cover three steps:

  • Bypass the cache for one run with Nx's skip-cache flag (currently --skip-nx-cache; confirm against the docs for your Nx version). This tells you quickly whether the cache is the cause.
  • Clear the affected prefix in the bucket. Your access logs help narrow down which entries to remove.
  • Force uncached runs on CI until the clean artifacts have been rebuilt and written back.

Track upstream

Self-hosted cache adapters, how they are configured and how they are licensed can all change between Nx releases. Assign someone to read the release notes before each upgrade.

#!/usr/bin/env bash
# Runbook: suspected bad artifact in the self-hosted Nx remote cache.
# Placeholder values: replace the bucket, prefix and target with your own.
set -euo pipefail

BUCKET="example-nx-cache"     # bucket name, no scheme
PREFIX="nx/main"              # prefix the affected pipeline writes to
TARGET="build"

# Step 1: bypass the cache for one run to check whether the cache is the cause.
# Confirm the flag name in the docs for your Nx version.
npx nx reset                   # also clear the local cache on this machine
npx nx run-many -t "$TARGET" --skip-nx-cache

# Step 2: clear the affected prefix. List and dry-run before deleting.
if [[ "${CLOUD:-aws}" == "aws" ]]; then
  aws s3 ls "s3://${BUCKET}/${PREFIX}/" | head
  aws s3 rm "s3://${BUCKET}/${PREFIX}/" --recursive --dryrun
  read -rp "Delete these objects? [y/N] " ok
  [[ "$ok" == "y" ]] && aws s3 rm "s3://${BUCKET}/${PREFIX}/" --recursive
else
  gcloud storage ls "gs://${BUCKET}/${PREFIX}/" | head
  read -rp "Delete these objects? [y/N] " ok
  [[ "$ok" == "y" ]] && gcloud storage rm --recursive "gs://${BUCKET}/${PREFIX}/"
fi

# Step 3: CI runs uncached until clean artifacts are rebuilt and written back.
# The CI pipeline is expected to check FORCE_UNCACHED, a variable you define,
# and add --skip-nx-cache when it is set. Unset it after the next clean run.
echo "Set FORCE_UNCACHED=1 in CI variables, then trigger a ${TARGET} run on main."

A runbook script for a suspected bad artifact that bypasses the cache for one run, clears the affected prefix in S3 or GCS after a dry-run listing, and reminds the operator to force uncached CI runs until clean artifacts are written back.

What are the alternatives to a self-hosted cache, and what do you give up versus Nx Cloud?

Self-hosting is one of four realistic setups. Each moves a different amount of operational work onto your team.

Option What you run What you get What you own
Nx Cloud Nothing beyond the workspace connection Remote cache, hosted dashboards, run analytics, managed distributed task execution Vendor relationship, data residency review
Official S3 or GCS adapter A bucket and IAM policies Remote cache only IAM, encryption, retention, cost monitoring, incident response
Custom HTTP cache server A service plus its storage Remote cache only, with full control over auth and storage Everything above, plus the service's uptime, patching and scaling
Local cache only Nothing Fast reruns on one machine or one CI job Cold CI builds on every fresh runner

As an Nx Cloud alternative, a bucket adapter replaces only the cache. You lose the run dashboards that show which tasks missed the cache and why. You lose the analytics that reveal slow or flaky targets. You also lose managed distribution of tasks across agents, so splitting work across CI machines becomes something you script yourself.

A custom HTTP server makes sense in a narrow set of cases. It fits when an official adapter cannot satisfy your authentication model or your storage location. Check that your Nx version documents the HTTP cache contract before you build against it.

Which option fits which team:

  • Small teams with short CI times: local cache only.
  • Teams on AWS or GCP with an existing IAM practice and strict data-location rules: the bucket adapters.
  • Teams that want distribution and visibility without building them: Nx Cloud.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx self-hosted remote cache, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.