Nx Self-Hosted Remote Cache on S3 and GCS Without Nx Cloud

How Nx remote caching works and what a self-hosted cache is

Before running a task, Nx computes a hash from everything it has been told can affect the result. That includes the source files matched by the task's inputs (including files from projects it depends on), the task's own configuration and command, declared environment variables, runtime inputs such as the output of a command that prints a tool version, and the resolved versions of external dependencies. If that hash matches an earlier run, Nx skips the task. It replays the stored terminal output and restores the files listed in the task's outputs.

A remote cache is a content-addressed store shared across machines. The key is the hash, and the value is an archive of the terminal output and artifacts. Nx never checks whether a cached artifact is still "correct". It only checks whether the key matches. This has two consequences:

• Misses come from any input change. Edit one file in scope, bump a dependency, or change a declared variable, and the hash changes. If inputs are too broad, for example a whole-repo glob, unrelated edits invalidate the cache constantly.
• Stale hits come from incomplete inputs. Suppose a build reads API_URL but the variable isn't declared as an input. Two builds with different values produce the same hash, and the second build silently receives the first build's artifact.

A self-hosted cache replaces the managed service with an S3 or GCS bucket that your team owns. Treat that bucket as untrusted shared infrastructure. Anyone who can write to it can store an artifact under a valid hash. Nx will then restore that artifact onto developer laptops and into CI without rebuilding it. That makes write access the real security boundary, a point the later sections build on.

Copy+-----------------------------+      +--------------+
| Task inputs                 |      | Nx computes  |
|  - files matched by inputs  |----->| task hash    |
|  - dependency project files |      | (cache key)  |
|  - task config + command    |      +------+-------+
|  - declared env vars        |             |
|  - runtime cmd output       |             | lookup by key only
|  - external dep versions    |             v
+-----------------------------+   +-----------------------+
                                  | Self-hosted bucket    |
 Undeclared input (API_URL)       | S3 or GCS, team-owned |
 is not hashed -> stale hit       | untrusted, shared     |
                                  | key:   task hash      |
                                  | value: archive of     |
                                  |   terminal output +   |
                                  |   outputs artifacts   |
                                  +---+---------------+---+
                                      |               ^
                                  hit |               | miss: run task,
                                      v               | then upload
                          +-----------------------+ +-+--------------------+
                          | replay terminal log   | | any writer: CI,    |
                          | restore outputs files | | laptops, or anyone |
                          | no rebuild, no        | | with write access  |
                          | correctness check     | | = trust boundary   |
                          +-----------------------+ +--------------------+
                            on laptops and in CI

Nx hashes declared inputs into a key and looks it up in a team-owned S3 or GCS bucket. A hit restores the stored output without rebuilding and a miss runs the task and uploads the result, so undeclared inputs and write access both decide which artifacts get trusted.

Who should self-host, when it fits, and when to avoid it

Self-hosting fits a platform team that already runs AWS or GCP identity and access management as routine work. If you already provision buckets with infrastructure-as-code, issue short-lived credentials to CI through workload identity, and review access policies, an Nx cache bucket is one more resource in a process you already have. You get shared build artifacts across CI and developer machines without adding a managed service, a new vendor relationship, or a second place where build outputs live.

When to avoid it

Skip self-hosting if nobody can own the bucket after setup. Someone has to maintain IAM policies, encryption settings, lifecycle rules that expire old artifacts, and an incident response plan for the day the cache misbehaves. Without that owner, the bucket accumulates stale data and broad permissions.

Location matters too. If your runners or developers sit in a different region or cloud from the bucket, every cache hit pays cross-region or internet egress, and that cost can exceed the compute you save. Artifact size has a similar limit. When an output is large enough that downloading it takes longer than rebuilding it, a cache hit slows the build down.

A storage bucket only stores artifacts. If you need distributed task execution or analytics on hit rates and task timings, you will have to build those yourself.

The main risk is cache poisoning. Anyone with write access can plant an artifact that later runs in CI or ships to production. Treat write credentials with the same care as deploy credentials.

Choosing an integration path: official storage plugins or the remote cache HTTP API

That risk also shapes how you connect Nx to storage. There are two self-hosted paths, and they differ mainly in where credentials live and who maintains the code between Nx and the bucket.

The official storage plugins let the Nx CLI read and write S3 or GCS directly, using credentials from the environment. Setup is mostly configuration. The cost is that every machine that writes to the cache holds cloud credentials, and authorization is limited to what bucket IAM can express.

The remote cache HTTP API is a small documented contract: retrieve an artifact by hash, upload an artifact for a hash, authenticate with a bearer token. You run a service that implements it in front of whatever storage you choose. Clients only hold a token, and the server decides who can read and who can write.

Nx has changed package names, licensing terms, and configuration keys for these features across releases. Check the documentation for your Nx version before you commit to either path. If neither path fits, the alternatives are managed Nx Cloud or local-only caching.

Designing the S3 or GCS bucket and scoping credentials

Give each repository, or each group of repositories that share a trust boundary, its own bucket. A shared bucket lets one repo's CI overwrite another repo's artifacts, and that cross-repo write path is hard to audit afterwards. On S3, turn on Block Public Access at the bucket level and use default server-side encryption (SSE-S3 or SSE-KMS). On GCS, enforce public access prevention and uniform bucket-level access. Add a lifecycle rule that deletes objects older than a fixed age, for example 14 or 30 days. The cache is disposable, and without expiry it grows with every commit you build.

Read-write versus read-only

Don't store access keys in CI. On AWS, use OIDC federation and restrict the role's trust policy with a condition on the sub claim, so that only your repository's protected branch can assume the write role. On GCP, use Workload Identity Federation with an attribute condition that checks both the repository and the ref.

Most incidents with this setup come from getting the fork pull request split wrong. A fork can change build scripts. If its job can write to the cache, it can upload a poisoned artifact under a valid hash, and your trusted builds will then restore that artifact as if it were real. Workflows that run fork code with base-repository credentials, such as pull_request_target checkouts of the PR head, open exactly this path.

Copy# Placeholders: replace with your project, repo, and group.
PROJECT_ID=example-project
PROJECT_NUMBER=123456789012
BUCKET=acme-web-nx-cache
POOL=github-ci
PROVIDER=github-main

# Dedicated bucket for one repo. Uniform access and public access prevention are on.
# Objects are encrypted at rest by default. Add --default-encryption-key to use CMEK.
gcloud storage buckets create gs://$BUCKET --project=$PROJECT_ID \
  --location=us-central1 --uniform-bucket-level-access --public-access-prevention

# The cache is disposable, so delete objects 30 days after creation.
echo '{"rule":[{"action":{"type":"Delete"},"condition":{"age":30}}]}' > lifecycle.json
gcloud storage buckets update gs://$BUCKET --lifecycle-file=lifecycle.json

# Workload Identity Federation replaces long-lived keys. The provider only accepts
# tokens from this repo's main branch. Fork PRs carry refs/pull/... and are rejected.
gcloud iam workload-identity-pools create $POOL --project=$PROJECT_ID --location=global
gcloud iam workload-identity-pools providers create-oidc $PROVIDER \
  --project=$PROJECT_ID --location=global --workload-identity-pool=$POOL \
  --issuer-uri="" \
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.ref=assertion.ref" \
  --attribute-condition="assertion.repository=='acme/web' && assertion.ref=='refs/heads/main'"

# Protected-branch CI gets read-write access to cache objects.
gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
  --member="principalSet://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$POOL/attribute.repository/acme/web" \
  --role=roles/storage.objectUser

# Developers can read the cache but cannot upload artifacts.
gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
  --member="group:web-devs@example.com" --role=roles/storage.objectViewer

# Fork PRs get no binding at all. Do not add one.

These gcloud commands create a private GCS cache bucket with a 30-day expiry rule and federate GitHub Actions through Workload Identity Federation so that only main-branch CI can write. Developers get read-only access, and fork pull requests get no credentials.

Wiring the cache into CI and making hashes trustworthy

…should get read-only access at most, and ideally no cache access at all.

Short-lived credentials

Don't store long-lived access keys in CI secrets. Configure your CI provider's OIDC identity to assume an AWS role or impersonate a GCP service account through workload identity federation. Scope the trust policy to specific repositories and branches, so only protected-branch builds receive write permissions and pull request jobs receive a read-only role. The cache plugin's SDK then picks up the federated credentials from the standard environment. Keep the cache section of nx.json limited to non-secret settings such as the bucket, region and key prefix.

Confirm the round trip

Run the target once with an empty cache. Delete every output directory, then run it again on a fresh runner. Nx labels each restored task in its output as coming from the local or remote cache, so check that every task reports a remote hit. Then diff the restored files against the files from the cold build. A missing file means an output was never declared, so list each target's outputs explicitly rather than relying on defaults.

Make the hash describe compatibility

Nx can only distinguish artifacts by what goes into the hash. Add any environment variable that changes build output as an env input. Add toolchain versions and platform details, such as the compiler version, node --version and OS/architecture, as runtime inputs. Without these, an artifact built on Linux x64 can be served to a macOS arm64 consumer.

Keep machine-specific values out of the hash, such as absolute paths, hostnames and timestamps. Each of these makes every hash unique, which drops your hit rate to zero without making the cache any safer.

Copy// nx.json: hash inputs and outputs only. Bucket and region settings for the cache plugin
// also live in this file, but credentials never do.
{
  "namedInputs": {
    // Source files, minus test files that never affect build output
    "production": ["default", "!{projectRoot}/**/*.spec.ts"],
    // Toolchain and platform facts that decide whether an artifact is compatible
    "toolchain": [
      { "runtime": "node --version" },
      { "runtime": "node -p \"process.platform + '-' + process.arch\"" },
      { "runtime": "npx tsc --version" },
      // Only env vars that change build output; no HOSTNAME, absolute paths or timestamps
      { "env": "NODE_ENV" },
      { "env": "API_BASE_URL" }
    ]
  },
  "targetDefaults": {
    "build": {
      "cache": true,
      "dependsOn": ["^build"],
      "inputs": ["production", "^production", "toolchain"],
      // Declare every directory the build writes, so a remote hit restores all of them
      "outputs": ["{projectRoot}/dist", "{projectRoot}/generated"]
    }
  }
}

An nx.json that lists build outputs explicitly and adds the Node version, OS/architecture, compiler version and output-affecting environment variables to the hash, while leaving machine-specific values out.

Developer access, failure fallback, observability and purging

Read-only access for developers

Developers should read from the cache without holding long-lived keys. Map their existing cloud login, such as AWS IAM Identity Center sessions or gcloud auth application-default login, to a role that can only list and get objects under the cache prefix. Only CI writes. This means a compromised laptop cannot plant an artifact that every pipeline will then trust.

Falling back to local-only caching

Short-lived credentials expire mid-day, and an expired session is a common cause of confusing errors. The failure often looks like an access-denied or signature error from the storage client, not an obvious "please log in again." Decide what should happen in that case and test it on purpose. Unset the credentials, then run a build with networking blocked. The build should finish using the local cache, and the output should show a clear warning. It should not hang on retries or fail the task. If your plugin or HTTP server treats cache errors as fatal, wrap the invocation or adjust its configuration so it doesn't. A cache that blocks work when it is unavailable will get disabled by frustrated developers.

Observability

Turn on server access logs (S3) or Cloud Audit data-access logs (GCS) and send them to a separate bucket. Use them to count GET versus PUT requests per prefix. Use storage metrics to track object count and total bytes over time. A falling GET-to-PUT ratio usually means hashes have become unstable. Steady byte growth without a matching rise in hits means your lifecycle rules are too lenient.

Purging a bad artifact

If you can identify the hash of the suspect artifact, delete that object. If you can't trace the problem to specific hashes, change the cache prefix in configuration and let the old prefix expire under lifecycle rules. Changing the prefix forces a cold cache, but it is atomic and easy to undo.

Copy#!/usr/bin/env bash
# Purge one suspect Nx cache artifact by task hash. Run from an admin/CI identity:
# the developer read-only role cannot delete objects.
set -euo pipefail

HASH="${1:?usage: purge-nx-artifact.sh <task-hash>}"
S3_PREFIX="s3://example-nx-cache/nx-cache/v1"
GCS_PREFIX="gs://example-nx-cache/nx-cache/v1"

# Fail fast with a readable message. An expired session otherwise shows up
# as an access-denied or signature error further down.
aws sts get-caller-identity >/dev/null || { echo "AWS session missing or expired" >&2; exit 1; }

# 1. List first. Do not assume the key layout; confirm which objects match.
aws s3 ls "${S3_PREFIX}/" --recursive | grep -- "${HASH}" || { echo "no S3 objects for ${HASH}"; }

# 2. Delete only the keys that contain this hash.
aws s3 rm "${S3_PREFIX}/" --recursive --exclude "*" --include "*${HASH}*"

# Same procedure on GCS. The wildcard matches objects whose name contains the hash.
if gcloud auth application-default print-access-token >/dev/null 2>&1; then
  gcloud storage ls "${GCS_PREFIX}/*${HASH}*" || echo "no GCS objects for ${HASH}"
  gcloud storage rm "${GCS_PREFIX}/*${HASH}*" || true
fi

# 3. Verify the artifact is gone. The next CI run rebuilds the task and
#    uploads a fresh artifact under the same hash.
aws s3 ls "${S3_PREFIX}/" --recursive | grep -- "${HASH}" && { echo "still present" >&2; exit 1; }
echo "purged ${HASH}"

# If no single hash can be identified, do not bulk-delete. Point the cache
# config at a new prefix (for example nx-cache/v2), roll it out to CI and
# developers, and let lifecycle rules expire nx-cache/v1.

A purge script that checks the session is valid, lists the objects matching a suspect task hash, deletes them from S3 or GCS and confirms they are gone, with a new prefix as the fallback when no hash can be identified.

Impact on cost and engineering time

The cache bill is rarely the deciding number. Ownership time and wrong hits usually are.

Illustrative estimate (assumed numbers, not a benchmark). These are the inputs:

• 400 CI runs per day
• 60% cache hit rate
• 50 MB of artifacts written per miss
• 14-day retention
• 6 CI minutes saved per hit
• $0.008 per CI minute
• $0.023 per GB-month of storage
• $0.02 per GB of cross-region egress
• $100 per engineer-hour
• 4 hours per month to own IAM policy, alerts and incidents

Request charges are left out of the table. They scale with task count rather than artifact size, so multiply your tasks per run by your provider's per-request price.

At these inputs, CI savings alone don't pay for ownership. Developer wait time is what tips the balance. If 30 engineers each avoid 10 minutes of local rebuilds per day, that is roughly 100 hours a month. The figure assumes about 20 working days; adjust it for your team.

Silently wrong hits are the cost that grows fastest. A local cache that produces a bad output affects one person. A poisoned remote entry reaches every CI job and laptop that computes the same hash. An assumed two engineers spending six hours each on such an incident costs $1,200. That single incident outweighs a year of storage and egress at these inputs. This is why time spent on hash inputs and on keeping write credentials restricted to CI is the part of the budget worth protecting.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx self-hosted remote cache, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.