Nx Remote Cache on S3: Self-Hosted Setup, IAM and Costs

How Nx remote caching works and why cache writes are trust-sensitive

Before running a task, Nx computes a hash from that task's inputs. These include the relevant source files, the hashes of the projects it depends on, the command and its flags, and any runtime values or environment variables declared as inputs. After the task runs, Nx stores the output files and the terminal log as an artifact keyed by that hash. When another machine computes the same hash, it downloads the artifact, restores the files and replays the log instead of running the task. On a large monorepo this skips repeated builds, tests and lint runs, which saves both CI minutes and the time engineers spend waiting.

The rule that drives the rest of this guide is that whoever can write to the cache decides what every other machine accepts as build output. Nx does not rebuild restored artifacts to check them. A matching hash is enough.

This creates two ways for bad artifacts to spread:

• Poisoning. A compromised runner or a leaked write credential uploads a tampered artifact under a valid hash. Every reader then restores it, including release jobs that ship it.
• False hits. A task reads something that is not declared as an input, such as an undeclared environment variable or a generated file. The hash then stays the same when the real inputs change. The first stale output gets stored and served to everyone with no attacker involved.

Who reads and who writes

All three principals reach the S3 bucket through the cache integration layer. Only trusted CI runners are allowed to write to it.

Copy+---------------------+  +---------------------+  +---------------------+
| CI runner           |  | CI runner           |  | Developer machine   |
| trusted branch      |  | pull request / fork |  | (local runs)        |
| (for example, main) |  |                     |  |                     |
+---------------------+  +---------------------+  +---------------------+
    |           ^                   ^                        ^
    | write     | read              | read only              | read only
    v           |                   |                        |
+-----------------------------------------------------------------------+
|  Nx cache integration layer                                           |
|  key = hash(task inputs)   artifact = outputs + terminal log          |
|  writes: trusted CI only     reads: all principals                    |
+-----------------------------------------------------------------------+
          |   ^
    PUT   |   |  GET
          v   |
+-----------------------------------------------------------------------+
|  S3 bucket                                                            |
|  objects keyed by task hash; restored without re-verification         |
+-----------------------------------------------------------------------+

All three principals read cached artifacts through the cache integration layer, but only trusted-branch CI runners can write. Every reader restores whatever is stored under a matching hash without checking it again.

Who should self-host on S3, when it fits, and when to avoid it

S3 self-hosting fits teams whose CI already runs on AWS, ideally in the same region as the bucket. That keeps transfer cheap and downloads fast. It also needs a platform team that can own the bucket, the IAM policies and the integration layer, and patch them when they break. The payoff is largest in monorepos with slow, deterministic tasks such as builds, type checks and long test suites. A cache hit on one of those saves minutes, and identical inputs reliably produce identical outputs.

Avoid it when runners sit outside AWS or in another region. Egress charges and round-trip latency can cancel out the time saved. Workloads made of many small, fast tasks are also a poor fit, because downloading and unpacking an artifact can take longer than rerunning the task. A raw bucket also gives you no hit-rate analytics, no distributed task execution and no per-user access management. If you need any of those, you would have to build them yourself.

Connecting Nx to S3: integration options and configuration

There are two practical ways to connect Nx to an S3 bucket. Both use the same bucket and IAM design described in the next sections. They differ in who maintains the code between Nx and S3.

Option 1: An Nx-supported S3 cache integration

Nx has offered packaged integrations for self-hosted caches, including one that targets S3. When you use one of these, Nx owns the client code, so changes to the cache protocol and Nx upgrades are handled for you. Package names, licensing terms and configuration keys have changed across Nx releases. Before you install anything, check the current Nx documentation for your installed Nx version. Don't copy a package name from an older blog post or from this article.

Option 2: A small HTTP service in front of S3

Nx documents a self-hosted remote cache interface. This is an HTTP API for storing and retrieving task artifacts by hash. You can write a small service that implements it, maps each hash to an S3 object key, and checks a bearer token on every request. This gives you full control over authentication, logging and object layout.

The cost of that control is maintenance. The server is a service you must patch, monitor and keep available. It can also break when the cache interface changes between Nx versions. Pin your Nx version, and test cache reads and writes as part of every Nx upgrade.

Configuration shape

With either option, the nx.json change has the same two parts:

• Mark targets as cacheable. On recent versions this usually means setting cache: true under targetDefaults for targets such as build, test and lint, along with accurate outputs paths.
• Point Nx at the remote cache. Provide the integration's settings, or your server's URL and token. Read the token from an environment variable rather than committing it.

The fragment below is a placeholder. Verify every key against the Nx docs for your version before you use it.

Copy// nx.json: PLACEHOLDER shape only. Verify every key against the Nx docs for your installed version.
// nx.json is parsed as JSON with comments. Remove the comments if your tooling expects strict JSON.
{
  "targetDefaults": {
    // Part 1: mark targets as cacheable and declare their outputs.
    // Nx stores and restores only the paths listed in "outputs".
    "build": {
      "cache": true,
      "dependsOn": ["^build"],
      "outputs": ["{projectRoot}/dist"]
    },
    "test": {
      "cache": true,
      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"]
    },
    "lint": {
      // Lint usually writes no files, so caching only replays its terminal output.
      "cache": true
    }
  },

  // Part 2: point Nx at the remote cache.
  // PLACEHOLDER: the real key name and fields depend on your Nx version and on the
  // option you chose (packaged S3 integration or your own HTTP server).
  // Look up the exact keys in the current Nx docs.
  "REMOTE_CACHE_SETTINGS_FROM_NX_DOCS": {
    // Option 1: the integration's own settings, such as bucket and region.
    "bucket": "example-nx-cache-bucket",
    "region": "us-east-1"
    // Option 2: your server's URL, for example
    // Never commit the access token here. Supply it from an environment variable
    // in CI, using the variable name your Nx version documents.
  }
}

A placeholder nx.json fragment that turns on caching for build, test and lint with explicit outputs and marks where the remote cache settings go; check the actual key names in the Nx docs for your version.

Creating the Nx cache bucket: isolation, encryption and lifecycle rules

Create a bucket that holds nothing but cache artifacts. Mixing the cache with release artifacts or backups makes it harder to write tight IAM policies and lifecycle rules, because every rule then has to work around unrelated data.

Bucket basics

• Region: Put the bucket in the same region as your CI runners. Cross-region reads add latency to every cache hit and incur inter-region transfer charges.
• Creation: Use aws s3api create-bucket. Outside us-east-1, pass a LocationConstraint that matches the region.
• Public access: Run aws s3api put-public-access-block with all four Block Public Access settings enabled. Recent AWS defaults may already do this, but setting it explicitly documents your intent.
• Encryption: Run aws s3api put-bucket-encryption to set default server-side encryption. SSE-S3 works. Choose SSE-KMS if you need key-level access auditing, and account for the extra KMS request costs.
• Prefixes: Give each repository or workspace its own key prefix, such as cache/<repo>/. IAM policies and lifecycle filters can then target one prefix without affecting the others.

Lifecycle rules

Apply rules with aws s3api put-bucket-lifecycle-configuration. The JSON needs two rules:

• An Expiration rule with Days set to your retention window.
• An AbortIncompleteMultipartUpload rule with DaysAfterInitiation set to a small value. Without it, interrupted CI jobs leave partial uploads that are billed but never visible as objects.

Leave versioning off. Nx can regenerate any artifact from source, so old versions only add storage cost.

Retention trade-off

S3 lifecycle expiration counts from when an object was created, not from when it was last read. A heavily reused artifact on a long-lived release branch expires on schedule like any other object. A short window keeps costs down but forces periodic rebuilds on those branches. Set the window to at least the typical gap between commits on your slowest-moving branch.

Treat the bucket like build logs

Cached terminal output and build artifacts can contain secrets, internal hostnames or customer data. Apply the same access controls you use for CI logs.

Copy#!/usr/bin/env bash
set -euo pipefail

# Placeholders: one bucket used only for Nx cache, in the same region as CI runners.
BUCKET="example-org-nx-cache"
REGION="eu-west-1"

# Outside us-east-1, LocationConstraint must match the region.
aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" \
  --create-bucket-configuration LocationConstraint="$REGION"

# Enable all four Block Public Access settings explicitly to document intent.
aws s3api put-public-access-block --bucket "$BUCKET" \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

# Default encryption with SSE-S3. Use "aws:kms" plus a key ID if you need key-level auditing.
aws s3api put-bucket-encryption --bucket "$BUCKET" \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

# Rule 1: expire cache objects under cache/ 14 days after creation (not after last read).
# Rule 2: abort interrupted multipart uploads so orphaned parts stop being billed.
cat > lifecycle.json <<'EOF'
{
  "Rules": [
    { "ID": "expire-nx-cache", "Status": "Enabled",
      "Filter": { "Prefix": "cache/" },
      "Expiration": { "Days": 14 } },
    { "ID": "abort-incomplete-mpu", "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } }
  ]
}
EOF

aws s3api put-bucket-lifecycle-configuration --bucket "$BUCKET" \
  --lifecycle-configuration file://lifecycle.json
# Versioning is left off on purpose: Nx can rebuild any artifact from source.

AWS CLI commands that create a dedicated Nx cache bucket with Block Public Access, default SSE-S3 encryption, and lifecycle rules that expire cache objects and abort incomplete multipart uploads.

IAM design and OIDC credentials for CI caching

Two principals cover most teams. Anything that writes to the cache can influence what every later build restores, so write access belongs only to builds whose inputs you already trust: CI jobs on protected branches after review.

ListBucket applies to the bucket ARN, while GetObject and PutObject apply to object ARNs. Write the policy statements against those two resource types separately.

Federate instead of storing keys

Configure your CI provider as an OIDC identity provider in IAM. Jobs then exchange a signed token for short-lived role credentials through sts:AssumeRoleWithWebIdentity. No access key sits in CI secrets where a leak would go unnoticed.

The trust policy is where you enforce "protected branches only." On GitHub Actions, add a condition that matches aud to sts.amazonaws.com and sub to a value like repo:ORG/REPO:ref:refs/heads/main. Claim names and formats differ between providers, so confirm them in your provider's OIDC documentation before relying on them.

Trade-offs

Fork PRs start cold, so their hit rates are lower and their builds run longer. Developers without write access never populate the cache from local builds. Neither cost justifies granting more access. A poisoned cache entry is harder to detect than a slow build.

Copy#!/usr/bin/env bash
# Create the cache writer role: assumable only by GitHub Actions OIDC tokens from main.
# Claim keys (aud, sub) and the sub format are GitHub-specific; check your provider's docs.
set -euo pipefail
ACCOUNT=123456789012; BUCKET=example-nx-cache; PREFIX=nx-cache
OIDC=token.actions.githubusercontent.com

cat > trust.json <<EOF
{"Version": "2012-10-17", "Statement": [{
  "Effect": "Allow",
  "Principal": {"Federated": "arn:aws:iam::${ACCOUNT}:oidc-provider/${OIDC}"},
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {"StringEquals": {
    "${OIDC}:aud": "sts.amazonaws.com",
    "${OIDC}:sub": "repo:example-org/example-repo:ref:refs/heads/main"
  }}
}]}
EOF

# ListBucket targets the bucket ARN; Get/Put target object ARNs under the prefix.
# The reader role uses the same policy without s3:PutObject.
cat > writer.json <<EOF
{"Version": "2012-10-17", "Statement": [
  {"Effect": "Allow", "Action": "s3:ListBucket",
   "Resource": "arn:aws:s3:::${BUCKET}",
   "Condition": {"StringLike": {"s3:prefix": ["${PREFIX}/*"]}}},
  {"Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"],
   "Resource": "arn:aws:s3:::${BUCKET}/${PREFIX}/*"}
]}
EOF

aws iam create-role --role-name nx-cache-writer \
  --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name nx-cache-writer \
  --policy-name nx-cache-rw --policy-document file://writer.json

Creates a cache writer role. Its trust policy accepts only GitHub Actions OIDC tokens from the main branch, and its identity policy splits bucket-level ListBucket from object-level GetObject and PutObject, scoped to the cache prefix.

Impact on cost and engineering time: an illustrative estimate

Every figure below is a variable. Plug in current S3 pricing for your region and your own CI numbers.

Illustrative estimate (assumed numbers, not a benchmark)

• 200 CI runs/day, 40 cacheable tasks per run, 20 MB average artifact
• 70% hit rate, 14-day lifecycle expiry
• 3-minute average task, about 10 seconds to download an artifact

That gives 8,000 lookups/day. The 2,400 misses upload 48 GB/day, so about 670 GB sits in the bucket at steady state. The 5,600 hits download about 112 GB/day, or roughly 3.4 TB/month.

Storage and requests stay modest because lifecycle expiry caps the bucket's size. Transfer scales with hits, so a better hit rate increases the egress bill. Hosted runners outside your AWS region pay internet or cross-region rates on every download.

Time savings: 5,600 hits × (3 min − 10 s) ≈ 15,900 CI minutes/day. Developer wait time drops by less than that. Only tasks on a pipeline's critical path shorten the wall-clock time, and parallel tasks overlap.

When rerunning is cheaper

Download an artifact only when both of these hold:

• size / bandwidth + request latency < task duration
• size × egress price < task minutes × runner price per minute

Fast tasks with large outputs often fail both tests. Exclude them from caching.

Verifying the setup and running the cache day to day

Verification checklist

1. Prove a cross-machine hit. On a clean CI runner, run npx nx reset to clear local state, then run a cacheable target such as npx nx build my-app. Repeat on a second clean runner with the same commit. The second run should label the task [remote cache]. A local hit proves nothing, because it never touched the bucket.
2. Inspect what was written. Run aws s3 ls s3://<bucket>/<prefix>/ --recursive --summarize. You should see new objects under the expected prefix and nowhere else. Objects outside the prefix mean a misconfigured key or an over-broad policy.
3. Test the read-only principal. With pull-request credentials, confirm identity with aws sts get-caller-identity, then attempt aws s3 cp ./probe.txt s3://<bucket>/<prefix>/probe.txt. It must fail with AccessDenied. If it succeeds, untrusted branches can poison artifacts that trusted builds will consume.

Operational hygiene

To invalidate everything, change the key prefix rather than deleting objects. The switch is atomic, and lifecycle rules expire the old prefix on their own.

Most cache bugs come from inputs that are missing from the hash. Declare { "env": "API_URL" } and { "runtime": "node --version" } in namedInputs, and exclude test and documentation files from production inputs. Without these, two builds that differ in environment or toolchain share a hash and restore each other's outputs.

What to monitor

The CloudWatch storage metrics BucketSizeBytes and NumberOfObjects are updated daily. Enable S3 request metrics to get AllRequests, 4xxErrors and 5xxErrors. A rise in 4xx errors usually points to expired credentials or a policy regression.

There is no built-in hit-rate metric. Count [remote cache] lines in CI logs and track the trend. A sudden drop often means an input such as a timestamp or absolute path became nondeterministic.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx remote cache s3, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.