Nx Remote Cache on S3: Setup, IAM Permissions and Eviction

How Nx remote caching works and who it is for

Nx caches tasks such as build, test and lint. Before running one, it computes a hash from the task's inputs: the relevant source files, project and workspace configuration, the hashes of dependent projects, and any runtime values or environment variables the task is configured to include. Nx uses that hash as the cache key. When the task finishes, it stores the declared output files and the captured terminal output as an artifact under the hash.

On the next run Nx recomputes the hash. It checks the local cache first, then the remote cache. On a remote hit it downloads the artifact, restores the outputs and replays the logs instead of running the task. The task looks like it ran, but it took as long as a download.

The value comes from sharing. In a monorepo with many developers, CI rebuilds the same unchanged libraries on every pull request, and developers rebuild what CI has already built. A shared cache turns that repeated work into downloads. A single developer on a small repo gets little from it beyond what the local cache already provides.

The cache is only correct if the hash captures everything that affects the output. Suppose a build reads an environment variable that is not declared as an input, depends on a generated file Nx does not track, or behaves differently across compiler or runtime versions. Two different environments then produce the same hash, and the first artifact uploaded wins. With a local cache, that mistake stays on one machine. With a remote cache, everyone who pulls the artifact gets the wrong output and has no sign that anything is off.

Copy +-------------------------------+
 | Task inputs                   |
 |  source files                 |
 |  project + workspace config   |
 |  dependent project hashes     |
 |  declared env vars, runtime   |
 +------+------------------------+
        |
        v
 +--------------+
 | Nx task hash |  = cache key
 +------+-------+
        | lookup
        v
 +--------------+   miss   +--------------+   miss   +------------------+
 | local cache  |--------->| remote cache |--------->| run task, store  |
 | (machine)    |          | (S3 bucket)  |<---------| outputs + logs   |
 +------+-------+          +------+-------+  upload  | under hash       |
        | hit                     | hit              +------------------+
        v                         v
 +------------------------------------------+
 | restore outputs, replay terminal logs    |
 +------------------------------------------+

 +--------------------------------------------------------------------+
 | Undeclared input -> different envs, same hash -> bad artifact      |
 | uploaded once -> restored for every developer and CI job           |
 +--------------------------------------------------------------------+

Nx hashes a task's declared inputs into a cache key, checks the local cache and then the S3 remote cache, and restores outputs and logs on a hit or runs the task and uploads the artifact on a miss, so any input left out of the hash spreads a wrong artifact to the whole team.

Choosing an integration path: Nx's S3 cache package or a custom HTTP cache service

Which integration path you choose decides who can write that artifact and whether you can trace where it came from.

There are three realistic options. Nx publishes its own self-hosted S3 cache package, which talks to the bucket directly from each workspace. Nx also documents an HTTP API for self-hosted remote caches, so you can run a small service that implements that API and stores artifacts in S3. The third option is a managed cache, which removes the storage and operations work but sends your artifacts to a third party.

Direct S3 is the simplest option, but every security decision has to be expressed in IAM policy. A custom service lets you scope tokens per pipeline, log who wrote each hash, and evict artifacts that nobody reads. In exchange, you are on call for that service.

Before you commit, check the current Nx documentation and licensing terms for the Nx version you run. The packaging and licensing of these options have changed over time. Avoid older community runner plugins, which were written against interfaces Nx has since replaced.

Bucket baseline and least-privilege IAM by trust level

Create a bucket used only for the Nx cache, in the same region as your CI runners. Same-region reads are faster, and they avoid the cross-region transfer charges that build up when every pipeline pulls artifacts. Turn on Block Public Access at the bucket level and set default encryption. Do not reuse a bucket that holds other data. A cache bucket gets broad write access and aggressive lifecycle rules, and neither belongs near logs or release artifacts.

IAM by trust level

• Trusted-branch CI (main and release branches) gets s3:GetObject and s3:PutObject on the bucket's object ARN.
• Developers and PR builds get s3:GetObject only. They can benefit from the cache but cannot write to it.
• Listing, if your integration needs it, uses s3:ListBucket on the bucket ARN. Scope it to the cache path with an s3:prefix condition.

SSE-KMS permissions

SSE-KMS adds two requirements on the key:

• Writers need kms:GenerateDataKey.
• Readers need kms:Decrypt.

If either is missing, S3 returns a 403 AccessDenied, even when the S3 permissions are correct. Nx usually reports this as a failed cache read or write and then rebuilds locally. The build stays green, so the failure is easy to miss. Check the error message or CloudTrail for the KMS action that was denied.

Copy{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TrustedCiCacheReadWrite",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::example-nx-cache-us-east-1/nx-cache/*"
    },
    {
      "Sid": "ListOnlyUnderCachePrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::example-nx-cache-us-east-1",
      "Condition": {
        "StringLike": { "s3:prefix": ["nx-cache/*"] }
      }
    },
    {
      "Sid": "SseKmsWriteAndRead",
      "Effect": "Allow",
      "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
    }
  ]
}

Read-write IAM policy for trusted-branch CI with prefix-scoped listing and SSE-KMS permissions; for the read-only developer/PR policy, remove s3:PutObject and kms:GenerateDataKey.

Delivering credentials to CI and laptops without long-lived keys

CI: OIDC federation instead of stored keys

Most CI providers can issue a signed OIDC token per job. Register the provider as an identity provider in IAM, then create roles whose trust policy accepts only specific token claims. On GitHub Actions, for example, the sub claim encodes the repository and ref, so a condition matching repo:your-org/your-repo:ref:refs/heads/main restricts the write-capable role to default-branch builds. Give pull-request jobs a separate read-only role with a looser sub match. Because the branch check sits in the trust policy, a workflow edit on a feature branch cannot assume the write role. The credentials last for the job and expire on their own, so there is no secret to rotate and nothing useful in a leaked log.

Laptops: SSO sessions

Developers should get credentials through IAM Identity Center: aws sso login with a named profile mapped to the read-only permission set. The AWS SDK's default credential chain picks up the session through AWS_PROFILE, so the cache client needs no Nx-specific configuration. When the session expires, the developer runs the login again. No access key ever lives in a dotfile.

Keep the remote cache optional

A developer without a valid session should still be able to build. Test what your integration does when authentication fails. If it errors out instead of warning and falling back to the local cache, enable the remote cache only when credentials are present. Gate it on an environment variable that CI sets and laptops opt into.

For a single debugging run, --skip-nx-cache forces the task to execute rather than restore from any cache.

Copy# .github/workflows/ci.yml
name: ci
on:
  push:
    branches: [main]
  pull_request:

permissions:
  id-token: write   # lets the job request a signed OIDC token
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Exchange the job's OIDC token for short-lived AWS credentials.
      # main asks for the write role and everything else asks for the read-only role.
      # The IAM trust policies enforce this split. Asking for the write role from a
      # feature branch fails because its sub claim does not match
      # repo:your-org/your-repo:ref:refs/heads/main.
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-region: eu-west-1
          role-to-assume: ${{ github.ref == 'refs/heads/main' && 'arn:aws:iam::123456789012:role/nx-cache-write' || 'arn:aws:iam::123456789012:role/nx-cache-read' }}

      - run: npm ci

      # Opt-in flag for the remote cache. CI sets it and laptops leave it unset,
      # so the workspace falls back to the local cache. Wire this variable into
      # your remote cache setup yourself. Nx does not read it.
      - run: npx nx affected -t build test
        env:
          NX_REMOTE_CACHE_ENABLED: 'true'

# Debugging a single run locally, ignoring every cache:
#   npx nx build my-app --skip-nx-cache

A GitHub Actions job exchanges its OIDC token for a branch-dependent IAM role (write on main, read-only elsewhere) and turns on the remote cache through an opt-in environment variable, with a comment showing how to bypass the cache for one debugging run.

Cache poisoning: when not to let a build write

Nx restores a cached artifact by matching a computation hash. It does not check who produced the artifact or whether its contents still match the source. Anyone who can write an object under a given hash decides what every other machine receives for that hash. If an attacker uploads a tampered dist/ folder or test result for a hash that main will compute, CI and developers will restore it without warning. Unless the cache is purged, they will keep restoring it until the inputs change.

Fork and pull request builds are the obvious entry point. PR code is untrusted by definition. A malicious change to a build script can write arbitrary outputs before the PR is ever reviewed. Give those jobs read-only credentials, or none at all. Only builds from protected branches, running on runners you control, should write.

Separating prefixes or buckets by trust level costs hit rate, because a PR build cannot seed artifacts that main later reuses. That trade-off is worth it. A missed cache hit costs build minutes. A poisoned artifact can ship to production.

Avoid a shared remote cache entirely in three cases:

• Untrusted contributors with no isolation. If you cannot separate write access, do not share a cache.
• Unreliable input definitions. If inputs miss environment variables, generated files, or tool versions, different builds collide on one hash, and you get stale outputs with no sign of the error.
• Slow or cross-region links. If downloading large artifacts takes longer than rebuilding them, and also adds transfer charges, the remote cache only adds cost.

Eviction with S3 lifecycle rules and choosing a retention window

A lifecycle rule scoped to the cache prefix should do three things. It should expire current objects after N days. It should abort incomplete multipart uploads after a day or two, because interrupted CI jobs leave orphaned parts that are billed but never listed as objects. If versioning is enabled, it should also expire noncurrent versions quickly, or every overwrite and expiration leaves a hidden copy that you keep paying for.

S3 expiration counts from object creation, not last access. An artifact read a thousand times a day is deleted on the same day as one nobody touched. The next build misses and re-uploads it. That costs one rebuild per hot artifact per retention window, which is usually acceptable. If you need true LRU eviction, you have to record reads yourself, for example through a custom cache service, and delete objects based on that data.

Illustrative estimate

Assumed inputs: 200 CI runs/day, 40 cacheable tasks per run, a 20% miss share, 15 MB average artifact, and 14-day retention.

• Uploads/day = runs × tasks × miss share = 200 × 40 × 0.2 = 1,600 PUTs
• Downloads/day = runs × tasks × (1 − miss share) = 6,400 GETs
• Steady-state storage ≈ uploads/day × size × retention days = 1,600 × 15 MB × 14 ≈ 336 GB

This is an upper bound because it assumes every miss produces a new hash.

Copy#!/usr/bin/env bash
set -euo pipefail

BUCKET="example-nx-cache-bucket"   # placeholder bucket name
PREFIX="nx-cache/"                 # only cache objects are affected
RETENTION_DAYS=14                   # matches the illustrative estimate

# Expiration counts from object creation, not last read.
# Aborting multipart uploads clears orphaned parts left by cancelled CI jobs.
# NoncurrentVersionExpiration only takes effect if versioning is enabled.
cat > lifecycle.json <<EOF
{
  "Rules": [
    {
      "ID": "nx-cache-eviction",
      "Filter": { "Prefix": "${PREFIX}" },
      "Status": "Enabled",
      "Expiration": { "Days": ${RETENTION_DAYS} },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 1 }
    }
  ]
}
EOF

# Warning: this call replaces the bucket's entire lifecycle configuration.
# Merge in any existing rules before applying.
aws s3api put-bucket-lifecycle-configuration \
  --bucket "$BUCKET" \
  --lifecycle-configuration file://lifecycle.json

# Confirm the rule is in place.
aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET"

# Steady-state storage, upper bound:
# runs/day x tasks/run x miss share x avg MB x retention days
echo "200 * 40 * 0.2 * 15 * ${RETENTION_DAYS} / 1000" | bc -l  # about 336 GB

Applies an S3 lifecycle rule scoped to the Nx cache prefix that expires objects after 14 days, aborts incomplete multipart uploads after one day and removes noncurrent versions after one day, then evaluates the storage formula from the illustrative estimate.

Verifying the cache, ongoing operations and cost impact

Verify end to end

Start with two machines that have no local Nx cache, such as two fresh CI runners or a CI runner and a freshly cloned laptop. Run the same task on the same commit on both. The first run should execute and upload. The second should report that its output came from the remote cache. If the second machine rebuilds anyway, the task hashes differ. Common causes are environment variables, OS-specific inputs or an unpinned toolchain. Compare the hash details for that task on each machine before you blame S3.

After a CI run on the main branch, list the cache prefix and confirm that new objects arrived with current timestamps. Then enable S3 request metrics or server access logs on the bucket and watch two signals:

• Writes from unexpected principals. These mean a role has broader access than you intended.
• Spikes in 403 errors. These usually mean expired credentials or a broken trust policy, which quietly downgrades every build to a cache miss.

Ongoing hygiene

When you upgrade Node, a compiler or the shared input definitions, rotate to a new cache prefix or purge the old one. Don't rely on hashing alone to catch every toolchain change. Define the bucket, its lifecycle rules and the IAM roles in infrastructure-as-code so that permission changes go through review and drift shows up in plan output.

Cost against time saved

Illustrative estimate (assumed numbers, not a benchmark)

Monthly net saving = (CI minutes saved × price per minute) + (developer hours saved × loaded hourly rate) − (storage + request + KMS charges) − platform team hours × rate

Assumed inputs:

• 20,000 CI minutes saved at $0.008 per minute ≈ $160
• 40 developer hours saved at $100 per hour = $4,000
• $30 in AWS charges
• 6 platform hours at $100 per hour = $600

Result: roughly $3,500 net per month.

The platform team's hours are the term people most often underestimate. Those hours cover availability, quarterly access reviews and incident response.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx remote cache s3, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.