How does Nx remote caching work, and what does an S3 cache actually store?

Nx computation caching treats a task such as build, test or lint as a deterministic function of its inputs. Before running a task, Nx computes a hash from:

  • the project's source files
  • the inputs configured for that target
  • the hashes of the projects it depends on
  • any declared runtime commands and environment variables

If that hash has been seen before, Nx skips execution. It restores the task's declared output files and replays the recorded terminal output, so the result looks the same as a fresh run.

A remote cache moves that lookup from a developer's local disk to shared storage. On S3, each entry is an object keyed by the task hash. The object holds the archived outputs and the captured terminal log. S3 does not interpret the contents. It is a key-value store.

  Developer laptop         CI runners
        |  ^                 |  ^
  write |  | read      write |  | read
        v  |                 v  |
   +--------------------------------+
   |  Integration layer             |
   |  (Nx S3 package or cache       |
   |   server in front of S3)       |
   +--------------------------------+
        |  write (PUT)   ^  read (GET)
        v                |
   +--------------------------------+
   |  S3 bucket: <hash> -> archive  |
   +--------------------------------+

This design has two consequences:

  • Undeclared inputs cause stale hits. If a task reads a file or environment variable that is not part of its inputs, two different builds can produce the same hash. The second build then receives the wrong artifact.
  • Write access is a trust boundary. Nx restores whatever is stored under a hash. Anyone who can write to the bucket can place content that other machines will reuse as if they had built it.

Teams accept these constraints because the payoff is large. A one-line change in one library no longer triggers rebuilding and retesting every unaffected project, on CI or on any laptop. That cuts pipeline time and reduces how long engineers wait on builds.

+----------------------+                  +----------------------+
|   Developer laptop   |                  |      CI runners      |
| hash = f(sources,    |                  | hash = f(sources,    |
| inputs, deps, env)   |                  | inputs, deps, env)   |
+----------------------+                  +----------------------+
    |  ^                                      |  ^
  W |  | R                                  W |  | R
    v  |                                      v  |
+----------------------------------------------------------------+
|                       Integration layer                        |
|  Nx S3 cache package -or- a cache server in front of S3        |
|  Credentials here decide who can read and who can write        |
+----------------------------------------------------------------+
        |  write: PUT <hash>            ^  read: GET <hash>
        v                               |
+----------------------------------------------------------------+
|  S3 bucket (opaque key-value store; contents not interpreted)  |
|    <hash-a>  ->  archive: declared outputs + terminal output   |
|    <hash-b>  ->  archive: declared outputs + terminal output   |
+----------------------------------------------------------------+

 W = write path: cache miss -> run task -> upload archive under hash
 R = read path:  cache hit  -> restore outputs, replay terminal log
 Trust boundary: anything written under a hash is restored as-is.

Developers and CI runners compute the same task hash, read cached archives on a hit and write new ones on a miss through the integration layer, which stores them in S3 as opaque objects keyed by hash.

Which Nx S3 integration path should you choose?

Before picking a path, run nx report (or check package.json) to confirm your installed Nx version, then read the docs for that major version. Support for custom runners, the Powerpack packages, and the self-hosted cache API changed between major releases, so advice that fit one version may not fit yours.

Option Suits Maintenance burden Upgrade risk Licensing
Legacy tasksRunnerOptions custom runner (community S3 runners) Teams pinned to older Nx with no near-term upgrade Low until it breaks, then high High. The API is deprecated and replaced, so runners can stop working on upgrade Depends on each community package's license
Official S3 cache package in Nx Powerpack Teams that want a supported integration and minimal custom code Low: configure the bucket and credentials Low, because it tracks Nx releases Separate terms; review them before adopting
Your own cache server implementing the self-hosted remote cache HTTP API, backed by S3 Platform teams that need custom auth, auditing, or storage control Highest: you own the code, uptime, and security Moderate. The documented API is a stable contract, but you must track changes to it Your code; check the terms for the API itself in your version's docs
No remote cache (local only) Small repos, solo work, or strict data rules None None None

If you still use a community runner, plan its migration now. Don't wait for an upgrade to force it. The choice that remains is whether you want to own a service. If not, Powerpack is the lower-effort route, provided its terms work for you. If you need control over authentication and storage, build the server.

#!/usr/bin/env bash
# Audit an Nx workspace before choosing a remote cache path.
# Run from the workspace root. Requires jq.
set -euo pipefail

# 1. Confirm the installed Nx version, then read that major version's docs.
echo "== Installed Nx version =="
npx nx --version

# Full environment report (Nx packages, plugins, Node version).
npx nx report

# 2. Detect the legacy tasksRunnerOptions config used by community S3 runners.
# jq -e exits non-zero when the key is missing or null.
echo "== Legacy custom runner check =="
if jq -e '.tasksRunnerOptions' nx.json >/dev/null 2>&1; then
  echo "tasksRunnerOptions found: plan a migration before upgrading Nx."
  # Show which runner package each entry points at.
  jq '.tasksRunnerOptions | to_entries[] | {name: .key, runner: .value.runner}' nx.json
else
  echo "No tasksRunnerOptions in nx.json."
fi

# 3. List cache-related dependencies so you can review each package's license.
echo "== Cache-related dependencies =="
jq -r '((.dependencies // {}) + (.devDependencies // {}))
  | to_entries[]
  | select(.key | test("cache|s3|powerpack"; "i"))
  | "\(.key)@\(.value)"' package.json

A shell audit that prints the installed Nx version, flags any legacy tasksRunnerOptions custom runner in nx.json, and lists cache-related dependencies so you can review their licenses before choosing a path.

How should you design and lock down the S3 bucket?

Start with a dedicated bucket for build artifacts, not a corner of a bucket that also holds backups or logs. Cache entries have a different trust level and retention profile from everything else, and a separate bucket keeps IAM policies and lifecycle rules simple. Inside it, give each repository its own prefix, such as repos/<repo-name>/. This lets you scope credentials per repository and purge one repository's cache without affecting the others.

Enable all four Block Public Access settings at the bucket level. A cache never needs anonymous reads, and these settings override any policy or ACL that someone adds by mistake later. Set default encryption: SSE-S3 is enough for most teams, and SSE-KMS is worth the extra key-policy work if you want a second, independent access check and key-usage audit trails. Leave versioning off. A cache entry that has been overwritten or deleted has no recovery value, and versioning would keep paying to store it.

Add a lifecycle rule scoped to the cache prefix that expires objects after a fixed number of days and aborts incomplete multipart uploads. Retention is a trade-off:

  • Short retention (for example, 7 days) lowers storage costs. Long-lived release branches will rebuild more often.
  • Long retention improves hit rates on those branches. It also keeps artifacts from toolchains or dependencies you have since stopped trusting.

The bucket policy should deny any request where aws:SecureTransport is false. It should also deny PutObject requests that specify an encryption method other than the bucket default. Default encryption already covers uploads that send no encryption header, so this check only needs to reject mismatches.

#!/usr/bin/env bash
# Apply the lifecycle rule and bucket policy to a dedicated Nx cache bucket.
set -euo pipefail
BUCKET="example-nx-cache"   # placeholder bucket name

# Expire cache entries after 14 days and clean up abandoned multipart uploads.
# Scoped to repos/ so any other prefix in the bucket is unaffected.
cat > lifecycle.json <<'EOF'
{"Rules": [{
  "ID": "expire-nx-cache", "Status": "Enabled",
  "Filter": {"Prefix": "repos/"},
  "Expiration": {"Days": 14},
  "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 1}
}]}
EOF

# Deny any non-TLS request. Deny uploads that send an SSE header other than
# AES256 (SSE-S3). The Null check means the second rule applies only when the
# header is present. Uploads without it fall back to default encryption.
cat > policy.json <<EOF
{"Version": "2012-10-17", "Statement": [
  {"Sid": "DenyInsecureTransport", "Effect": "Deny", "Principal": "*",
   "Action": "s3:*",
   "Resource": ["arn:aws:s3:::${BUCKET}", "arn:aws:s3:::${BUCKET}/*"],
   "Condition": {"Bool": {"aws:SecureTransport": "false"}}},
  {"Sid": "DenyMismatchedEncryption", "Effect": "Deny", "Principal": "*",
   "Action": "s3:PutObject", "Resource": "arn:aws:s3:::${BUCKET}/*",
   "Condition": {
     "Null": {"s3:x-amz-server-side-encryption": "false"},
     "StringNotEquals": {"s3:x-amz-server-side-encryption": "AES256"}}}
]}
EOF

aws s3api put-bucket-lifecycle-configuration --bucket "$BUCKET" \
  --lifecycle-configuration file://lifecycle.json
aws s3api put-bucket-policy --bucket "$BUCKET" --policy file://policy.json

Uses the AWS CLI to add a 14-day expiry rule on the repos/ prefix and a bucket policy that rejects non-TLS requests and uploads that ask for an encryption method other than SSE-S3.

What IAM permissions and CI credentials does an Nx S3 cache need?

Cache credentials need the same care as deploy credentials. Anyone who can write to the cache can change the build output that other machines restore. Build two IAM roles scoped to one bucket and one prefix, such as nx-cache/. Grant only the actions your integration actually calls.

Role Object actions on bucket/nx-cache/* Bucket action Who assumes it
Read-only s3:GetObject s3:ListBucket, conditioned on s3:prefix matching nx-cache/* Pull-request CI, developers
Read-write s3:GetObject, s3:PutObject Same as read-only CI on main and other trusted branches

Put s3:ListBucket in both roles, even though Nx never lists objects. If a principal lacks list permission and requests a key that doesn't exist, S3 returns 403 AccessDenied instead of 404 NoSuchKey. S3 does this so a caller can't learn which keys exist. As a result, an ordinary cache miss looks like a permissions failure, and depending on the client, the run logs an error or fails. If the bucket uses SSE-KMS, add kms:Decrypt to both roles and kms:GenerateDataKey to the writer.

In CI, use OIDC to assume the role. The workflow exchanges its identity token for short-lived STS credentials, so there are no access keys to leak or rotate. Restrict the read-write role in its trust policy with a StringEquals condition on token.actions.githubusercontent.com:sub equal to repo:your-org/your-repo:ref:refs/heads/main, plus an aud condition of sts.amazonaws.com. Pull requests, especially from forks, assume only the read-only role. Fork code is untrusted, so it must never be able to write artifacts that main will restore.

Give developers read-only access by default. The trade-off is that local builds never warm the cache, so a developer who works ahead of CI pays the full build cost until main catches up.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowMainBranchOnly",
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"
        }
      }
    }
  ]
}

Trust policy for the read-write cache role: only GitHub Actions OIDC tokens for the main branch of your-org/your-repo can assume it, so pull requests and forks cannot obtain write credentials.

How do you build a minimal self-hosted Nx cache server in front of S3?

Keep the server small: two routes, one auth check, one bucket. The Nx docs define the exact paths, headers and status codes for each version, and they also name the environment variables that point Nx at the server and supply its token. Copy those from the docs for your version rather than from memory.

Illustrative handler sketch (not production code):

  • Auth: Read the bearer token and compare it in constant time against a read token or a write token. Return 401 if the token is missing and 403 if a read-only token attempts a PUT.
  • GET by hash: Request the object from S3 and pipe its body stream directly into the response. If S3 reports that the key does not exist, return 404. Nx treats a 404 as a cache miss and builds the task locally.
  • PUT by hash: Stream the request body into a multipart S3 upload so that a large artifact never sits whole in memory. Use a conditional create-only write so the upload fails if the key already exists, and return 409 in that case. A HEAD-then-PUT check is racy because two runners can both see the key as absent and then both write.

Running this server makes you responsible for its uptime, its token issuance and rotation, OS and dependency patching, and scaling under parallel CI load. Because the server streams rather than buffers, it is mostly network-bound, so scale it horizontally.

Treat the cache as an optimization. If the server is down, builds should fall back to running without the remote cache, not fail. Test how your Nx version behaves when the cache is unreachable, and alert on error rates so that a silent fallback does not hide an outage.

When should you avoid a self-hosted Nx remote cache, and what are the risks?

Cache poisoning is the main risk

Anyone with write access can upload an artifact under a hash that other machines will restore, then execute or ship. A compromised CI job or a leaked laptop credential is enough. Write-once objects stop an attacker from overwriting a known-good entry, and branch-scoped roles stop pull-request builds from writing entries that main consumes. Neither prevents a malicious first write to a hash nobody has computed yet. Keep write access on trusted CI runners and give developers read-only access.

Undeclared inputs fail more quietly. If a task reads an environment variable or file that is missing from its declared inputs, two different builds produce the same hash and one restores the other's output. A wrong hit is worse than a miss because it is fast and silent, and the bad output can reach production.

Cost and latency

The following is an illustrative estimate. It assumes a 20 MB/s download from a distant region and about 0.5 s of lookup overhead.

Scenario Local run Remote restore Verdict
Lint task, 80 MB output 4 s ~4.5 s Slower than running locally
Build task, 200 MB output 180 s ~10.5 s Clear win
1,000 restores/day at 200 MB n/a ~200 GB/day egress Multiply by your egress rate

Large outputs increase storage and transfer costs. Hashes that change on every run, for example because of embedded timestamps, push the hit rate toward zero while you keep paying for uploads.

Skip remote caching entirely when the repo is small, most tasks finish in seconds, or a measured hit rate stays low.

How do you verify cache correctness and operate the cache over time?

Make the hash trustworthy first

A remote cache is only correct if the hash captures everything that affects the output. Declare inputs and outputs explicitly in nx.json (via targetDefaults and namedInputs) or in project configuration rather than relying on defaults. Add environment variables that change build output as env inputs, and add tool versions such as the Node or compiler version as runtime inputs. If these are missing, two machines can produce different artifacts under the same hash, and the cache will serve the wrong one without reporting an error. Keep timestamps, build IDs and absolute paths out of outputs. Each of these changes the artifact on every run, and files that feed downstream inputs can then invalidate dependent hashes.

Verify before you trust it

Run nx reset to clear local state, then run the same task twice. The second run should be served from the remote cache. To check that a cached artifact matches a real build, run the task with --skip-nx-cache and diff the result against the restored output. In CI logs, confirm that tasks report remote hits and not only local ones. In S3, look for objects that are written but never read. Those usually mean a hash is unstable.

Operate it like any other service

Track hit rate, artifact size and read/write latency, either from your cache server's metrics or from S3 access logs. Alert on elevated error rates, because a failing cache usually shows up as slower CI rather than as failed builds. Document a purge procedure, such as deleting a prefix or the whole bucket. Purging is safe because every entry can be rebuilt from source. The only cost is a period of cold builds.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx remote cache s3, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.