Nx Remote Cache on S3 or MinIO: Self-Hosted Setup and Security

How Nx remote caching works and what a self-hosted backend does

Before running a task, Nx computes a hash from everything that is supposed to determine its result: the source files of the project and its dependencies (filtered by the task's configured inputs), the task's target options, declared runtime inputs and environment variables, and relevant workspace configuration. If two machines produce the same hash, Nx assumes the task would produce the same outputs, so it can skip execution.

On a miss, Nx runs the task, then packages the declared output directories and the captured terminal output into an artifact stored under that hash. On a hit, it downloads the artifact, restores the files to their output paths, and replays the logs. The logs are replayed so the run looks as if the task had executed.

A remote cache shares those artifacts between machines. With S3 or MinIO, the bucket only stores and returns blobs by key. It does not know how a hash was derived, whether the artifact matches its key, or who should be trusted to write it. Nx trusts whatever it downloads. All integrity therefore depends on who holds write access, which shapes the rest of this article.

Copy  Developer machines ──read──┐
                             │
  PR / fork CI jobs ───read──┼──► [ HTTP cache server ] ──► [ S3 / MinIO bucket ]
                             │      (optional; or direct
  Protected-branch CI ─read──┤       SDK access to bucket)
                      └write─┘

Only protected-branch CI writes. Everything else reads.

Copy ┌──────────────────────┐                   ┌──────────────────────┐
 │ Developer machines   │── read (GET) ────►│ HTTP cache server    │
 └──────────────────────┘                   │ (optional)           │
 ┌──────────────────────┐                   │                      │
 │ PR / fork CI jobs    │── read (GET) ────►│ grants PUT only to   │
 └──────────────────────┘                   │ protected-branch CI  │
 ┌──────────────────────┐                   │                      │
 │ Protected-branch CI  │── read (GET) ────►│ or: direct SDK access│
 │ (trusted refs only)  │══ write (PUT) ═══►│ with IAM policies    │
 └──────────────────────┘                   └──────────┬───────────┘
                                                       │ GET / PUT
                                                       │ object by hash key
                                                       ▼
                                            ┌──────────────────────┐
                                            │ S3 / MinIO bucket    │
                                            │ stores blobs by key  │
                                            │ no integrity checks  │
                                            └──────────────────────┘

 ──► read path:  cache hit -> restore outputs, replay terminal logs
 ══► write path: cache miss -> run task, upload outputs + logs under hash

Every client reads from the cache, but only protected-branch CI can write, because the S3 or MinIO bucket stores and serves blobs by hash key without verifying them.

Who should self-host, when it pays off, and when to avoid it

A remote cache pays off when the same task runs with the same inputs in many places. A CI job on main builds a library, and every later pull request, developer checkout, and deploy pipeline that hashes to the same inputs downloads the output instead of rebuilding it. The savings come from deduplicating work across machines, not from making any single build faster.

Who it suits

Self-hosting fits platform teams that already run object storage and have a reason to keep build artifacts in-house:

• On-prem or air-gapped networks, where CI runners cannot reach external services.
• Data-residency requirements, where compiled output and source-derived artifacts must stay in a specific jurisdiction or account.
• Existing S3 or MinIO operations, where bucket policies, IAM, monitoring, and lifecycle rules are already routine.

When to avoid it

• Small or fast tasks. If a lint or unit-test target finishes in two seconds, fetching and extracting its artifact over the network can take longer. Measure before you enable caching for these targets.
• Cross-region or cross-cloud runners. Latency and egress charges can cancel the benefit, especially for large artifacts.
• No owner. Someone has to rotate credentials, watch uptime, and respond when a poisoned or corrupted entry ships. If nobody has capacity for that, don't run a cache.

MinIO in particular adds stateful infrastructure: disks, replication, and upgrades. Because every entry can be rebuilt from source, some teams deliberately run a single node without backups and accept that a lost cache only costs one cold rebuild.

Integration paths and alternatives

There are four realistic ways to wire Nx to shared cache storage. They differ mainly in who owns the code between Nx and the bucket, and that owner determines your trust controls and how much breaks during upgrades.

The direct integration is the fastest to adopt. Every client needs storage credentials, though, so the bucket policy is your only enforcement point. The HTTP server costs you a process to run. In return, you get a place to separate CI writers from developer readers without handing anyone raw bucket keys. A vendor-hosted cache removes operations work but moves artifacts and access policy outside your boundary. Local-only caching is still the right answer for small repositories or for teams whose CI runners already persist a workspace.

Nx's package names, entitlements, and licensing for self-hosted caching have changed across releases. Check the current Nx documentation before you commit to an approach. If you pin one integration, expect some migration work when you upgrade Nx.

Provisioning the bucket on S3 or MinIO

On AWS S3

Create a bucket that holds nothing but cache artifacts, in its own account or project if you can. This keeps IAM policies short and stops a lifecycle rule from deleting unrelated data. Use aws s3api create-bucket, then aws s3api put-public-access-block with all four settings set to true. A misconfigured bucket policy then can't expose build outputs, which often embed source and environment details.

Run put-bucket-encryption to set default server-side encryption. Run put-bucket-versioning so an overwritten or deleted artifact can be recovered during a poisoning investigation. Run put-bucket-logging to send access logs to a separate bucket. Those logs are how you will later attribute a suspicious write to a specific credential.

Apply expiry with put-bucket-lifecycle-configuration. Here is an illustrative rule set:

• Expiration: {Days: 30}
• NoncurrentVersionExpiration: {NoncurrentDays: 7}
• AbortIncompleteMultipartUpload: {DaysAfterInitiation: 1}

Without the noncurrent rule, versioning keeps every overwritten object forever. Shorter expiry lowers storage cost. It also evicts artifacts that long-lived release branches still need, and those branches then rebuild from scratch. Pick the window based on how often you build old release lines, not on the build rate of main.

On MinIO

S3 clients target AWS by default. You need to override the endpoint (with the AWS CLI, --endpoint-url) and usually enable path-style addressing, because virtual-hosted bucket DNS is rarely configured. Terminate TLS on MinIO itself so credentials and artifacts never travel in plaintext. Treat MinIO access keys as production secrets: store them in your secret manager, rotate them, and never use the root credentials. Use mc mb and mc version enable for setup and mc ilm for expiry.

Copy#!/usr/bin/env bash
set -euo pipefail
BUCKET=nx-cache-example        # cache artifacts only, nothing else
LOG_BUCKET=nx-cache-logs-example  # separate bucket; must already allow S3 log delivery
REGION=eu-west-1

# Create the bucket and block every form of public access.
aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" \
  --create-bucket-configuration LocationConstraint="$REGION"
aws s3api put-public-access-block --bucket "$BUCKET" --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

# Default encryption, versioning for poisoning forensics, access logs for attribution.
aws s3api put-bucket-encryption --bucket "$BUCKET" --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
aws s3api put-bucket-versioning --bucket "$BUCKET" --versioning-configuration Status=Enabled
aws s3api put-bucket-logging --bucket "$BUCKET" --bucket-logging-status \
  '{"LoggingEnabled":{"TargetBucket":"'"$LOG_BUCKET"'","TargetPrefix":"nx-cache/"}}'

# Illustrative expiry: tune Days to how often old release lines still build.
aws s3api put-bucket-lifecycle-configuration --bucket "$BUCKET" --lifecycle-configuration '{
  "Rules": [{
    "ID": "expire-nx-cache", "Status": "Enabled", "Filter": {"Prefix": ""},
    "Expiration": {"Days": 30},
    "NoncurrentVersionExpiration": {"NoncurrentDays": 7},
    "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 1}
  }]}'

# --- MinIO: same bucket settings, different endpoint and addressing ---
# Path-style URLs, since virtual-hosted bucket DNS is rarely set up for MinIO.
aws configure set default.s3.addressing_style path
MINIO=   # TLS terminated on MinIO itself
aws s3api create-bucket --bucket "$BUCKET" --endpoint-url "$MINIO"

# Or with mc. Keys come from the secret manager via env vars, never the root user.
mc alias set nxcache "$MINIO" "$MINIO_CACHE_ACCESS_KEY" "$MINIO_CACHE_SECRET_KEY"
mc mb nxcache/"$BUCKET"
mc version enable nxcache/"$BUCKET"

Provisioning a dedicated, private, encrypted, versioned and logged Nx cache bucket with an illustrative lifecycle expiry rule on S3, plus the endpoint override, path-style addressing and mc commands needed for MinIO.

Least-privilege credentials: a split read/write model

The cache's security comes down to who can write to it. Any principal that can PutObject into the cache prefix can plant an artifact that every other machine will restore without question. Limit writes to CI jobs on protected branches, where code has already passed review. Developer laptops and pull-request jobs, especially those from forks, get read-only access or no remote cache at all.

Two policies, one prefix

Create two policies scoped to the cache prefix (for example arn:aws:s3:::build-cache/nx/*), not the whole bucket. The writer policy allows s3:GetObject and s3:PutObject on that prefix, plus s3:ListBucket on the bucket, conditioned on the same prefix. The reader policy omits PutObject. Neither policy should include DeleteObject, because expiry belongs to lifecycle rules. MinIO accepts the same S3 policy grammar, so you can attach identical documents to MinIO users or groups and keep a single definition for both backends.

Prefer OIDC over static keys

On AWS, have CI exchange its OIDC token for a short-lived session on an assumable role. Write the role's trust policy to match the token's subject claim so that only protected-branch workflows can assume the writer role. With no stored secret, a fork has nothing to leak.

If you must use static keys, rotate them with overlap. Create a second key, update the secret store, confirm that CI succeeds with the new key, deactivate the old one, watch for failures, then delete it.

The cost of this model is that read-only developers rebuild unmerged changes locally until CI on the protected branch populates the cache.

Copy{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOnlyTheNxCachePrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::build-cache",
      "Condition": {
        "StringLike": { "s3:prefix": ["nx/*"] }
      }
    },
    {
      "Sid": "ReadAndWriteCacheArtifacts",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::build-cache/nx/*"
    }
  ]
}

The writer policy for protected-branch CI grants list access and object read/write only under the nx/ prefix and leaves out DeleteObject; the reader policy for developers is the same document without s3:PutObject, and MinIO accepts both unchanged.

Cache poisoning and hash correctness

A remote cache is trusted by its key. Nx computes a hash from a task's inputs, looks up that hash, and if an artifact exists, it restores the files and replays the terminal output without running anything. Whoever writes an artifact for a given hash therefore decides what every later consumer of that hash receives, including CI jobs that build release images. A plain S3 or MinIO bucket records no provenance and verifies no signature, so a planted artifact cannot be told apart from a legitimate one.

Limiting who can poison the cache

• Grant write credentials only to pipelines running on protected branches, so that an unreviewed change never produces a shared artifact.
• Protect those branches with required review, and keep write credentials out of jobs triggered by forks or pull requests.
• Turn on bucket versioning and access logging. Versioning keeps overwritten objects, and logs show which principal wrote each key and when.
• Document a purge procedure: rotate the writer credentials, delete or expire the affected prefixes, and force affected pipelines to skip the cache until it has been repopulated.

Hash correctness

Inputs that leave something out produce false cache hits. A build that depends on an environment variable or tool version missing from the hash will restore stale output, and the result looks like flaky, environment-dependent behavior. The example below declares named inputs, file sets that exclude tests, env entries, and a runtime entry for the Node version.

Never hash secrets, cache them in outputs, or print them in logs. Anyone with read access to the bucket can read all three.

Copy// nx.json: comments are accepted by Nx's JSON parser
{
  "namedInputs": {
    // Every file in the project, plus root config that affects all builds
    "default": ["{projectRoot}/**/*", "sharedGlobals"],
    "sharedGlobals": [
      "{workspaceRoot}/tsconfig.base.json",
      // Node version is part of the key, so different runtimes never share artifacts
      { "runtime": "node --version" }
    ],
    // Production inputs drop tests, so editing a spec does not invalidate builds
    "production": [
      "default",
      "!{projectRoot}/**/*.spec.ts",
      "!{projectRoot}/jest.config.ts"
    ]
  },
  "targetDefaults": {
    "build": {
      "cache": true,
      "dependsOn": ["^build"],
      "inputs": [
        "production",
        "^production",
        // Non-secret values that change the bundle must be in the hash
        { "env": "API_BASE_URL" },
        { "env": "BUILD_TARGET" }
        // Do NOT list tokens such as NPM_TOKEN here, and never write them into outputs
      ],
      "outputs": ["{workspaceRoot}/dist/{projectRoot}"]
    },
    "test": {
      "cache": true,
      "inputs": ["default", "^production"]
    }
  }
}

An nx.json that defines named inputs, excludes test files from production builds, and adds non-secret environment variables and the Node version to the build hash.

Verification, operations, and impact on cost and engineering time

Verify with two clean CI runs on the same commit. The first run should show cache misses and upload artifacts. The second run, on a fresh runner with no local cache, should restore most tasks from the remote cache, and Nx marks those tasks as remote hits in its output. Then run the same pipeline with --skip-nx-cache to get an uncached baseline. If the remote-hit run isn't clearly faster than the baseline, your hash inputs are probably too broad. During the trial, review bucket access logs (S3 server access logs or CloudTrail data events; MinIO audit logs). Any PUT from an identity other than the trusted CI writer is a policy gap and should be treated as an incident.

Two operational rules matter most. First, keep the bucket in the same region as the runners. Cross-region transfer adds both latency and egress charges to every restore. Second, test the failure path: block the endpoint or revoke the credentials, then confirm tasks run locally. A cache outage should make builds slower, not cause them to fail.

Illustrative estimate (assumed numbers, not a benchmark)

Net monthly value = (CI runs × minutes saved × CI $/min) + (developer minutes saved × loaded $/min) − (storage + requests + egress + host + ownership hours × $/hour)

In this example the net is about +$1,944. Developer wait time dominates, so measure it before you commit to running the cache yourself.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nx remote cache s3, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.