Docker BuildKit Cache Mount: Faster npm, Maven and pip in CI

What a cache mount does that layer caching cannot

Docker's layer cache works per instruction and is all-or-nothing. BuildKit reuses a layer only when the instruction and its inputs are unchanged, including the checksums of files brought in by COPY. Change one line in package-lock.json, pom.xml or requirements.txt, and the install layer is invalidated. The step then runs from an empty dependency cache and downloads every package again, including the hundreds that didn't change.

A docker buildkit cache mount (RUN --mount=type=cache) fixes this by attaching a persistent directory to a single RUN step. Examples are ~/.npm, ~/.m2/repository or ~/.cache/pip. The package manager reads from it and writes to it as usual, and BuildKit keeps it between builds on the same builder. When the lockfile changes, the step still re-runs, but the package manager finds most artifacts locally and fetches only the new or updated ones.

Two properties make this safe. First, the mount's contents are never committed to the image layer, so the cache does not bloat what you ship. Second, the contents are not part of the cache key, so a warm or cold cache never affects whether a layer is reused.

The engineers who benefit most are backend and platform engineers whose builds spend significant time resolving and downloading dependencies.

Prerequisites and how RUN --mount=type=cache works

Cache mounts are a BuildKit feature, so BuildKit has to be the builder that executes your Dockerfile. Recent Docker installations use it by default, and docker buildx build always does. On older setups that fall back to the legacy builder, the --mount flag fails as a parse error. Check this before you debug anything else.

The second requirement is the Dockerfile frontend, the component that parses the file. A # syntax= directive on the first line names which frontend image BuildKit should fetch and use. This is the conventional way to opt into a frontend that understands RUN --mount. Pointing it at the stable channel of the official Dockerfile frontend keeps the syntax available without depending on whatever parser your daemon happens to bundle.

A cache mount attaches a persistent directory at target for the duration of one RUN step. Files written there never enter the image layer. The remaining options control who shares that directory and who can write to it.

Copy# syntax=docker/dockerfile:1
# ^ Must be the first line. Opts into the stable Dockerfile frontend, which understands RUN --mount.
# Build with BuildKit, e.g.: docker buildx build -t example/app:dev .

FROM node:20-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
# Mount npm's download cache, not node_modules, so installed packages still land in the layer.
# The explicit id lets other stages or Dockerfiles on this builder reuse the same cache.
# sharing=locked serializes concurrent builds that touch this cache.
RUN --mount=type=cache,id=npm-cache,target=/root/.npm,sharing=locked \
    npm ci

FROM python:3.12-slim AS app
# Create a non-root user with fixed numeric IDs so the mount options can match them.
RUN groupadd --gid 10001 builder \
 && useradd --uid 10001 --gid 10001 --create-home builder
USER builder
WORKDIR /home/builder
COPY requirements.txt .
# The cache directory is created root-owned by default. uid/gid/mode make it writable
# for the builder user. A distinct id keeps the pip cache separate from the npm cache.
RUN --mount=type=cache,id=pip-cache,target=/home/builder/.cache/pip,uid=10001,gid=10001,mode=0755,sharing=locked \
    pip install --user -r requirements.txt

A multi-stage Dockerfile that opts into the Dockerfile frontend with the syntax directive and uses cache mounts with explicit ids, sharing=locked, and uid/gid/mode so that a non-root pip install can write to its cache.

Implementing cache mounts for npm, Maven and pip

npm

Mount the cache at /root/.npm and run npm ci. npm stores downloaded tarballs there, so a warm mount turns most of the install into local extraction. Do not point the mount at node_modules. Anything written to a cache mount is left out of the layer, so the dependencies would be missing from the final image. npm ci also deletes node_modules before installing, so a cached copy would be thrown away on every build.

Maven

Copy pom.xml alone and run a resolution step such as mvn dependency:go-offline before copying the source. Mount /root/.m2/repository on both this step and the later mvn package step. The resolution step builds up the local repository, and the package step fetches anything go-offline missed, such as some plugin dependencies. Plugin artifacts are stored in the same repository as dependencies, so the mount saves you from downloading the compiler, surefire and packaging plugins again on each build.

pip

Mount /root/.cache/pip and remove --no-cache-dir from the install command. That flag turns pip's cache off, which leaves the mount empty. People add it to keep images small, but the mount is never committed to the layer, so dropping the flag does not increase the image size.

Running as non-root

When the step runs under a non-root USER, set uid and gid on the mount to match that user. Point target at that user's home, for example /home/app/.cache/pip, not /root.

Mistakes here usually don't break the build, so they are easy to miss. If the target path is wrong, the tool writes its cache to its default location, which ends up in the layer, and the mount is never used. If the ownership is wrong, pip prints a warning, turns its cache off and installs from the network. Either way the build succeeds but gets no faster.

Copy# syntax=docker/dockerfile:1

# npm: cache the tarball store, not node_modules
FROM node:20-slim AS web
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
COPY . .

# Maven: resolve from pom.xml alone, share the repo with the package step
FROM maven:3-eclipse-temurin-17 AS api
WORKDIR /src
COPY pom.xml .
RUN --mount=type=cache,target=/root/.m2/repository \
    mvn -B dependency:go-offline
COPY src ./src
# Same mount, so plugins and anything go-offline missed are reused next time
RUN --mount=type=cache,target=/root/.m2/repository \
    mvn -B package -DskipTests

# pip as non-root: uid/gid and target must match the runtime user
FROM python:3.12-slim AS worker
RUN useradd --create-home --uid 10001 app
USER app
WORKDIR /home/app
COPY --chown=app:app requirements.txt .
# No --no-cache-dir: the cache stays in the mount and never enters the layer.
# A wrong target or uid does not fail the build. pip just skips the cache.
RUN --mount=type=cache,target=/home/app/.cache/pip,uid=10001,gid=10001 \
    pip install --user -r requirements.txt

One Dockerfile with three stages that shows BuildKit cache mounts for npm ci, for Maven resolution and packaging, and for pip running as a non-root user with matching uid/gid and home path.

Where the cache lives in CI and how to make it survive

Cache mounts are stored in the BuildKit builder's local state, next to its own layer store. They are not part of any image, and the registry and inline cache exporters never write them out. On an ephemeral runner the builder is created at the start of the job and discarded at the end, so every job starts with empty mounts.

To keep the mounts between runs, you need one of these:

• a long-lived builder that many jobs connect to,
• a self-hosted runner whose builder state survives between jobs, or
• a deliberate step that saves the builder's state directory at the end of a job and restores it at the start of the next. The cost is archive time and storage, so measure it against the downloads it saves.

Copy CI runner                                  Registry
 ┌──────────────────────────────┐           ┌────────────────────┐
 │ BuildKit builder state       │  export/  │ exported layer     │
 │  ├─ layer store              │◄─import──►│ cache (install     │
 │  └─ cache mounts (npm, m2,   │           │ layer, base, etc.) │
 │     pip) ← stays local       │           └────────────────────┘
 └──────────────────────────────┘

The two mechanisms work together. Because the manifest is copied before the source, the exported layer cache can skip the install step entirely. The mount only matters when that step has to run again, and then it supplies every package except the ones that changed.

A cold runner with an unchanged lockfile still gets a hit on the install layer from the registry. The cost of a cold runner only shows up when dependencies change.

Copy CI runner                                      Registry
 ┌───────────────────────────────────┐          ┌──────────────────────┐
 │ docker build                      │          │ exported layer cache │
 │   │                               │  export  │  ├─ base image       │
 │   ▼                               │  import  │  ├─ COPY lockfile    │
 │ BuildKit builder state            │          │  └─ RUN install      │
 │  ├─ layer store ◄─────────────────┼─────────►│                      │
 │  └─ cache mounts                  │          └──────────────────────┘
 │      npm cache, ~/.m2, pip        │
 │      stay local, never           │
 │      exported                     │
 └──────────────┬────────────────────┘
                ▲ restore at job start
                ▼ save at job end
 ┌──────────────┴────────────────────┐
 │ archived builder state            │
 │ (only if you save it yourself)    │
 └───────────────────────────────────┘

 Source change only : install layer hit (registry); mount unused; no fetch
 Lockfile, warm     : install reruns; mount hit; fetch changed packages only
 Lockfile, cold     : install reruns; mount empty; fetch full dependency set

Layer cache moves between the builder and the registry, but cache mounts stay in the builder's local state, so they survive only on a long-lived builder or when you save and restore that state yourself.

Verifying the cache works and estimating the payoff

Confirming cache hits

Change one dependency in the lockfile, then build twice with --progress=plain so BuildKit prints full RUN output instead of collapsing it. The install layer reruns both times because the lockfile changed, so the package manager's log is where you see whether the cache mount helped. For Maven, the second build should show few or no Downloading from lines. For pip, look for Using cached next to each package. For npm, raise the log level and compare how many network fetches each run makes. If the second build re-downloads everything, the mount target does not match the tool's real cache directory, or the builder was replaced between runs.

Next, run docker buildx du (or docker system df for the default builder) and check the cache-mount entries. They should grow after the first build and stay roughly the same size after the second.

Estimating the payoff

This is an illustrative estimate, not a measurement. Assume 800 packages averaging 200 KB, so about 160 MB per full install. Assume 10 MB/s effective throughput, which is lower than raw bandwidth because registry round-trips add latency. A full download then takes about 16 seconds. With a warm cache, a single-package bump downloads almost nothing. At 100 builds a day, that saves roughly 27 runner-minutes a day, and each developer waits about 16 seconds less per build.

The savings apply only to download time. Compiling sources, building native extensions and running postinstall scripts take the same time with or without the cache mount.

Copy#!/usr/bin/env bash
# Check that the pip cache mount is reused, then reproduce the illustrative payoff arithmetic.
# Assumes the Dockerfile installs with:
#   RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
set -euo pipefail

# Build 1: populates the cache mount. --progress=plain prints the full RUN output.
docker buildx build --progress=plain -t app:cachetest . 2>&1 | tee build1.log

# Bump one pin so the install layer is invalidated and pip runs again.
sed -i 's/^requests==.*/requests==2.32.3/' requirements.txt

# Build 2: the install step reruns, so pip's log shows whether the mount was used.
docker buildx build --progress=plain -t app:cachetest . 2>&1 | tee build2.log

# pip prints "Downloading" for network fetches and "Using cached" for cache hits.
# The changed package should be the only new download in build 2.
for f in build1.log build2.log; do
  echo "$f: downloads=$(grep -c 'Downloading' "$f" || true) cached=$(grep -c 'Using cached' "$f" || true)"
done

# Inspect builder disk usage. Cache-mount entries should grow after build 1
# and stay about the same size after build 2.
docker buildx du --verbose | grep -i -B2 -A6 'exec.cachemount' || docker buildx du

# Illustrative estimate, not a measurement. All inputs are assumptions.
packages=800          # dependency count
avg_kb=200            # average package size in KB
throughput_mb_s=10    # effective registry throughput in MB/s
builds_per_day=100

total_mb=$(( packages * avg_kb / 1000 ))               # 160 MB per full install
secs_per_build=$(( total_mb / throughput_mb_s ))       # 16 s of download time
saved_min_day=$(( secs_per_build * builds_per_day / 60 ))  # ~26 runner-minutes/day (integer)
echo "Download time per cold build: ${secs_per_build}s; saved per day: ~${saved_min_day} min"
echo "Compile, native-extension and postinstall time are not included and do not change."

A shell script that builds twice with plain progress output around a lockfile bump, counts pip download lines versus cache-hit lines, inspects cache-mount usage with docker buildx du, and works through the illustrative download-time estimate.

When to avoid cache mounts and what can go wrong

Where cache mounts fit poorly

Ephemeral runners. Hosted runners that start with a fresh builder get nothing from a cache mount. Exporting a registry or inline cache does not help, because those exports hold image layers and never include mount contents. To keep mount data across runs you need a persistent builder or a separate save-and-restore step.

Stale or corrupted caches. A cache with a truncated tarball or a mismatched artifact can stay in place for many builds without any error. Correctness has to come from the lockfile and the package manager's integrity checks, such as npm ci with lockfile hashes or pip's --require-hashes. Do not rely on cache freshness.

Concurrent writers. The default sharing=shared lets parallel builds write to the same directory at once. Use sharing=locked for package managers that are not safe under concurrent writes.

Non-root users. If the RUN step runs as a different uid than the owner of the mount, or the tool's cache path differs from the mounted target, the install still succeeds. It just never uses the cache. Set uid/gid on the mount and check that the target path is correct.

Compute-bound installs. If most of the time goes to compiling native extensions, the mount saves almost nothing.

Tooling compatibility. The --mount syntax breaks the legacy builder, and it also breaks any scanner or build tool that parses Dockerfiles without BuildKit semantics.

Masked network problems. A warm local cache can hide an expired private-registry token or a removed package version. You only find out when CI runs cold.

Housekeeping

BuildKit's garbage collector evicts cache under storage pressure, so set limits on long-lived builders. docker builder prune clears the cache manually. Run a scheduled build with --no-cache to confirm the image still builds correctly from an empty cache.

Alternatives and how they compare

These options are not mutually exclusive. Most of them address a different cost than cache mounts do, so they often work best in combination.

The key difference is granularity. Layer-based approaches are all-or-nothing: changing one line in a lockfile invalidates the layer, and every package is installed again. A cache mount keeps the package manager's own cache, so only the changed packages are downloaded. However, cache mount contents are not included in exported layer cache, so on ephemeral runners you need to persist them separately.

Pick a combination based on your setup:

• Small teams on hosted runners: use careful ordering plus registry cache. Add cache mounts only if lockfiles change often and you can persist the builder state.
• Platform teams serving many repositories: run a shared proxy. It benefits every pipeline without per-repo work, and cache mounts layer on top of it.
• Slow-moving services with heavy native dependencies: prebuilt base images justify their upkeep here.
• Air-gapped or strict-provenance environments: use vendoring or a controlled mirror.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to docker buildkit cache mount, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.