Why do Docker builds miss the cache on CI runners?

If your CI runs on ephemeral or autoscaled runners and your builds spend minutes reinstalling dependencies that haven't changed, this article is for you. The cause is usually not a badly ordered Dockerfile. The runner starts with nothing.

On a long-lived build host, the layer cache from the previous build is still on disk, so unchanged steps are skipped. An ephemeral runner is a fresh VM or container with empty builder state. Every job starts cold, so every RUN step executes again, including the package install that dominates build time.

The classic workaround was to pull the last pushed image and pass it as a cache source. This is fragile with multi-stage builds. The pushed image contains only the final stage's layers. The builder stage, where compilers and dependency downloads live, never reaches the registry, so it can't be reused. You can push each stage separately, but that adds bookkeeping that breaks easily.

BuildKit is the build backend that fixes this. It tracks cache independently of the final image and can export and import it as a separate artifact. In practice you work with it through buildx builder instances, and the instance determines where cache lives and what you can do with it:

  • docker driver (the default): uses the BuildKit built into the Docker daemon. It is convenient, but it supports fewer cache export options.
  • docker-container driver: runs BuildKit in its own container. It supports exporting cache to external backends such as a registry, which is what you need when runners don't share disks.
 CI job N (ephemeral runner)           CI job N+1 (fresh runner)
+------------------------------+       +------------------------------+
| docker buildx build          |       | docker buildx build          |
|   |                          |       |   |                          |
|   v                          |       |   v                          |
| +--------------------------+ |       | +--------------------------+ |
| | buildx builder instance  | |       | | buildx builder instance  | |
| |  driver: docker-container| |       | |  driver: docker-container| |
| |  BuildKit (own container)| |       | |  local cache: EMPTY      | |
| +--------------------------+ |       | +--------------------------+ |
+------------|-----------------+       +-------------^----------------+
             | export cache                          | import cache
             v                                       |
      +---------------------------------------------------------+
      | Registry (shared by all runners)                        |
      | app:latest  image: final-stage layers only              |
      | app:cache   cache export: builder + final stages        |
      +---------------------------------------------------------+

 Runner deleted after job -> local layer cache is deleted with it
 Classic: pull app:latest as cache -> builder stage missing -> cold RUNs
 Default docker driver: BuildKit in dockerd, fewer cache export options

Ephemeral runners lose local builder state after every job, so a docker-container buildx builder exports cache for all stages to a shared registry, while a pulled final image carries only the last stage's layers.

How does Docker layer cache invalidation work?

BuildKit decides whether to reuse a step by computing a cache key from the step's parent and its definition. Once one step misses, every step after it misses too, because each key depends on its parent. Order matters more than anything else.

Different instructions are checked in different ways:

  • RUN: only the command string is compared. BuildKit does not re-check what the command fetched, so RUN apt-get update stays cached even after upstream packages change.
  • COPY and ADD: BuildKit checksums the contents of the copied files. Editing any file in the copied set invalidates that step, even a README or a test file. File modification times do not count.
  • Build args: changing an ARG value invalidates the cache at the first instruction that uses it. Passing a commit SHA or timestamp as an arg near the top of the file defeats caching on every build.

The standard fix is to copy only the dependency manifests first, install dependencies, and then copy the source. Examples are package.json with its lockfile, go.mod with go.sum, or requirements.txt. Manifests change much less often than code, so the expensive install step stays cached across most commits. A good .dockerignore also helps by keeping .git, build output and local artifacts out of the checksum.

No cache backend fixes a Dockerfile that runs COPY . . early. Every commit changes the checksum, so everything after it rebuilds. Exporting that cache to a registry just moves layers that will never match again. This is the most common reason teams see little gain from registry caching.

# syntax=docker/dockerfile:1
FROM node:20-slim AS build
WORKDIR /app

# 1. Copy only the dependency manifests. This step's cache key depends
#    on the checksum of these two files, not on the rest of the repo.
COPY package.json package-lock.json ./

# 2. Install dependencies. Stays cached until a manifest changes,
#    so ordinary code commits skip this expensive step.
RUN npm ci

# 3. Copy the source last. Any code edit invalidates from here down,
#    but the install layer above is still reused.
COPY . .
RUN npm run build

# Declare per-build values as late as possible. Changing GIT_SHA
# only invalidates instructions after this point that use it.
ARG GIT_SHA=unknown
RUN echo "$GIT_SHA" > /app/dist/VERSION

# Anti-pattern for comparison (do not do this):
#   COPY . .
#   RUN npm ci
# Every commit changes the checksum of COPY . ., so npm ci reruns
# on every build no matter which cache backend you export to.

# Same idea for other ecosystems:
#   Go:     COPY go.mod go.sum ./   then RUN go mod download
#   Python: COPY requirements.txt ./ then RUN pip install -r requirements.txt

A Dockerfile ordered so the dependency install is keyed only on the manifest files, with source code copied and per-commit build args declared afterward to keep the expensive layers cached.

How do you export and import cache with --cache-to and --cache-from registry?

Once the Dockerfile orders layers sensibly, the next problem is getting that cache onto a fresh runner. With docker buildx build, --cache-to type=registry,ref=<repo>:buildcache pushes cache after the build, and --cache-from type=registry,ref=<repo>:buildcache pulls it before the next one. The runner can start empty and still reuse layers from earlier builds.

Store the cache under its own reference, not the tag you deploy. This lets you set retention for cache artifacts separately from release images, so you can prune cache aggressively without touching anything production depends on. It also keeps cache metadata out of the images you ship, and it lets you overwrite the cache ref on every build without moving a release tag.

The mode parameter controls what gets exported:

Option What is cached Push cost Good fit
mode=min (default) Layers of the final image only Smaller Single-stage builds
mode=max Layers from all stages, including intermediate builder stages Larger Multi-stage builds with expensive compiles or dependency installs
type=inline Final-image layers, embedded in the image itself No separate push Simple setups that don't need a separate cache ref

With mode=min, a multi-stage build loses its builder stages. The next run reinstalls every dependency even though the final image hits the cache. mode=max costs more upload time and storage, but it pays off when a builder stage takes minutes to rebuild.

Inline cache, set with type=inline or the BUILDKIT_INLINE_CACHE=1 build arg, is the simplest option. It only covers final-image layers, so it has the same limitation as mode=min.

Exporter support varies by registry and by builder driver. Some drivers can't export registry cache, and some registries reject the cache manifest format. Run a test export against your own setup before you depend on it.

#!/usr/bin/env bash
set -euo pipefail

IMAGE=registry.example.com/team/api
TAG="${GIT_SHA:-dev}"

# The default "docker" driver may not support registry cache export,
# so create a docker-container builder for this job.
docker buildx create --name ci-builder --driver docker-container --use

# Option 1: registry cache under its own ref (:buildcache), separate from
# the deployable tag. mode=max also exports intermediate builder stages,
# so expensive dependency installs survive across fresh runners.
docker buildx build \
  --cache-from type=registry,ref="$IMAGE:buildcache" \
  --cache-to type=registry,ref="$IMAGE:buildcache",mode=max \
  --tag "$IMAGE:$TAG" \
  --push \
  .

# Option 2: inline cache. Cache metadata is embedded in the pushed image,
# there is no separate cache ref, and only final-image layers are covered
# (same limitation as mode=min). Pull from the previously pushed image.
# docker buildx build \
#   --cache-from type=registry,ref="$IMAGE:latest" \
#   --cache-to type=inline \
#   --tag "$IMAGE:latest" \
#   --push \
#   .

# Check your registry and builder driver support the exporter you choose
# before relying on it in CI.

A CI build script that imports and exports BuildKit cache through a dedicated registry ref with mode=max. The inline-cache alternative is shown commented out for comparison.

What are BuildKit cache mounts and why don't they persist across runners?

Layer cache stores the result of a step. A cache mount stores a directory that persists between builds without becoming part of any layer. With RUN --mount=type=cache,target=<path>, a package manager can reuse its download and compile caches even when the step itself has to rerun, for example after the lockfile changes.

Ecosystem Typical target Notes
apt /var/cache/apt, /var/lib/apt Use sharing=locked
pip /root/.cache/pip Path depends on the build user
npm /root/.npm Works with npm ci
Go /go/pkg/mod, /root/.cache/go-build Module cache plus GOCACHE

The sharing option controls concurrent access. shared is the default and lets parallel builds read and write the mount at the same time. private gives each concurrent writer its own copy. locked makes writers wait for each other, which apt and dpkg require. Debian and Ubuntu base images also ship /etc/apt/apt.conf.d/docker-clean, which deletes downloaded packages after every install. Remove it in an earlier step, or the apt mount stays empty.

Cache mounts do not persist across runners. They live in the builder's local storage, and --cache-to does not export them. On a fresh ephemeral runner they start empty and help only within that job. You can work around this in two ways:

  • Use a persistent or self-hosted builder.
  • Copy the mount directory out of the builder and back in yourself.

Both approaches add maintenance work. Mount contents are also not part of the cache key, so stale or corrupted files can change build output without invalidating anything. Builds that use cache mounts are therefore less hermetic.

# syntax=docker/dockerfile:1

FROM python:3.12-slim AS py
# Debian-based images ship docker-clean, which deletes downloaded .debs.
# Remove it and keep packages so the apt cache mount actually fills up.
RUN rm -f /etc/apt/apt.conf.d/docker-clean; \
    echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
# apt/dpkg cannot tolerate concurrent writers: use sharing=locked.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends build-essential
WORKDIR /app
COPY requirements.txt .
# Running as root, so pip's cache is /root/.cache/pip.
# No --no-cache-dir here, because the cache is what we want to keep.
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

FROM node:20-slim AS web
WORKDIR /web
COPY package.json package-lock.json ./
# npm ci still wipes node_modules but reuses tarballs from ~/.npm.
RUN --mount=type=cache,target=/root/.npm \
    npm ci

FROM golang:1.22 AS go
WORKDIR /src
COPY go.mod go.sum ./
# Module cache (GOPATH=/go in this image) survives go.mod changes.
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
COPY . .
# GOCACHE for root defaults to /root/.cache/go-build, which enables incremental compiles.
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -o /out/app ./cmd/app

# None of these mount directories end up in image layers or in --cache-to exports.
# On a fresh ephemeral runner every mount starts empty.

A multi-stage Dockerfile that uses cache mounts for apt (with sharing=locked and docker-clean removed), pip, npm, and Go module and build caches. The mounts persist only in the local builder's storage.

How should you choose cache refs for branches and parallel jobs?

The pattern that works for most teams is to read from two refs and write to one. Pass --cache-from twice: once for the branch's own cache ref and once for the main branch's ref. Pass --cache-to only for the branch ref. A new feature branch has no cache of its own yet, so it falls back to main's cache and starts warm. Its results never overwrite main's cache. Only builds on main write to main's ref.

Writing to a single ref from concurrent jobs is a race. A cache ref is a tag, and the last push wins. If two jobs on the same branch finish close together, the later export replaces the earlier one completely. The cache is not merged, so any layers only the first job produced are dropped from the ref. Avoid this by serializing builds per branch, or by giving each matrix target or platform its own ref, such as a suffix per target.

Per-branch refs cost storage. The registry deduplicates identical blobs, but each branch adds its own unique layers, and with mode=max that includes intermediate stages. Nothing removes these automatically. Delete the cache tag when a branch is merged or deleted. Add a registry retention policy that expires untouched cache tags, then make sure the registry's garbage collection actually reclaims the unreferenced blobs.

Put the cache in a separate repository so you can scope permissions to it. CI jobs need push rights only to that repository. Builds from untrusted sources, such as fork pull requests, should get read-only credentials. Anyone who can write to main's cache ref can poison the layers that later trusted builds reuse.

How can you verify the build cache is saving time?

Run the build with --progress=plain so CI logs record each step on its own line instead of a redrawn terminal view. Each step is numbered. A step served from cache prints CACHED, and an executed step ends with DONE and its duration. The cache transfer steps, importing the cache manifest and exporting cache to the registry, also print their own durations. This matters because the true cost of caching is often hidden in those transfer steps rather than in the build steps.

To measure, trigger a cold run with no cache import, then a warm run on an unchanged commit, then a warm run after a typical change, such as a source edit with untouched dependency files. Read the step timings from the logs, not the total job time, which includes runner startup and queueing.

Illustrative estimate (assumed numbers, not a benchmark)

Net saving per build = cold build − (warm build + cache import + cache export)

  • Assumed cold build: 9 min.
  • mode=min: warm build 2 min, import 0.5 min, export 0.5 min. Saving = 9 − 3 = 6 min.
  • mode=max on a multi-stage build: warm build 1.5 min, import 1.5 min, export 3 min. Saving = 9 − 6 = 3 min.
  • mode=max when builder-stage layers change on most commits: warm build 7 min plus 4.5 min of transfer. Net = −2.5 min, so each build is slower than building with no cache.

Multiply the per-build saving by your monthly build count to estimate both engineering wait time and CI minutes.

When should you avoid heavy caching, and what are the alternatives?

A warm cache is also a stale cache. If apt-get install or npm ci hits the cache, that layer keeps whatever versions existed when it was first built, and patched OS packages never land. Bust the cache on purpose. Run a scheduled rebuild with --no-cache. Alternatively, pass a changing build arg, such as a date stamp, just before the install step so only the layers after it rebuild. You can also pull a refreshed base image, because its new digest invalidates every layer downstream.

Never write secrets into a layer or a cache mount. Layers are pushed to the registry along with exported cache, and cache mount contents stay on the builder for the next job. Use RUN --mount=type=secret instead. It exposes the file only during that step and keeps it out of the image and the cache.

Skip heavy caching when the image is small, when installs are cheap, or when exporting and importing cache takes longer than a clean rebuild, as in the negative result above.

Option Strengths Trade-offs
Registry cache Works on any runner. Can export intermediate layers. Upload and download time, plus registry storage.
Inline cache No separate cache ref. Only covers layers in the final image, so multi-stage builds lose most of the benefit.
CI-provider cache backend Often faster because storage sits close to the runner. Ties you to that provider, and size limits apply.
Persistent builder Local disk cache and cache mounts survive between jobs. You run, scale and clean up that builder yourself.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to docker buildkit cache ci, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.