How does BuildKit decide a layer cache hit?
BuildKit turns each Dockerfile instruction into a node in a build graph and computes a cache key for it. The key combines the parent step's key, the instruction's definition (the command string, build args in scope, environment), and, for COPY and ADD, a checksum of the contents of the files being copied. File timestamps do not count; content does. If a step's key matches a stored result, BuildKit reuses the snapshot instead of executing the step.
Because each key includes its parent's key, a miss at one step invalidates every step after it. This is why instruction order matters so much. If you copy the whole source tree before installing dependencies, any edit to any file changes the COPY checksum and forces a full reinstall. The standard fix is to copy only the dependency manifests and lockfiles (package.json and package-lock.json, go.mod and go.sum, requirements.txt), run the install, and then copy the rest of the source. Routine code changes then invalidate only the final layers, and the dependency layer survives until the lockfile changes.
The cache key for a RUN step depends on the command text, not on what the command produces. A cached RUN apt-get update && apt-get install ... will keep returning the package versions from the day it first ran, including versions that have since received security fixes. Floating base tags like :latest or :3 cause a similar problem when the builder reuses a locally stored copy instead of pulling the current one. To get patched packages, pin base images by digest and update them deliberately, or run scheduled builds with --pull --no-cache.
FROM node:20-bookworm-slim@sha256:<pinned-digest>
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
CMD ["node", "server.js"]
A Dockerfile that copies the lockfile before the source so dependency installs stay cached across code edits, pins the base image by digest, and notes that a scheduled --pull --no-cache build is needed to pick up patched packages.
How does BuildKit caching work in a CI pipeline?
A cached CI build relies on two separate mechanisms. Each one stores something different.
- The builder starts.
docker buildxcreates or selects a builder. On an ephemeral runner, it starts with no layer cache and no cache-mount contents. - Cache is imported. Each
--cache-fromsource is read. For a registry source, BuildKit fetches the cache manifest first. It downloads layer blobs only when a matching step needs them. - The graph is solved. BuildKit computes each step's cache key from its parent key, its instruction definition, and, for
COPYandADD, the file contents. It compares each key against the local state and the imported metadata. Matching steps reuse the stored snapshot. The first step that misses, and every step after it, executes. - Cache mounts attach to executing steps. When a
RUNstep with--mount=type=cacheruns, BuildKit attaches the named directory from the builder's local state, so a package manager or compiler can reuse earlier downloads and outputs. Mounts do not affect cache keys. They only make steps that do run faster. - Cache is exported.
--cache-towrites the layers and metadata to the configured destination. Withmode=min, only layers in the final image are written. Withmode=max, intermediate stages are included too. Cache-mount contents are never part of this export. - The image is pushed and the runner is discarded. Only the image and the exported cache survive. Anything in the builder's local state, including cache mounts, disappears unless the builder is persistent or a separate step saved it.
The sections below cover each stage in detail. They explain why local state is lost, how to configure the export and import, how cache mounts behave, how to scope caches safely, and how to confirm that hits are happening.
Why does Docker build cache disappear in CI?
On a hosted runner, every job starts on a fresh virtual machine. The BuildKit daemon, or the builder container that docker buildx creates, starts with an empty content store and no layer metadata. When the job ends, the machine and everything the build produced are thrown away. Local caching works well on a laptop because the daemon survives between builds. In CI it only helps on persistent or self-hosted builders that keep their state from one job to the next.
That missing state causes two separate problems:
- Cross-run layer reuse. The next job has no record of which instructions already ran or what they produced. Registry caching fixes this. BuildKit exports cache metadata and layers to an image reference, and the next run imports them.
- Package-manager and compiler caches. Cache mounts keep directories such as a module download cache or a compiler's object cache. Those contents live only in the builder's local state. Registry cache export does not include them, so on an ephemeral runner they start empty on every run.
┌─────────────── ephemeral runner (discarded after job) ───────────────┐
│ │
│ ┌──────────── builder local state ────────────┐ │
│ │ layer cache cache mounts │ │
│ └──────┬───────────────────▲──────────────────┘ │
│ │ │ (no path out by default) │
└──────────┼───────────────────┼───────────────────────────────────────┘
cache-to │ ▲ cache-from │ save/restore directory (workaround)
▼ │ │
┌──────────────────┐ ┌─────┴──────────────┐
│ registry cache │ │ CI system's cache │
│ ref (layers) │ │ (files on runner) │
└──────────────────┘ └────────────────────┘
Teams on hosted ephemeral runners gain the most from fixing this, especially when dependency installs or compilation take up most of the build time. For those teams, every cache miss repeats the slowest part of the build.
┌──────────────── ephemeral runner (discarded after job) ────────────────┐
│ │
│ ┌────────────────────── builder local state ───────────────────────┐ │
│ │ ┌──────────────────┐ ┌──────────────────────────┐ │ │
│ │ │ layer cache │ │ cache mounts │ │ │
│ │ │ (instructions + │ │ (module downloads, │ │ │
│ │ │ layer results) │ │ compiler objects) │ │ │
│ │ └──────┬─────▲─────┘ └─────────────▲────────────┘ │ │
│ └──────────┼─────┼─────────────────────────────┼───────────────────┘ │
│ │ │ │ │
└─────────────┼─────┼─────────────────────────────┼──────────────────────┘
cache-to │ │ cache-from │ save ▼ / ▲ restore
(export) │ │ (import) │ (workaround only)
▼ │ ▼
┌────────────────┴───┐ ┌───────────────────────────┐
│ registry cache ref │ │ CI system's cache │
│ layers + metadata │ │ arbitrary directories │
│ persists across │ │ persists across runs │
│ runs │ │ outside BuildKit │
└────────────────────┘ └───────────────────────────┘
Registry cache export does not include cache-mount contents. Without
the workaround, cache mounts start empty on every ephemeral run.
On an ephemeral runner, layer cache reaches the next job only through cache-to and cache-from against a registry cache ref. Cache-mount contents have no default path out of the builder and survive only if a separate step saves and restores them through the CI system's cache.
Who should use registry caching and cache mounts?
These techniques are worth the setup cost when the expensive part of a build is repeatable work whose inputs rarely change. The strongest fits are:
- Teams on hosted, ephemeral runners. Every job starts cold, so without an exported cache, every build repeats every step. Registry caching is the main way to carry layer cache between jobs.
- Multi-stage builds with a heavy builder stage. Compiled languages such as Go, Rust, Java, and C/C++, along with Node or Python builds that compile native extensions, spend most of their time in a stage that never reaches the final image.
mode=maxregistry caching is what lets that stage hit. - Projects with large dependency trees and stable lockfiles. If the lockfile changes on only a small share of commits, the dependency layer hits on most builds.
- Repositories that build many images or run on every pull request. Small savings per build add up across many builds per day.
- Teams with persistent or self-hosted builders. Local layer cache already survives here, so cache mounts become the main lever. They speed up the steps that do rerun after a lockfile change.
The engineers who set this up are usually the people who own the CI pipeline and the Dockerfiles: platform, build, or DevOps engineers, or application developers on smaller teams. Configuring it requires registry push credentials, permission to change builder setup, and enough Dockerfile knowledge to order instructions for cache hits.
It is a weaker fit for small images that build quickly from scratch, for builds whose dependencies change on most commits, and for release builds where a clean, uncached rebuild is a requirement. The final section covers these cases in more detail.
How do you set up registry caching with cache-from and cache-to?
Registry caching moves the cache out of the runner and into storage that outlives it. With docker buildx build, you export cache using --cache-to type=registry,ref=<registry>/app:buildcache and import it on the next run with --cache-from type=registry,ref=<registry>/app:buildcache. The dedicated :buildcache reference is a cache manifest, not a runnable image. Keeping it separate from release tags means cache churn never touches what you deploy, and the cache stays in the same registry, under the same credentials, as the image.
You can also point --cache-from at a previously pushed image such as app:latest. This only produces hits if that image was built with inline cache metadata. A plain pushed image does not carry the information BuildKit needs to match steps.
| Approach | Where metadata lives | Modes | Multi-stage intermediates cached | Cost |
|---|---|---|---|---|
| Inline cache | Inside the pushed image | min only |
No, only layers in the final image | Little extra storage, no separate push |
| Separate registry export | Its own reference, e.g. :buildcache |
min or max |
Yes, with mode=max |
More storage, longer export step |
mode=max is usually the right choice for multi-stage builds. With inline or min caching, the builder stage that compiles or installs dependencies is not cached, and that stage is often the expensive part of the build.
Which backends and options are available depends on your BuildKit version, your buildx version, and the driver. The default docker driver does not support some cache exporters, so you may need a builder created with the docker-container driver. Check the documentation for the tool versions you actually run in CI.
#!/usr/bin/env bash
set -euo pipefail
REGISTRY="registry.example.com/team"
IMAGE="${REGISTRY}/app"
TAG="${CI_COMMIT_SHA:-dev}"
docker buildx create --name ci-builder --driver docker-container --use 2>/dev/null \
|| docker buildx use ci-builder
docker buildx build \
--cache-from "type=registry,ref=${IMAGE}:buildcache" \
--cache-to "type=registry,ref=${IMAGE}:buildcache,mode=max" \
--tag "${IMAGE}:${TAG}" \
--push \
.
A CI build script that creates a docker-container builder and imports and exports a mode=max registry cache under a dedicated :buildcache tag, with inline cache shown as the commented-out alternative.
How do BuildKit cache mounts work, and why are they empty in CI?
A cache mount (RUN --mount=type=cache,target=...) attaches a directory that BuildKit keeps outside the layer graph. When an instruction's inputs change and the layer is rebuilt, the mount still holds whatever the previous run left there, so the package manager only fetches what is new. Common targets are /root/.cache/pip for pip, /root/.npm for npm, /go/pkg/mod and /root/.cache/go-build for Go modules and compiled packages, /var/cache/apt and /var/lib/apt for apt, and Cargo's registry directory plus the target directory for Rust builds. The mount's contents are never written into the image, so they add no size and leave no files in the final layers.
The catch in CI is that the mount lives in the builder's local state, and --cache-to does not export it. On an ephemeral runner, every job gets a fresh builder, and every mount starts empty. You have two ways to fix this:
- Keep the builder. Use a persistent or self-hosted builder whose state survives between jobs.
- Move the contents yourself. Copy the mount directories out after the build, store them with the CI system's own cache, and seed them back in before the next build.
Several options affect correctness:
idnames the cache. It defaults to the target path. Set it explicitly to share one cache across stages or Dockerfiles, or change it to deliberately invalidate a cache.sharing=lockedserializes access between concurrent builds. Use it for apt, which cannot tolerate concurrent writers.uid,gidandmodeset ownership and permissions on the mount. A non-rootUSERneeds these, or it cannot write to the cache directory.
FROM python:3.12-slim
RUN rm -f /etc/apt/apt.conf.d/docker-clean && \
echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update && \
apt-get install -y --no-install-recommends build-essential
RUN groupadd --gid 1000 app && \
useradd --uid 1000 --gid 1000 --create-home app
USER app
WORKDIR /home/app
ENV PATH=/home/app/.local/bin:$PATH
COPY --chown=app:app requirements.txt .
RUN --mount=type=cache,id=pip-app-v1,target=/home/app/.cache/pip,uid=1000,gid=1000 \
pip install --user -r requirements.txt
COPY --chown=app:app . .
CMD ["python", "-m", "app"]
A Dockerfile that uses sharing=locked for apt's cache mounts, then gives a non-root user its own pip cache mount with an explicit id and matching uid/gid.
How should you scope build caches across branches and handle secrets?
Read broadly, write narrowly
Pass more than one --cache-from source. BuildKit consults all of them, so a feature branch build can list its own cache ref first and main's ref second. The first build on a new branch then starts from main's layers instead of running cold, and later builds pick up the branch's own work. Write with --cache-to only to the branch's ref, such as cache:branch-name, and let only builds on main write cache:main. If feature branches export to main's ref, an experimental dependency change can replace the cache every other branch relies on.
Writing to the same ref from parallel jobs is unsafe in its own way. A cache export replaces the ref's manifest instead of merging into it, so the last job to finish wins. Layers produced by the other jobs are no longer referenced, and the next build misses on steps it should have hit. These misses are intermittent and hard to reproduce. Give concurrent jobs separate refs, or serialize writes to shared ones.
Pull requests from forks usually run without registry credentials, because most CI systems withhold secrets from untrusted code. Those builds can often still read a public cache, but they cannot export one, so expect them to stay cold.
Keep secrets out of layers
Never pass tokens through ARG or ENV. Build arguments used in a RUN step can show up in image history. Any file a step writes, even one deleted in a later step, remains in that layer and in the exported cache. Arguments are also part of the cache key, so a rotating token invalidates every step after it. RUN --mount=type=secret exposes the secret as a file for that single step only. It is never written to a layer, and its contents do not affect cache hits.
How can you verify that CI build caching actually works?
Run the build with --progress=plain in CI. The default TTY progress display collapses finished steps, so logs lose detail. Plain output prints every step with its instruction, and steps served from cache are marked CACHED. A healthy warm run should show CACHED on the base image, system packages, and dependency install steps. The first non-cached step should be the one whose inputs actually changed, usually COPY of application source. If dependency install reruns on a commit that only touched source files, the cache key changed upstream. Check the order of COPY instructions and look for a --build-arg whose value changes on every run.
Plain output also prints per-step durations. Compare a cold run, made with --no-cache or a fresh cache ref, against a warm run on the same commit. Also time the cache import and export phases, because they are real costs that a step-level view hides.
Illustrative estimate. Assumptions: dependency install takes 4 min, the remaining steps take 2 min, cache import takes 0.5 min, export with mode=max takes 0.75 min, there are 40 builds per day, and the dependency step hits on 80% of builds.
| Scenario | Import | Deps | Other steps | Export | Total |
|---|---|---|---|---|---|
| No cache configured | 0 | 4 | 2 | 0 | 6.0 min |
| Cache configured, hit | 0.5 | 0 | 2 | 0.75 | 3.25 min |
| Cache configured, miss | 0.5 | 4 | 2 | 0.75 | 7.25 min |
The expected time per build is 0.8 × 3.25 + 0.2 × 7.25 ≈ 4.05 min. That saves about 2 minutes per build, or roughly 78 CI minutes a day. Most of that time is also time engineers spend waiting on feedback.
Caching breaks even at a hit rate of about 31%. Below that rate, import and export overhead costs more than the cache saves. This happens with builds that change dependencies on most commits, and with small images where the steps finish faster than the cache can be transferred.
How can BuildKit caching save engineering time?
The direct saving is shorter build time. The bigger effect is on how long engineers wait, and how often.
- Shorter feedback loops. When a commit only touches application source, a warm cache skips the base image, system package, and dependency steps, so the build time is mostly the work the change actually requires. Pull request checks finish sooner, and developers spend less time waiting or switching to other tasks.
- Cheaper reruns. Flaky tests, retried jobs, and small review fixes are common in CI. With a warm cache, rerunning a pipeline repeats only the steps whose inputs changed.
- Faster dependency updates. When a lockfile changes, the dependency layer misses, but cache mounts (on persistent builders, or with save/restore) keep earlier downloads and compiler outputs. The package manager fetches only what is new, and compilers can reuse unchanged artifacts.
- Fewer network fetches per build. Cached layers and populated mounts mean fewer downloads from package registries and base-image registries. This reduces how often a temporary upstream outage or rate limit breaks a build.
- Lower runner usage. Fewer executed steps means fewer billable or queued runner minutes. On shared runner pools, it also means shorter queues for everyone else.
Use the illustrative estimate in the previous section to model your own pipeline. Replace the assumed step times, overhead, build count, and hit rate with values from --progress=plain logs from your builds, then multiply the per-build saving by daily build volume. Count the costs too. Engineers spend time ordering Dockerfile instructions, scoping cache refs, managing cache storage, and occasionally debugging a stale cache. The saving is real only when the measured hit rate stays comfortably above the break-even point for your build.
When should you avoid these caching approaches, and what are the alternatives?
Small images with a handful of fast steps often build quicker from scratch than they can pull and push cache metadata and blobs. Time a cold build against a warm one before adding cache flags, and keep them only if the gap covers the export.
mode=max stores layers for every intermediate stage, including builder stages that never reach the final image. The cache can grow to several times the image size. Give the cache repository its own retention policy, and expire old cache tags the same way you expire old branch images.
Multi-platform builds need a separate cache per platform. An arm64 compile layer cannot satisfy an amd64 build, so storage grows with each target, and each platform should write to its own cache ref.
Cache mounts are not exported with registry or inline cache. On a runner that is destroyed after every job, the mount starts empty each time and adds nothing.
Package caches also cost you reproducibility. A corrupted or stale entry in a mounted cache does not change any layer checksum, so the bad state can persist without a visible cause. Plan a periodic no-cache build, or a way to purge the builder, so you can rule the cache out when debugging.
| Approach | Development cost | Storage | Credentials | Fits |
|---|---|---|---|---|
| Local cache, persistent builder | Low | Builder disk | None | Long-lived self-hosted runners |
| Inline cache | Lowest | Inside image, min only |
Registry push | Simple single-stage images |
| Separate registry cache | Moderate | Registry, can be large | Registry push | Ephemeral runners, multi-stage builds |
| Cache mounts, persistent builder | Moderate | Builder disk | None | Heavy dependency downloads |
| CI cache save/restore | Moderate | CI cache quota | CI token | Pipelines without registry write access |
| No cache | None | None | None | Release builds, fast images, debugging |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to buildkit cache ci, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.