Docker for Backend Developers: Image Layers and Build Cache Explained

How Docker image layers work

A Docker image is an ordered stack of read-only filesystem layers. Each RUN, COPY and ADD instruction in a Dockerfile produces a new layer that records only the files added, changed or removed relative to the layer below. Instructions like ENV, WORKDIR, EXPOSE and CMD change image metadata rather than file contents, so they contribute no filesystem data. At runtime, the storage driver merges the layers into a single view of the filesystem. When you start a container, Docker adds a thin writable layer on top. Writes go there using copy-on-write, and the image beneath is never modified. Many containers can therefore share one image on disk.

To inspect an image, run docker history <image>. It lists each layer with the instruction that created it and its size. Add --no-trunc to see full commands. Metadata-only steps show 0B, and the large rows show which steps are responsible for most of the image size.

Why deleting files doesn't help

Layers are additive. If one RUN downloads a 300 MB toolchain and a later RUN deletes it, the later layer only records a "whiteout" marker that hides the file. The 300 MB is still stored and shipped in the earlier layer. To keep the files out of the image, download, use and delete them within the same RUN instruction.

The same rule applies to secrets. A credential copied in with COPY stays in that layer even if a later step removes it, and anyone who can pull the image can recover it with docker save. Values passed through ARG and used in a RUN step can also appear in docker history output. Neither mechanism is safe for credentials.

Copy#!/usr/bin/env bash
set -euo pipefail

docker build -t layers-bad - <<'EOF'
FROM alpine:3.19
RUN dd if=/dev/zero of=/opt/toolchain.bin bs=1048576 count=300
RUN rm /opt/toolchain.bin
ENV APP_ENV=production
CMD ["sh"]
EOF

docker build -t layers-good - <<'EOF'
FROM alpine:3.19
RUN dd if=/dev/zero of=/opt/toolchain.bin bs=1048576 count=300 \
    && rm /opt/toolchain.bin
ENV APP_ENV=production
CMD ["sh"]
EOF

docker history layers-bad
docker history --no-trunc layers-good

docker image ls layers-bad
docker image ls layers-good

Builds two images that end with identical filesystems. docker history and docker image ls show that deleting a file in a later RUN leaves its bytes in the earlier layer, and that only deleting it in the same RUN keeps it out of the image.

How build cache invalidation works

Docker evaluates cache step by step, from the top of the Dockerfile down. For each instruction it checks whether a matching layer already exists. As soon as one step misses, every instruction after it is rebuilt, even if those later steps have not changed. The order of instructions therefore decides how much work a small edit costs.

Each instruction type builds its cache key from different inputs:

The RUN behaviour causes the most trouble. Docker does not know that a command reaches the network, so a security patch released upstream will not reach your image until something above that step changes or you build with --no-cache.

COPY is sensitive to whatever is in the build context. A broad COPY . . includes .git, node_modules, build output and local .env files, so a commit or a test run can invalidate the layer. Listing these paths in .dockerignore keeps unrelated changes out of the cache key. It also keeps secrets out of the image.

Dependency caching only gives reliable results when you use a lockfile. Without one, a cached install layer and a fresh install can resolve to different versions, and nothing reports the difference. Backend engineers feel this as slow local rebuilds. Platform engineers feel it as CI minutes spent on rebuilds and as images that behave differently depending on which layers were reused.

CopyFROM node:20-bookworm-slim

RUN apt-get update \
 && apt-get install -y --no-install-recommends ca-certificates \
 && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY package.json package-lock.json ./

RUN npm ci --omit=dev

COPY . .

ENV NODE_ENV=production
CMD ["node", "src/server.js"]

In this Dockerfile, the steps that rarely change come before the steps that change often. It copies the lockfile before the source code and relies on .dockerignore, so a code edit only rebuilds the final layers and the dependency install stays cached and reproducible.

Writing a Dockerfile as a dependency graph

The fix starts with how you read a Dockerfile. Treat it as a dependency graph, not a script. Each instruction depends on everything above it, so a change near the top invalidates every layer below. Put the inputs that change least at the top and the ones that change most at the bottom.

In practice, a backend Dockerfile goes in this order: base image, system packages, dependency manifests, dependency install, then application source. The naive version runs COPY . . before installing dependencies. Editing one handler changes the build context, which invalidates the COPY layer, so the full install runs again. The reordered version copies only the manifests first: package.json with its lockfile, requirements.txt, go.mod and go.sum, or pom.xml. Then it installs dependencies and copies the source last. Source edits now rebuild only the final layers. The install step reruns only when a manifest actually changes.

Remove package-manager caches in the same RUN that created them. For example, end an apt-get update && apt-get install chain with rm -rf /var/lib/apt/lists/*. If you delete those files in a later RUN, they still sit in the earlier layer and still ship with the image. For the same reason, chain closely related commands with && so they succeed, fail and get cached together.

Don't take this too far. If you merge the whole build into one large RUN, you get fewer layers but a coarser cache, because any change reruns everything. Put layer boundaries where the rate of change differs. The aim is good cache reuse, not the smallest number of layers.

CopyFROM python:3.12-slim
WORKDIR /app
COPY . .
RUN apt-get update
RUN apt-get install -y --no-install-recommends libpq5
RUN rm -rf /var/lib/apt/lists/*
RUN pip install -r requirements.txt
CMD ["python", "-m", "app"]

FROM python:3.12-slim
WORKDIR /app

RUN apt-get update \
 && apt-get install -y --no-install-recommends libpq5 \
 && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "-m", "app"]

The naive Dockerfile copies all source before installing dependencies and cleans the apt cache in a separate layer. The reordered version cleans up in the same RUN, copies only requirements.txt before the install, and copies source last, so code edits reuse the cached dependency layers.

Multi-stage builds for compiled and interpreted services

A single-stage image ships everything used to build the service: compilers, headers, package manager caches and test tooling. Multi-stage builds separate those concerns. The builder stage starts from a full SDK image, resolves dependencies and produces an artifact. The runtime stage starts from a small base and uses COPY --from=builder to bring in only that artifact.

Go: copy one binary

For Go, the builder compiles with CGO_ENABLED=0, which produces a statically linked binary. The runtime stage needs almost nothing else: the binary, CA certificates and a non-root USER. Pin both stages by tag or, better, by digest (image@sha256:...). A floating tag can change under you between two builds of the same commit.

Python: copy a built environment

Interpreted services still need their runtime, so the gain comes from leaving build tools behind. The builder installs compilers and builds wheels or a virtualenv. The runtime stage copies that environment onto a slim image with the same Python version, then switches to a non-root user.

Trade-offs

• More complexity. Two stages mean two base images to pin and update, and more stage logic to review.
• Builder/runtime mismatches. If you build against glibc and run on musl (Alpine), or the runtime image lacks a shared library that a native extension links against, the build passes. The failure shows up only at import time or at startup in production. Use the same distro family in both stages, and run a smoke test against the final image in CI.
• No shell in minimal images. Distroless or scratch-based images are smaller and expose less attack surface, but they have no shell, package manager or debug tools. Plan for that before an incident. Attach an ephemeral debug container (for example, kubectl debug on Kubernetes), or publish a -debug variant that has the same layers plus tooling.

CopyFROM golang:1.22-bookworm AS builder
WORKDIR /src

COPY go.mod go.sum ./
RUN go mod download

COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" \
    -o /out/server ./cmd/server

FROM scratch

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /out/server /server

USER 65532:65532

EXPOSE 8080
ENTRYPOINT ["/server"]

A two-stage Go Dockerfile: the builder compiles a static binary with the full toolchain, and the runtime stage copies only that binary and the CA certificates onto an empty scratch base, then runs it as a non-root numeric user.

BuildKit cache mounts and secret mounts

Layer ordering decides whether a step reruns. Cache mounts decide how expensive the rerun is. Add one dependency to package-lock.json and the install layer is invalidated, so the package manager starts from an empty cache and downloads everything again. RUN --mount=type=cache,target=/root/.cache/pip gives that step a persistent directory that BuildKit keeps on the builder between builds. The layer is still rebuilt, but most packages come from the local cache instead of the network. The mounted directory is never written into the image, so you also skip the usual rm -rf cleanup that exists only to keep caches out of layers.

The cache lives on the builder, not in the image or registry. On ephemeral CI runners it starts empty unless the runner persists builder state, which section 6 covers.

Credentials need the opposite guarantee: present during one step, absent afterwards. A value passed with ARG is recorded in the image history. A file added with COPY stays in that layer even if a later step deletes it. RUN --mount=type=secret,id=npmrc,target=/root/.npmrc exposes the file only while that RUN executes, and you supply it at build time with docker build --secret id=npmrc,src=$HOME/.npmrc. Nothing is committed to a layer, and changing the secret does not invalidate the cache.

Caching in CI and keeping images fresh

A CI runner that starts from a fresh VM has no local layers, so every RUN step executes again unless you give BuildKit a cache to work from. The usual approach is to export the cache at the end of a build and import it at the start of the next one. You can store it in a container registry or in another storage backend supported by buildx. The Docker build cache documentation lists the available backends and their export modes.

Imported cache has costs. The registry stores cache blobs that need retention rules, and each job downloads those blobs before it can reuse them. On a project where most layers change on every commit, pulling the cache can take longer than rebuilding. Time both approaches in your own pipeline before you commit to one.

A cache can also be too effective. A layer that ran apt-get upgrade or resolved loose dependency ranges months ago will keep getting reused until something above it changes, and it will carry any vulnerabilities fixed since then. Plan for refreshes: run a scheduled build with --pull to fetch the current base image, run an occasional --no-cache build to re-resolve packages, or let a dependency bot open pull requests that bump the base image.

How it works, end to end

The earlier sections each cover one part of the system. This is the full sequence for a single docker build:

1. The build context goes to the builder. The client sends the files in the build directory, minus anything matched by .dockerignore, to BuildKit. Every COPY and ADD reads from this context, so its contents feed directly into cache keys.
2. The Dockerfile becomes a graph of steps. BuildKit parses the instructions into a graph. Stages that don't depend on each other can build in parallel, and stages the target doesn't need are skipped.
3. Each step gets a cache key. The key combines the parent step's key with the step's own inputs: the command string for RUN, file checksums for COPY and ADD, and values for ENV and ARG. Because the parent's key is part of every key, one change cascades to everything below it.
4. Hit or execute. If a layer with a matching key exists in the local cache or in an imported cache, BuildKit reuses it. If not, it runs the step in a temporary container, with any cache or secret mounts attached, and snapshots the filesystem changes as a new layer. Mounted directories are left out of that snapshot.
5. Layers are content-addressed. Each layer is stored and identified by a digest of its contents. The image is a manifest that lists those layer digests plus a config that holds metadata such as ENV, USER and CMD.
6. Push and pull move only missing layers. A registry already holding a layer digest doesn't receive it again, and a host that already has it doesn't download it again. This is why small, late-changing layers make pushes and deploys cheap, and why a large layer near the top of the Dockerfile is expensive every time it changes.
7. A container adds a writable layer. At runtime, the storage driver stacks the read-only layers, places a copy-on-write layer on top, and discards that layer when the container is removed.

Every technique in this article works on one of these steps. .dockerignore controls step 1. Instruction ordering and lockfiles control step 3. Cache mounts, secret mounts and CI cache import control step 4. Multi-stage builds decide which layers end up in the manifest at step 5.

Impact on build time, cost and engineering time

Illustrative estimate (assumed numbers, not a benchmark). Assume a service where installing dependencies takes 90 s and produces a 400 MB layer, while copying and building source takes 10 s and produces a 20 MB layer. One source file changes and the lockfile does not.

The general formula is: time saved per period = rebuilds per period × dependency install time, as long as dependencies did not change. With an assumed 15 local rebuilds per developer per day, the reordered layout saves 15 × 90 s = 22.5 minutes per developer per day. With an assumed 40 CI runs per day that hit a warm cache, it saves 40 × 90 s = 60 runner-minutes per day. The push size also drops from 420 MB to 20 MB per build, which lowers registry transfer and storage costs.

Most of the cost is engineering time, not runner time. A 100-second wait is long enough for a developer to switch context. A 10-second wait is not. Faster rebuilds also make people more willing to run the real container locally before pushing, so they find packaging bugs earlier.

The extra structure is not worth much for throwaway prototypes, one-off debugging images, or images with no meaningful dependency step. In those cases, a two-line Dockerfile is fine. Restructure the Dockerfile once the image is rebuilt many times a day or is built by more than one person.

Who should use these techniques

These practices are worth adopting if any of the following describes you:

• Backend developers who rebuild images during the inner loop. You run the service in a container locally, through Compose or a local cluster, and rebuild after code changes. Ordering layers correctly is the biggest win here, because it turns most rebuilds into source-only rebuilds.
• Teams with dependency-heavy services. Large node_modules trees, Python packages with native extensions, or JVM dependency sets make the install step the expensive part of the build. Lockfile-first ordering and cache mounts target that step directly.
• Platform and CI owners. You pay for runner time and registry storage, and you own build reproducibility. Cache import and export, digest pinning and scheduled refresh builds fall under your responsibility.
• Anyone shipping images that handle credentials or reach production. Even if build speed doesn't matter to you, knowing that deleted files and ARG values persist in layers is required to avoid leaking secrets. Multi-stage builds and non-root users reduce what an attacker gets from a compromised container.

The return is small if you build an image rarely, if the image has no real dependency step, or if it's a short-lived experiment. You still need the secrets guidance in those cases. The caching structure can wait.

Alternatives

Restructuring a Dockerfile is not the only way to get fast, small and reproducible images. Each alternative below solves part of the problem and comes with its own trade-offs.

• Run the service natively during development. Run the process directly on your machine with the language's own tooling, and build the image only in CI. Iteration is as fast as the toolchain allows. The downside is drift between your laptop and the container: packaging bugs, missing system libraries and file-path assumptions show up later.
• A pre-built dependency base image. Build a separate image that contains the OS packages and the resolved dependencies, publish it to your registry, and base service images on it. Service builds only copy source. You now maintain two images, and the base image needs its own rebuild schedule or it accumulates unpatched packages.
• Language-specific image builders. Some ecosystems, notably Go and the JVM, have tools that assemble an OCI image directly from build output without a Dockerfile. They usually split dependencies and application code into separate layers automatically and produce reproducible output. You lose the flexibility of arbitrary RUN steps, and adding OS packages or native libraries is harder.
• Buildpack-based builders. Buildpacks detect the language from the source tree, then install the runtime and dependencies into well-defined layers. They can also rebase an image onto a patched base without rebuilding the application layers. You give up fine control over the image contents and accept the builder's conventions.
• Daemonless Dockerfile builders. For CI environments where running a Docker daemon is not allowed, there are builders that execute Dockerfiles without one. The layer and cache concepts in this article still apply, but cache storage and flag names differ, so check how each one persists cache between jobs.
• Declarative, fully pinned package systems. Some package managers describe the entire dependency graph, including system libraries, in pinned form and can emit container images from it. They give strong reproducibility and precise layer reuse at the cost of a steep learning curve and a separate toolchain the whole team has to adopt.
• Plain --no-cache builds. Always building from scratch removes every stale-cache problem and makes each build pick up current packages. Every build pays the full cost, so this works as a scheduled refresh rather than a default.

For most backend teams that already use Docker, a well-ordered Dockerfile with a lockfile, multi-stage builds and BuildKit mounts covers most needs with the fewest new tools. Move to one of the alternatives when it removes a specific cost you can measure in your own pipeline.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to docker for backend, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.