Container Registry Storage Cost: Retention Policies and Pull-Through Caches

What drives container registry storage cost

This section is for platform and backend engineers who own registries with many repositories or high CI build volume. At that scale, small per-gigabyte charges add up across thousands of images that nobody tracks.

Local image sizes are misleading. docker images reports uncompressed size, but the registry bills compressed layers. Layers shared within one repository are stored once. A 1.2 GB local image that shares its base layers with the previous build may add only tens of megabytes.

Storage grows without anyone noticing, for several reasons:

• Per-branch CI pushes create images that outlive their branches.
• Mutable tags such as latest move to each new push. The previous image stays behind, untagged, and is still billed.
• Multi-arch manifest lists store one set of layers per architecture.
• Build cache and attestation artifacts (SBOMs, signatures, cache exports) are stored alongside images.
• No default deletion. A repository keeps everything until you add a retention rule.

Illustrative estimate (assumed numbers, not a benchmark)

Monthly storage growth = repositories × pushes per repository per month × unique compressed MB per push.

With 200 repositories × 150 pushes × 40 MB, that is about 1.2 TB of new billed storage each month. It accumulates indefinitely unless something deletes it.

Sizing the problem before you change anything

…a retention policy caps it. To see the steady state rather than the monthly growth, multiply by how long images are kept: repositories × builds per day × unique compressed layer delta × retention days.

Illustrative estimate (assumed inputs): 50 repositories × 10 builds per day × 60 MB delta × 90 days ≈ 2.7 TB held at steady state. Now compare the base layers. Assume a 300 MB base image stored once per repository: 50 × 300 MB ≈ 15 GB. Layers are deduplicated by digest, so a base layer that doesn't change between builds is stored once, no matter how many tags reference it. The per-build application layers are what accumulate: dependency bundles, compiled artifacts and copied source trees. Retention days scale only that term, so shrinking a bloated application layer can beat an aggressive expiry rule.

Check the estimate against billing data before writing any rules:

If registry storage is a small share of the bill next to NAT processing or egress, retention work saves little. Fix transfer first.

Cross-region replication changes the calculation. Each replica multiplies the stored bytes. It also spreads a bad expiry rule to every region at once, which removes the copy you would otherwise restore from.

Copy# Illustrative estimate only: every input is an assumption, not a measurement.
REPOS = 50
BUILDS_PER_DAY = 10
APP_DELTA_MB = 60      # unique compressed layer bytes each build adds
RETENTION_DAYS = 90
BASE_IMAGE_MB = 300    # shared base layers, stored once per repo (deduplicated by digest)
REPLICA_REGIONS = 1    # set > 1 if the registry replicates cross-region


def app_layers_gb(delta_mb, retention_days):
    # Steady state: retention deletes old layers as fast as new builds add them.
    return REPOS * BUILDS_PER_DAY * delta_mb * retention_days / 1000


app_gb = app_layers_gb(APP_DELTA_MB, RETENTION_DAYS)
base_gb = REPOS * BASE_IMAGE_MB / 1000
per_region_gb = app_gb + base_gb

print(f"per-build app layers : {app_gb:8,.0f} GB")
print(f"shared base layers   : {base_gb:8,.0f} GB")
print(f"app share of storage : {app_gb / per_region_gb:8.1%}")
# Replication multiplies stored bytes, and any expiry mistake, by region count.
print(f"total, {REPLICA_REGIONS} region(s)  : {per_region_gb * REPLICA_REGIONS:8,.0f} GB")

# Compare two levers on the dominant term before writing expiry rules.
print(f"retention cut to 60d : {app_layers_gb(APP_DELTA_MB, 60):8,.0f} GB")
print(f"app delta cut to 25MB: {app_layers_gb(25, RETENTION_DAYS):8,.0f} GB")

A back-of-envelope calculator, using assumed inputs and labelled as an illustrative estimate, showing that per-build application layers make up nearly all steady-state storage, that replication multiplies the total, and that shrinking the layer delta can save more than shortening retention.

How ECR lifecycle policies work and how to design the rules

An ECR lifecycle policy is a set of rules attached to a single repository. Each rule has a rulePriority, a selection, and an expire action. ECR evaluates rules in ascending priority order. Once a rule's selection matches an image, rules with lower priority (higher numbers) cannot expire that image. This protects your keep-rules from catch-all rules, so place narrow keep-rules first and broad sweeps last. A rule using tagStatus: any must carry the highest priority number.

Selection has two parts:

• tagStatus: tagged, untagged, or any. Tagged rules also take a tagPrefixList such as release- or feature-.
• countType:
  • imageCountMoreThan keeps the newest N matching images by push time and expires the rest.
  • sinceImagePushed expires matching images older than a set number of days. It measures time since push, not since last pull, so an old image that is still deployed can still expire.

The baseline policy in the example below has three rules:

1. Expire untagged images after a few days. These are mostly superseded pushes and leftover manifests.
2. Keep the last N images for each release prefix. N must be large enough to cover your rollback window.
3. Expire feature-branch prefixes after a fixed age.

Use the console's lifecycle preview to see what a policy would expire before you apply it. Expiry runs asynchronously, so it does not happen immediately.

Expect less savings than the image count suggests. Storage is reclaimed only for layers that no remaining image references. Most images share base and dependency layers, so deleting 80% of the images may delete only the small application layers at the top. The large shared layers stay.

Copy# Lifecycle policy for one repository. Rules are evaluated in ascending
# rulePriority, and an image matched by an earlier rule is not expired by a later one.
# Release images are kept by count, feature branches expire by age, and
# untagged leftovers are swept last.
cat > policy.json <<'EOF'
{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Keep the newest 20 release images (covers the rollback window)",
      "selection": {
        "tagStatus": "tagged",
        "tagPrefixList": ["release-"],
        "countType": "imageCountMoreThan",
        "countNumber": 20
      },
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 2,
      "description": "Expire feature-branch images 14 days after push",
      "selection": {
        "tagStatus": "tagged",
        "tagPrefixList": ["feature-"],
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 14
      },
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 3,
      "description": "Expire untagged images 3 days after push",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 3
      },
      "action": { "type": "expire" }
    }
  ]
}
EOF

# Preview what the policy would expire, then apply it.
aws ecr start-lifecycle-policy-preview --repository-name my-app --lifecycle-policy-text file://policy.json
aws ecr get-lifecycle-policy-preview --repository-name my-app
aws ecr put-lifecycle-policy --repository-name my-app --lifecycle-policy-text file://policy.json

This script defines a three-rule baseline ECR lifecycle policy, previews what it would expire, and then applies it to a repository with the AWS CLI. The policy keeps the last 20 release images, expires feature-branch images by age, and sweeps untagged images last.

When retention policies break deploys and rollbacks

A lifecycle policy evaluates push time, tags and image counts. It has no view of what is actually running. Images referenced by running pods, ECS task definitions, Helm release history or GitOps manifests are all eligible for expiry. The deletion causes no immediate problem, because nodes that already pulled the image keep running it from their local cache. The failure arrives later, when the autoscaler adds a node, a pod is evicted and rescheduled, or a spot instance is reclaimed. The new node tries to pull, gets a "not found" response, and the workload sits in ImagePullBackOff or the ECS task fails to start. This often happens during a traffic spike, which is exactly when you need the capacity.

Who gets hit first

Stable, rarely deployed services are the most exposed. An age-based rule such as "expire images older than 90 days" removes the only image of a service that last shipped four months ago. Count-based rules have a related failure. If feature-branch builds share a tag prefix with release builds, a busy week of branch pushes can push the production image out of the "keep last N" window.

Mutable tags make this harder to predict. When prod or latest moves to a new image, the previous image becomes untagged. A rule that expires untagged images will then delete it, even if a manifest still pins that image by digest.

The trade-off

Every image you expire is one you can no longer roll back to. It is also one you can no longer pull during incident forensics to inspect its exact dependencies or reproduce a bug. The storage saved is usually small compared to the cost of losing either option during an outage.

Making retention safe: immutable tags, protected prefixes and audits

The fix is to make the set of images that must never expire explicit, then verify that the policy agrees.

Immutable release tags. Turn on tag immutability for repositories that hold deployable images. A tag like v2.14.3 then always resolves to one digest, so "the version in production" names a single image. This requires changes in CI. Pipelines that re-push latest or overwrite a branch tag on every build will start failing. Release jobs need unique tags derived from the commit SHA or version, and a retried job must skip the push when the tag already exists. Deploy manifests should reference those unique tags or the digests.

A protected prefix. Have the deploy step add a dedicated tag, such as deployed-, to anything it ships. Give that prefix the highest-priority rule with a count threshold far above what you will ever reach. An image claimed by a higher-priority rule cannot be expired by a lower-priority one, so this rule protects deployed images from cleanup rules further down.

Preview, review, audit. Before you enable or change a policy, run aws ecr start-lifecycle-policy-preview and read the expiry list it returns. Keep policies in version control and require review. The usual causes of deleted production images are:

• Rule priorities in the wrong order.
• A prefix like v that matches far more tags than intended.

On a schedule, compare the digests running in your clusters against the preview's expiry list, and alert on any overlap.

These checks prevent a slow, late-surfacing failure. Without them, a deleted image surfaces weeks later as a pull error during autoscaling or node replacement. By then no recent change explains it, and diagnosing the failure takes hours.

Copy#!/usr/bin/env bash
# Pre-merge / scheduled check: fail if the lifecycle policy would expire any running image.
set -euo pipefail
REPO="payments-api"
POLICY="lifecycle/payments-api.json"   # policy kept in version control and reviewed

# One-time setting: release tags always resolve to a single digest.
aws ecr put-image-tag-mutability --repository-name "$REPO" \
  --image-tag-mutability IMMUTABLE

# Dry-run the candidate policy. A preview does not delete anything.
aws ecr start-lifecycle-policy-preview --repository-name "$REPO" \
  --lifecycle-policy-text "file://$POLICY" > /dev/null

# Wait for the preview to finish. Treat FAILED or EXPIRED as an error.
while :; do
  status=$(aws ecr get-lifecycle-policy-preview --repository-name "$REPO" \
    --query status --output text)
  case "$status" in
    COMPLETE) break ;;
    IN_PROGRESS) sleep 5 ;;
    *) echo "preview status: $status" >&2; exit 2 ;;
  esac
done

# Digests the policy would expire.
aws ecr get-lifecycle-policy-preview --repository-name "$REPO" \
  --query 'previewResults[].imageDigest' --output text \
  | grep -o 'sha256:[0-9a-f]\{64\}' | sort -u > expiring.txt || true

# Digests actually running in the current cluster context.
kubectl get pods -A \
  -o jsonpath='{range .items[*].status.containerStatuses[*]}{.imageID}{"\n"}{end}' \
  | grep -o 'sha256:[0-9a-f]\{64\}' | sort -u > deployed.txt

# Any overlap means a rule-order or prefix mistake. Block the change or alert.
overlap=$(comm -12 deployed.txt expiring.txt)
if [ -n "$overlap" ]; then
  echo "Policy would expire running images:" >&2; echo "$overlap" >&2; exit 1
fi
echo "OK: no running digests in the expiry preview"

This script enables tag immutability, previews a version-controlled ECR lifecycle policy, and fails if any digest running in the Kubernetes cluster appears in the preview's expiry list, so you can run it on policy pull requests and on a schedule.

Pull-through caches for public base images

Base images such as language runtimes, distros and sidecars are often pulled thousands of times a day from public registries. When nodes run in private subnets, every one of those pulls crosses a NAT gateway, so you pay data processing on each layer and you are exposed to anonymous rate limits upstream. A pull-through cache mirrors those upstreams into your private registry namespace. After the first fetch, pulls stay inside your account and region, and the cache becomes a single place to apply scanning and access control to third-party images.

CopyWithout cache:  node ──> NAT gateway ──> internet ──> public registry   (every pull)
With cache:     node ──> private registry /docker-hub/... ──> upstream  (first pull only)

Implementation.

1. Create a rule per upstream that maps it to a repository prefix, such as docker-hub/ or quay/.
2. For upstreams that require authentication, store the credentials in your cloud provider's secrets store and reference them from the rule.
3. Make workloads actually use the cache. You can rewrite image references in manifests and Dockerfiles, or configure registry mirrors in the container runtime. Without this step the rule exists, but traffic still goes out through NAT.

Limits.

• New dependency. The cache now sits in the pull path. If it is misconfigured or cannot reach the upstream, uncached images fail to pull.
• Cold pulls. The first pull of an image is slower than pulling directly.
• Tag lag. Cached tags can lag behind upstream, so pin images by digest.
• No effect on your own images. The cache does nothing for the storage your own images consume.
• Default settings. Cache repositories are created on demand and receive default settings. Use creation templates or automation to apply lifecycle policies, scanning and permissions, or the cache becomes a new source of unbounded storage.

Copy#!/usr/bin/env bash
# Point containerd at the pull-through cache for Docker Hub images, so
# unchanged references like "python:3.12-slim" stop going out through NAT.
# Assumes containerd's CRI registry config_path is set to /etc/containerd/certs.d.
set -euo pipefail

CACHE_HOST="registry.example.internal"   # private registry endpoint (placeholder)
CACHE_PREFIX="docker-hub"                # prefix mapped to Docker Hub in the cache rule

sudo mkdir -p /etc/containerd/certs.d/docker.io

sudo tee /etc/containerd/certs.d/docker.io/hosts.toml >/dev/null <<EOF
# Fallback: if the cache host fails, containerd tries the upstream directly.
# Remove this line if private nodes have no internet route.
server = ""

# Try the cache first. override_path makes containerd use the path as written,
# so pulls go to /v2/docker-hub/<repo> instead of /v2/<repo>.
[host.""]
  capabilities = ["pull", "resolve"]
  override_path = true
EOF

# The alternative is to rewrite references in manifests and Dockerfiles. Pin by digest
# so a cached tag that lags upstream cannot change what you run:
#   FROM registry.example.internal/docker-hub/library/python@sha256:<digest>

# Check that the cache serves the image. The first pull is a cold fill and is slower.
sudo crictl pull docker.io/library/python:3.12-slim

A containerd registry-host configuration that sends Docker Hub pulls to a pull-through cache prefix first, with digest-pinned direct references shown as the alternative.

Alternatives and adjacent levers, and their effect on cost and engineering time

Retention and caching are not the only levers. Several build and network changes reduce different line items, so check which one your measured breakdown says is largest before choosing.

The endpoint change is easy to overlook because registry traffic does not show up on the registry's own bill. Layer downloads from private subnets that go through NAT are charged as NAT processing. To estimate the effect, compare these two figures:

• Current NAT processing cost for pulls: pulled GB per month × NAT per-GB rate
• Endpoint cost: endpoint hours × hourly rate + interface GB × endpoint per-GB rate

S3 gateway endpoints carry no data charge, so most layer bytes leave the NAT bill without adding an endpoint data charge.

Prioritise by the breakdown:

• Storage is the largest item: start with retention, then multi-stage builds.
• NAT processing or egress is the largest item: start with endpoints and caching, then smaller images, because every byte you remove from an image is saved again on each pull.
• API request charges are the largest item: look at how often clients poll. Image size will not change that line.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to container registry storage cost, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.