What are ephemeral preview environments and where does their cost come from?

A preview environment, also called a pull request environment, is a short-lived copy of your application stack that a CI pipeline deploys for each open PR. On Kubernetes this is usually one namespace per PR, holding the changed services, their dependencies, and often a database. Reviewers, QA, and product engineers get a working URL before merge. Backend and platform teams get to test migrations and service interactions against something closer to production than a laptop.

Most of the cost comes from capacity reserved for review traffic that rarely arrives:

Cost source Why it accumulates
CPU/memory requests on idle pods The scheduler reserves requested resources whether or not the pod handles traffic, so nodes scale up for environments nobody is using
Per-PR database instances or StatefulSets Databases are usually the largest pods, and they hold memory for buffers even when idle
Persistent volumes and snapshots With a Retain reclaim policy, or when volumes are provisioned outside the namespace, storage keeps billing after the namespace is deleted
Per-environment load balancers and public IPs A Service of type LoadBalancer provisions a cloud load balancer per PR, and each one is billed hourly
Environments left running after merge or abandonment No PR event triggers teardown, so the stack keeps running indefinitely

These costs share a cause. The pipeline that creates an environment rarely owns its deletion. Creation runs on every push. Deletion depends on a merge webhook firing, a job succeeding, and every external resource being tied to the namespace lifecycle. When any of those steps fails, the resources are orphaned and nothing reports them.

How do you estimate preview environment cost for your own team?

…keep billing until someone notices. Before choosing a fix, model the bill from your own numbers.

The core relationship is that the number of environments alive at once equals arrival rate times lifetime:

  • Concurrent environments = PRs opened per day × average lifetime in days
  • Monthly cost = concurrent environments × (compute cost per environment + database cost per environment), plus any shared fixed cost

Lifetime appears as a multiplier on everything. A PR that sits open over a weekend, or an environment whose teardown silently failed, costs as much as one under active review.

Illustrative estimate (assumed numbers, not a benchmark). Assume 20 PRs per day and a 5-day average lifetime. Assume compute (1 vCPU, 2 GiB requested) costs $0.05/hour, or about $36.50/month. Assume a dedicated database per environment costs $0.10/hour, or about $73/month. Assume a shared database instance costs $300/month.

Scenario Concurrent envs Monthly cost
Baseline: 5-day lifetime, dedicated DB 100 $10,950
Halve pod requests 100 $9,125
1-day lifetime, dedicated DB 20 $2,190
5-day lifetime, shared DB 100 $3,950
1-day lifetime, shared DB 20 $1,030

With these inputs, halving pod size saves about 17%. Cutting lifetime or sharing the database each saves far more, because those changes act on the largest terms in the formula. Change lifetime first, since a TTL is usually a small change. Sharing the database pays off when per-environment database cost is larger than compute cost, as it is here. If your environments already use a lightweight in-cluster database, rerun the numbers before you invest in that change.

How does a namespace TTL with a scheduled reaper work?

Start by marking each preview namespace with two pieces of metadata when CI creates it. A label such as preview=true lets the reaper select candidates cheaply with a label selector. An annotation such as preview/expires-at holds an ISO 8601 UTC timestamp. Use an annotation for the expiry because label values cannot contain colons, so they cannot hold a timestamp in that format.

The reaper is a scheduled job, either a Kubernetes CronJob or a scheduled CI pipeline, that runs every 15 to 30 minutes. It lists namespaces matching the label with kubectl get namespaces -l preview=true -o json, compares each expiry to the current time, and runs kubectl delete namespace on the expired ones. If it runs in-cluster, its ServiceAccount needs a ClusterRole that can only list and delete namespaces.

The namespace is the right unit of cleanup because deleting it cascades to every namespaced object inside it: Deployments, Services, Secrets, PersistentVolumeClaims, and Ingresses. You do not need to track individual resources. Cluster-scoped objects do not cascade, so keep previews from creating them. Also check that any PersistentVolumes use a Delete reclaim policy.

Tie lifetime to pull request events through CI:

  • Create the namespace when a PR opens or receives a preview label.
  • Extend the expiry annotation on each push.
  • Delete the namespace on merge or close.

Event-driven teardown is not enough on its own. Webhooks get dropped, teardown jobs fail halfway, CI runners go down, and force-pushes or branch deletions can skip the close handler entirely. Each failure leaves a namespace running with nothing scheduled to remove it. The reaper is the backstop: it guarantees that no environment outlives its expiry, whatever happened to the event that should have removed it.

#!/usr/bin/env bash
# Preview namespace lifecycle: stamp on create/push, reap on schedule.
# Requires GNU date (Linux CI runners), kubectl and jq.
set -euo pipefail

TTL="48 hours"

# Called from CI on PR open/label and on every push, e.g. stamp pr-1234.
# Re-running it pushes the expiry forward, which is how a push extends the TTL.
stamp() {
  local ns="$1"
  local expires; expires=$(date -u -d "+${TTL}" +%Y-%m-%dT%H:%M:%SZ)
  kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
  kubectl label namespace "$ns" preview=true --overwrite
  # Annotation, not label: label values cannot contain ':'.
  kubectl annotate namespace "$ns" preview/expires-at="$expires" --overwrite
}

# Run every 15-30 minutes by a CronJob or scheduled CI pipeline.
# Backstop for missed webhooks and failed teardown jobs.
reap() {
  local now; now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  kubectl get namespaces -l preview=true -o json \
    | jq -r '.items[]
        | [.metadata.name, (.metadata.annotations["preview/expires-at"] // "")]
        | @tsv' \
    | while IFS=$'\t' read -r ns expires; do
        if [[ -z "$expires" ]]; then
          echo "skip $ns: no expiry annotation"; continue
        fi
        # Same fixed UTC format on both sides, so string comparison orders correctly.
        if [[ "$expires" < "$now" ]]; then
          echo "deleting $ns (expired $expires)"
          # Cascades to every namespaced object inside it.
          kubectl delete namespace "$ns" --wait=false
        fi
      done
}

"$@"   # usage: ./preview.sh stamp pr-1234 | ./preview.sh reap

A single script that CI calls to create a preview namespace or extend its expiry on each push, and that a scheduled job calls to delete every preview namespace whose expiry annotation has passed.

How does the shared database pattern work for pull request environments?

Rather than giving each pull request its own database server, run one long-lived PostgreSQL instance per cluster or region and give each PR its own logical database. Idle servers are a fixed cost, while idle logical databases cost only their storage, so this design removes most of the per-PR baseline.

A provisioning job runs before the application deploys:

  • CREATE ROLE pr_482 LOGIN PASSWORD '…'
  • CREATE DATABASE pr_482 OWNER pr_482
  • REVOKE CONNECT ON DATABASE pr_482 FROM PUBLIC

The REVOKE matters because every role can connect to a new database by default. Without it, one PR's credentials could reach another PR's data. The job stores the credentials as a secret in the PR's namespace, and migrations then run against that database alone.

Teardown runs in reverse, before the namespace is deleted. It first terminates any open sessions with pg_terminate_backend, then runs DROP DATABASE pr_482 and DROP ROLE pr_482. Dropping the database first matters because PostgreSQL will not drop a role that still owns objects.

Seeding strategy Provisioning time Storage Privacy risk
Migrations plus small fixtures Grows with migration count Minimal Low
CREATE DATABASE … TEMPLATE Fast file-level copy Full template size per PR Low if template is synthetic
Sanitized snapshot restore Slowest Largest Depends on sanitization quality

Cloning from a template has two catches. The template drifts from the PR's schema, so you still need to run that PR's migrations after cloning. The copy also fails if any other session is connected to the template, so nothing may hold a connection to it.

Production data must never be copied into preview environments unsanitized.

-- Provisioning: run as an admin role with CREATEDB and CREATEROLE,
-- in autocommit mode (CREATE/DROP DATABASE cannot run inside a transaction).
CREATE ROLE pr_482 LOGIN PASSWORD 'example-not-a-real-secret';

-- Option A: empty database, then run this PR's migrations and fixtures.
CREATE DATABASE pr_482 OWNER pr_482;

-- Option B (instead of A): file-level copy of a synthetic template.
-- Fails if any other session is connected to app_template.
-- Still run this PR's migrations afterwards, because the template drifts.
-- CREATE DATABASE pr_482 TEMPLATE app_template OWNER pr_482;

-- New databases grant CONNECT to PUBLIC by default; remove it so
-- other PRs' roles cannot reach this database.
REVOKE CONNECT ON DATABASE pr_482 FROM PUBLIC;
GRANT CONNECT ON DATABASE pr_482 TO pr_482;

-- Teardown: run before the PR's namespace is deleted.
-- Block new connections so nothing reconnects mid-teardown.
ALTER DATABASE pr_482 WITH ALLOW_CONNECTIONS false;

-- End any sessions still attached (app pods, migration jobs).
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = 'pr_482'
  AND pid <> pg_backend_pid();

-- Drop the database first: a role that still owns objects cannot be dropped.
DROP DATABASE IF EXISTS pr_482;
DROP ROLE IF EXISTS pr_482;

SQL a provisioning job and a teardown job could run to create a per-PR PostgreSQL database and role, either empty or cloned from a synthetic template, lock down CONNECT, and later terminate sessions and drop the database before the role.

How do you make cleanup reliable and cap total spend?

Deleting a namespace removes only what Kubernetes owns. The database on the shared server, the DNS record and any cloud bucket survive unless something removes them first. Run teardown as an explicit step: a finalizer on a custom resource, or a teardown job that drops the database and only then deletes the namespace. If the drop fails, the namespace stays and the failure is visible. Back this up with a periodic reconciler that lists preview databases, compares them with open PRs and flags or drops the orphans. Tag every external resource (DNS records, volumes, cloud resources) with the PR identifier. When the reconciler finds a leak, it can then trace the resource to a specific PR and remove it safely.

Spend guardrails

Give each ephemeral namespace a ResourceQuota and a LimitRange. The quota caps the namespace's total CPU, memory and PVC usage. The limit range sets default requests and limits, so a pod without them cannot claim a whole node. Cap the number of concurrent preview environments and queue or reject requests beyond that limit, so a burst of PRs cannot scale the cluster without bound. Schedule previews onto a separate node pool, optionally on spot or preemptible capacity. Previews tolerate eviction, and the separate pool keeps their cost out of the production budget.

Scale-to-zero and off-hours shutdown reduce replicas while keeping the namespace and its configuration. Wake-up is fast compared with rebuilding the environment, but the first request after idle pays a cold start.

Observe the system

Export these metrics:

  • active environment count
  • age distribution
  • orphaned databases
  • failed teardowns
  • cost by PR label

A growing tail of old environments or a rising orphan count shows a cleanup regression on a dashboard weeks before it appears on the invoice.

# Applied to every ephemeral namespace when the preview is created.
# The pr label lets cost tools and the reaper trace resources to a PR.
apiVersion: v1
kind: ResourceQuota
metadata:
  name: preview-quota
  namespace: preview-pr-1234
  labels:
    preview.example.com/pr: "1234"
spec:
  hard:
    # Ceiling for the whole namespace, whatever the pods ask for
    requests.cpu: "2"
    requests.memory: 4Gi
    limits.cpu: "4"
    limits.memory: 8Gi
    persistentvolumeclaims: "2"
    requests.storage: 10Gi
    pods: "20"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: preview-defaults
  namespace: preview-pr-1234
  labels:
    preview.example.com/pr: "1234"
spec:
  limits:
    - type: Container
      # Applied when a container omits requests or limits
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        cpu: 500m
        memory: 512Mi
      # No single container can claim more than this
      max:
        cpu: "1"
        memory: 2Gi

A per-namespace ResourceQuota and LimitRange for one preview environment. The quota caps total CPU, memory and storage, the limit range gives defaults to pods that omit requests and limits, and the PR label lets cost reports and the reaper trace each resource back to its PR.

When should you avoid shared databases and aggressive TTLs?

These patterns trade isolation and fidelity for cost, and that trade is not always worth making.

A shared server is a shared failure domain. One PR with a runaway migration or a missing index can saturate CPU, I/O or connections for every other preview. Per-role connection limits and statement timeouts bound the damage. They do not isolate memory, disk or lock contention, and when the server goes down, every open PR goes down with it.

Schema-per-PR is not production. Extensions, server settings, the engine version and cluster-wide objects such as roles are shared across all schemas. A PR that adds an extension or changes a setting cannot be tested safely, and a test can pass only because another PR left state behind. Version upgrades, configuration changes, failover drills and realistic performance tests need a dedicated, full-fidelity environment.

Aggressive TTLs cost people time. If an environment expires overnight and takes several minutes to rebuild, reviewers and QA lose context and start asking for exemptions. A "pin" label helps, but pinned environments are rarely unpinned. Give pins their own, longer expiry and show them on the dashboard.

Spot evictions look like application bugs. A pod killed mid-request produces errors that reviewers will file against the PR. Mark spot-backed environments clearly and show eviction events next to application logs.

The provisioner has a wide blast radius. It can create and drop databases and namespaces, so one bad filter in teardown can delete every environment at once. Scope its permissions narrowly, require an ownership label before any deletion, and dry-run changes to destructive logic.

What are the alternatives to full per-PR preview environments?

Per-PR environments are one point on a spectrum. The options trade cost against how closely the environment matches production.

Option Relative cost Fidelity Setup effort Main risk
Full stack + dedicated DB per PR Highest High High Idle spend, leaked resources
Full stack, per-PR DB on shared server Medium High, except DB isolation Medium Noisy neighbours on the DB server
Shared staging Low Medium Low Queueing, conflicting changes
Routing-based (changed services only) Low–medium High for the changed path High Header propagation, baseline drift
Local or containerized stack Near zero infrastructure Low–medium Low–medium "Works on my machine" gaps

Small teams with a handful of services usually do well with a shared staging environment plus solid local stacks. Contention on staging stays rare at that size, and per-PR infrastructure would cost more to maintain than it saves. As concurrent PRs grow, staging becomes a queue. At that point, per-PR namespaces backed by a shared database server are the common middle ground. Routing-based environments make sense when the system has many services but a typical PR touches one or two. Deploying the whole stack per PR would then be mostly redundant. The cost is real investment in request routing and in keeping the baseline healthy.

To weigh engineering time saved, count two things. The first is hours reviewers spend waiting for staging or reproducing changes locally. The second is hours spent chasing broken or leaked environments.

As an illustrative estimate with assumed inputs: suppose 20 PRs a week each save 30 minutes of review friction. That recovers 10 engineer-hours weekly, which you can compare directly against the added cloud bill and maintenance time.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to preview environments cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.