Preview Environment Cost: Auto-Sleeping Kubernetes Namespaces

How to model preview environment cost

A preview environment is a short-lived copy of your application, usually one namespace per pull request, deployed so reviewers can exercise the change. Its cost is mostly compute that stays reserved while nobody is looking at it.

Model it from requests, not usage. The scheduler places pods by their CPU and memory requests, and the cluster autoscaler adds nodes when pending pods' requests don't fit. An idle namespace that requests 2 vCPU holds 2 vCPU of node capacity even at near-zero utilisation.

Cost = live namespaces × awake hours × price per namespace-hour

Price per namespace-hour = node hourly price × max(CPU requested ÷ node CPU, memory requested ÷ node memory)

The max reflects whichever resource fills the node first. This formula ignores bin-packing waste and per-node DaemonSet overhead, so real costs run somewhat higher.

Illustrative estimate (assumed numbers, not a benchmark). These inputs are assumed:

• 20 live namespaces.
• Each namespace requests 2 vCPU and 4 GiB.
• Nodes have 4 vCPU and 16 GiB and cost $0.20/hour.
• A month is 730 hours with 22 workdays.

CPU binds first (2 ÷ 4 is larger than 4 ÷ 16), so each namespace costs $0.10 per hour.

Auto-sleep is worth building when idle namespace-hours are large. That happens when many pull requests stay open for days, reviews happen in short bursts, or the per-namespace requests are heavy. Schedule-based sleep helps less when a team spans time zones, because some reviewer is always in working hours. In that case, sleeping after a period of inactivity captures more of the idle time.

What scale to zero in Kubernetes means in practice

At the workload level, "sleep" means setting spec.replicas to 0 on every Deployment and StatefulSet in the namespace. The objects, Services, ConfigMaps, Secrets and PVCs stay in place, so waking only means scheduling pods again rather than redeploying. The replica count is the only state you lose, so record it before scaling down. A common pattern is an annotation on each workload, such as kubectl annotate deployment api sleep.example.com/replicas=3 --overwrite, followed by kubectl scale deployment api --replicas=0. On wake, read the annotation back and patch spec.replicas to that value. Without the stored count, every service wakes at 1 replica or at whatever the manifest says, which may not match what reviewers were testing.

A standard HorizontalPodAutoscaler cannot take a workload to zero, because minReplicas has a floor of 1. When you scale the target to 0 yourself, the HPA controller stops acting on it. It will not bring the workload back when traffic arrives, so the wake path has to restore replicas explicitly. Some sleep tools also delete or suspend the HPA and recreate it on wake, so it does not compete with the restore step.

GitOps controllers create a second conflict. Argo CD with self-heal, or Flux on its next reconcile, sees replicas: 0 as drift and reverts it within minutes. There are three common ways to coordinate:

• Leave replicas out of the Git manifests.
• Configure Argo CD to ignore differences on /spec/replicas.
• Have the sleep job suspend reconciliation, for example with flux suspend kustomization, and resume it as part of wake.

Copy#!/usr/bin/env bash
# Sleep/wake a preview namespace by scaling workloads to 0 and storing the
# original replica count in an annotation. Usage: ./preview-sleep.sh sleep|wake
set -euo pipefail
NS="pr-1234"                 # preview namespace
KS="pr-1234"                 # Flux Kustomization that manages it
KEY="sleep.example.com/replicas"

sleep_ns() {
  # Stop Flux first so it does not revert replicas: 0 as drift.
  flux suspend kustomization "$KS" -n flux-system
  for kind in deployment statefulset; do
    for name in $(kubectl get "$kind" -n "$NS" -o jsonpath='{.items[*].metadata.name}'); do
      current=$(kubectl get "$kind" "$name" -n "$NS" -o jsonpath='{.spec.replicas}')
      # Skip workloads already at 0 so a second run keeps the stored count.
      [ "$current" = "0" ] && continue
      kubectl annotate "$kind" "$name" -n "$NS" "$KEY=$current" --overwrite
      kubectl scale "$kind" "$name" -n "$NS" --replicas=0
    done
  done
}

wake_ns() {
  for kind in deployment statefulset; do
    for name in $(kubectl get "$kind" -n "$NS" -o jsonpath='{.items[*].metadata.name}'); do
      # Dots in the annotation key must be escaped in kubectl JSONPath.
      saved=$(kubectl get "$kind" "$name" -n "$NS" -o jsonpath='{.metadata.annotations.sleep\.example\.com/replicas}')
      [ -z "$saved" ] && continue
      # Restore explicitly: an HPA will not scale a target up from 0 by itself.
      kubectl scale "$kind" "$name" -n "$NS" --replicas="$saved"
      kubectl annotate "$kind" "$name" -n "$NS" "$KEY-"   # trailing '-' removes it
    done
  done
  # Resume reconciliation only after replicas are back to their recorded values.
  flux resume kustomization "$KS" -n flux-system
}

case "${1:-}" in
  sleep) sleep_ns ;;
  wake)  wake_ns ;;
  *) echo "usage: $0 sleep|wake" >&2; exit 1 ;;
esac

A bash script that suspends a Flux Kustomization, stores each Deployment and StatefulSet's replica count in an annotation before scaling it to 0, and on wake restores the counts and resumes reconciliation.

Namespace auto sleep approaches compared

Time-based schedules are the model kube-downscaler uses. A downscaler/uptime or downscaler/downtime annotation on a namespace or workload declares a window such as weekdays 08:00–19:00 in a named time zone. A downscaler/exclude annotation exempts a namespace or workload, for example a shared mock service. The signal is the clock, so it is fully predictable. It is also blind to use: an engineer in another region, or anyone testing a hotfix at 22:00, finds the environment asleep. Daylight-saving changes and a cluster-wide default zone that nobody overrode are the usual causes of "it slept during my demo."

Inactivity-based sleep measures actual use. It watches ingress request counts, or the time since the last commit or pull request event, and scales down after a quiet period. This is more accurate per environment, but you have to build and own it. Traffic signals are noisy, because health checks, uptime monitors and crawlers look like users unless you filter them. Git events miss a reviewer who is clicking through the app without pushing anything.

Plain CronJobs that patch replicas take minutes to write. They have no exclusion model, no memory of the original replica counts unless you store them, and no handling of time zones beyond the schedule you hard-code.

Use schedules when your team works in one region. Use inactivity signals when there are many environments spread across time zones. Use CronJobs only for a handful of environments.

Copy# Preview namespace on a weekday schedule, in the team's own time zone.
# kube-downscaler scales workloads here to zero outside this window.
apiVersion: v1
kind: Namespace
metadata:
  name: preview-pr-1234
  annotations:
    # Set the zone explicitly. Do not rely on a cluster-wide default.
    # A named zone such as Europe/Berlin follows daylight-saving changes.
    downscaler/uptime: "Mon-Fri 08:00-19:00 Europe/Berlin"
---
# A shared mock service that other previews call stays up around the clock.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-mock
  namespace: preview-pr-1234
  annotations:
    # Per-workload exclusion. This overrides the namespace schedule.
    downscaler/exclude: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: payments-mock
  template:
    metadata:
      labels:
        app: payments-mock
    spec:
      containers:
        - name: mock
          image: registry.example.com/payments-mock:1.0.0
          ports:
            - containerPort: 8080

A preview namespace with a kube-downscaler uptime window in an explicit time zone, plus a shared mock Deployment excluded so it stays running outside that window.

Making savings real: releasing nodes with the cluster autoscaler

Scaling Deployments to zero removes pods, not cost. You pay for nodes, so the savings only appear when the cluster autoscaler deletes the machines those pods were running on.

Sleep controller → sets replicas to 0 → pods terminate → nodes become empty or underutilized → cluster autoscaler marks them unneeded, waits, drains them → cloud node group shrinks its desired size → the instances stop billing.

The autoscaler only removes a node when it can move every remaining pod somewhere else. Several common conditions stop it from doing so:

• PodDisruptionBudgets. A PDB with minAvailable equal to the replica count allows zero evictions, so the node stays.
• Local storage. Pods using emptyDir or hostPath can block scale-down, depending on the autoscaler's local-storage setting.
• System pods. Pods in kube-system without a PDB, bare pods with no controller, and pods annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "false" all block removal.
• DaemonSet-only nodes. DaemonSet pods do not block removal. However, large DaemonSet requests can push a node's utilization above the threshold if the autoscaler counts them toward utilization.
• Autoscaler settings. A node group minimum size above zero, a long unneeded-time window, or a scale-down-disabled annotation on the node will keep nodes running regardless of load.

To find out why a specific node stays up, start with the node's events. The autoscaler emits events explaining why it did not scale a node down. Next, check the cluster-autoscaler-status ConfigMap in kube-system. If those don't explain it, the autoscaler logs name the specific pod that is blocking removal.

Copy#!/usr/bin/env bash
# Diagnose why the cluster autoscaler is not removing a node after preview namespaces go to sleep.
set -euo pipefail
NODE="${1:?usage: $0 <node-name>}"

# 1. Events on the node. The autoscaler records why it did not scale this node down.
kubectl get events -A \
  --field-selector involvedObject.kind=Node,involvedObject.name="$NODE" \
  --sort-by=.lastTimestamp

# 2. Is scale-down explicitly disabled on this node?
kubectl get node "$NODE" \
  -o jsonpath='{.metadata.annotations.cluster-autoscaler\.kubernetes\.io/scale-down-disabled}{"\n"}'

# 3. What is still running here? Look for bare pods (no OWNER),
#    DaemonSet pods, and pods marked safe-to-evict=false.
kubectl get pods -A --field-selector spec.nodeName="$NODE" \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,SAFE_TO_EVICT:.metadata.annotations.cluster-autoscaler\.kubernetes\.io/safe-to-evict'

# 4. PDBs that currently allow zero disruptions will block draining.
kubectl get pdb -A

# 5. The autoscaler's own summary of node groups and scale-down candidates.
kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml

# 6. The logs name the specific pod blocking removal.
#    The deployment name depends on how the autoscaler was installed; adjust it.
kubectl -n kube-system logs deployment/cluster-autoscaler --since=30m | grep "$NODE" || true

A shell script that checks a node's events, its scale-down-disabled annotation, the pods still on it, PodDisruptionBudgets, the cluster-autoscaler-status ConfigMap and the autoscaler logs to show why a node is not being removed.

Stateful dependencies and costs that sleep does not touch

Draining nodes removes the compute line, but a sleeping namespace still holds billable objects. A PersistentVolumeClaim keeps its backing disk after its pod scales to zero, and block storage is billed on provisioned capacity whether or not anything is attached to it.

For most preview environments, fixtures are the better default. Nobody needs last Tuesday's test rows, and a seeded database exposes broken migrations instead of hiding them.

Abrupt scale-down is the next risk. A database pod that exceeds its termination grace period gets killed and must run crash recovery on start. A queue broker can hold unacknowledged messages that consumers redeliver after wake. A cache comes back empty, or it comes back holding objects serialized for an older schema. Schema drift is the most common failure. If a migration lands on the branch while the namespace sleeps, the app wakes against an old schema. Make idempotent migrations a required step of the wake path.

Some costs keep running while pods sleep:

• LoadBalancer Services: each one provisions a cloud load balancer, which is billed hourly with or without healthy backends. A shared ingress with host-based routing avoids this.
• Managed databases per environment: these sit outside the cluster, so the cluster autoscaler never touches them.
• Reserved static IPs: these are often billed whether or not they are attached.

Designing the wake-up path and cold-start time

A sleeping namespace saves money only if waking it doesn't waste engineer time. There are three common ways to wake one:

• On-demand wake through a proxy. Traffic goes to a lightweight proxy that serves a "waking up" page, scales the workloads back up, and redirects once readiness checks pass. This catches every visitor, including manual testers. The cost is a new failure point in the request path, an extra network hop, and a component that needs permission to scale workloads. Give that component narrow RBAC.
• CI-triggered wake on new commits. The pipeline scales the namespace up before deploying. Nothing sits in the request path. The gap is a reviewer who opens the preview link two days later and finds it asleep.
• Manual wake. A button or chat command. It is explicit and simple, but it is one more step people forget.

Many teams pair CI wake with a proxy fallback.

Cold start is a sum of stages: node provisioning (if the autoscaler released nodes), image pulls onto fresh nodes, migrations or seed jobs, and cache warmup.

Illustrative estimate (assumed numbers, not a benchmark): wake time = node + pull + migrate + warmup = 90 s + 45 s + 30 s + 15 s = 180 s. At 10 wakes a day where someone waits, that is 30 minutes of waiting per day.

If wakes feel slow, developers will annotate their namespaces to opt out, and the savings disappear. To cut each stage:

• Node provisioning: keep a small buffer of warm node capacity.
• Image pulls: keep images small.
• Migrations: skip them when the schema hasn't changed.
• Seed jobs: persist volumes instead of reseeding.
• Cache warmup: warm caches lazily rather than blocking readiness.

When to avoid auto-sleep, alternatives, and verifying savings

Auto-sleep is the wrong tool when environments are short-lived or constantly in use. If a preview lives for a few hours during active review, there is little idle time to recover, and engineers pay a cold start on every return. It also does nothing about environments that should not exist at all. Abandoned namespaces from merged PRs are usually a larger leak than overnight idle time, because they run around the clock.

Verifying savings

Do not trust the configured schedule as proof of savings. Annotations can be overridden, controllers fail silently, and a single pod with a pod disruption budget can keep a node alive. Track the sum of CPU and memory requests over time, grouped by your preview namespace label, and plot it next to node count. Savings are real only when both lines drop during sleep windows.

Lifecycle hygiene usually saves more infrastructure money than sleep schedules, and it costs less engineering time to run. Add auto-sleep once deletion is reliable.

Copy#!/usr/bin/env bash
# Garbage-collect preview namespaces whose PR is no longer open.
# Catches environments left behind when the teardown hook on PR close/merge failed.
# Assumes each preview namespace carries labels: preview=true and pr=<PR number>.
set -euo pipefail

REPO="example-org/example-app"   # placeholder repository
DRY_RUN="${DRY_RUN:-true}"        # default to reporting only; set DRY_RUN=false to delete

kubectl get namespaces -l preview=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.pr}{"\n"}{end}' |
while IFS=$'\t' read -r ns pr; do
  # Without a PR number we cannot prove the namespace is orphaned; leave it for a human.
  if [ -z "$pr" ]; then
    echo "skip $ns: no pr label"
    continue
  fi

  # gh reports OPEN, CLOSED or MERGED. On API errors, skip rather than delete.
  state=$(gh pr view "$pr" --repo "$REPO" --json state --jq .state 2>/dev/null || echo UNKNOWN)

  case "$state" in
    OPEN|UNKNOWN)
      continue ;;
    *)
      echo "orphaned: $ns (PR #$pr is $state)"
      if [ "$DRY_RUN" = "false" ]; then
        # Deleting the namespace removes in-cluster Services (and the cloud LBs they own).
        # DNS records or LBs created outside the cluster need their own cleanup step.
        kubectl delete namespace "$ns" --wait=false
      fi
      ;;
  esac
done

A scheduled garbage-collection script that finds preview namespaces whose pull request is closed or merged and deletes them. It runs in dry-run mode by default, and it skips any namespace whose PR state cannot be confirmed.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to preview environment cost, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.