What does it mean to scale a dev environment to zero, and where does the money go?
To scale a dev environment to zero means setting the replica count of Deployments and StatefulSets in non-production namespaces to zero outside working hours, then restoring them in the morning. Off-hours scheduling is the policy layer on top: which namespaces, which hours, which time zone, and which workloads are exempt.
Zero replicas alone save nothing. You pay for nodes, not pods. The savings appear only when the cluster autoscaler sees underutilized nodes, drains them, and removes them from the node group. If the node group has a minimum size above zero, or something on each node blocks eviction, the bill stays the same.
Compute versus always-on costs
Scaling to zero shrinks compute: VM instances and any per-node licensing. Several costs keep running overnight:
- Control plane: managed Kubernetes offerings often charge a flat per-cluster fee whether nodes exist or not.
- Load balancers: each
Serviceof typeLoadBalancerkeeps its cloud load balancer and hourly charge. - Persistent disks: volumes behind PersistentVolumeClaims are billed for provisioned capacity while detached.
- NAT gateways and egress: NAT usually has an hourly component plus per-GB processing. Egress drops at night but the gateway still costs money.
Who it fits
The approach fits teams with dev and staging clusters that sit mostly idle outside business hours and run on autoscaled node pools. It fits poorly for teams spread across many time zones, for clusters that run nightly batch or integration jobs, and for fixed-size node pools, where scaling down pods leaves the nodes in place.
OFF-HOURS SCHEDULE (namespaces, hours, time zone, exemptions)
+----------------------------------+
| Scale Deployments/StatefulSets |
| in dev namespaces: replicas -> 0 |
+----------------------------------+
|
v
+----------------------------------+ blocked if:
| Pods terminate; nodes now idle | - node group min size > 0
| (no savings yet: billed per node)| - pods that block eviction
+----------------------------------+ => nodes stay, bill unchanged
|
v
+----------------------------------+
| Cluster autoscaler drains and |
| removes underutilized nodes |
+----------------------------------+
|
v
+----------------------------------+
| COMPUTE SAVED: VM instances, |
| per-node licensing |
+----------------------------------+
ALWAYS-ON OVERNIGHT (unaffected by scale-to-zero)
+---------------+ +---------------+ +---------------+ +---------------+
| Control plane | | LoadBalancer | | Persistent | | NAT gateway + |
| per-cluster | | Services keep | | disks billed | | egress (hourly|
| fee | | cloud LBs | | while detached| | + per-GB) |
+---------------+ +---------------+ +---------------+ +---------------+
Scaling replicas to zero cuts compute cost only after the cluster autoscaler removes the idle nodes, while control plane, load balancer, disk and NAT charges continue overnight.
How does overnight scale-to-zero work end to end?
The mechanism is a chain of separate Kubernetes controllers. Each one reacts to the step before it, and none of them knows about the schedule except the first.
- The scheduler writes the replica count. A controller such as kube-downscaler, or a CronJob running
kubectl scale, records each workload's current replica count (usually in an annotation) and then setsspec.replicasto zero through the workload'sscalesubresource. - Workload controllers delete pods. The Deployment's ReplicaSet controller or the StatefulSet controller sees the new desired count and terminates pods. Each pod gets its normal graceful shutdown:
preStophooks run, then SIGTERM, and SIGKILL if the termination grace period runs out. - Nodes become underutilized. Once the pods are gone, the nodes they ran on carry only DaemonSet pods and whatever was left behind. Requested CPU and memory on those nodes drops.
- The cluster autoscaler removes nodes. The autoscaler periodically checks for nodes whose requested resources are below its utilization threshold and whose remaining pods can all be evicted or ignored. After a node has been unneeded for a configured period, the autoscaler cordons it, drains it, and asks the cloud provider to shrink the node group. The threshold and delay are configurable, so check the values your autoscaler runs with. Deleting the VM is the step that stops the compute charge.
- Morning restore reverses the chain. The scheduler reads the saved count back and sets
spec.replicas. New pods are created and goPendingbecause no node has room for them. The autoscaler sees unschedulable pods and scales the node group up. Nodes boot and join the cluster, images are pulled, containers start, and pods become ready once their readiness probes pass.
The schedule controls only step 1 and step 5. Every step in between depends on cluster configuration: node group minimums, PodDisruptionBudgets, eviction annotations, and node provisioning time. Most of the rest of this article covers those dependencies.
When should you scale dev environments to zero?
Use overnight scale-to-zero when most of the following are true:
- The environment is non-production. Dev, staging, QA, and per-branch preview namespaces are the usual candidates. Nobody is paged when they are down.
- There is a real idle window. The people who use the environment work in one time zone or a few adjacent ones, so there is a block of hours every night, and usually the whole weekend, when nobody makes requests.
- Node pools autoscale and can shrink far enough. The node group's minimum size is zero, or low enough that removing the dev workloads frees whole nodes. Without that, scaling pods saves nothing.
- Compute is a large share of the environment's bill. If VM instances dominate cost, and not load balancers, disks, or a per-cluster fee, removing nodes makes a visible difference.
- Workloads restart cleanly. Services are stateless, or their state lives on persistent volumes or in managed services that survive the night. Startup does not need manual steps.
- Morning startup fits before people arrive. Node provisioning, image pulls, and readiness can finish in the gap between the scale-up time and the start of the working day.
A good first target is a single dev namespace with stateless services, owned by one team. It shows quickly whether nodes really leave overnight before the pattern spreads to the rest of the cluster.
When should you avoid scaling to zero?
Skip the approach, or limit it to a subset of namespaces, in these situations:
- Production, and anything that serves real users or partners. This includes environments that external systems call on their own schedule, such as webhook receivers or partner sandboxes that expect your endpoint to answer.
- No common idle window. Teams spread across many time zones may have no hours when nobody uses the environment. A schedule that covers everyone's working day saves little, and one that doesn't interrupts someone's working day.
- Overnight work runs in the cluster. Nightly batch jobs, long integration or load test suites, data imports, and queue consumers that must keep up with producers all need the cluster at night. Exclude them, or the schedule breaks them.
- Fixed-size node pools. If the node group cannot shrink, or its minimum already equals the nighttime footprint, the nodes stay and the bill stays.
- Always-on costs dominate. When most of the cost comes from the control plane fee, load balancers, persistent disks, or NAT gateways, removing nodes changes little. Consolidating load balancers or deleting unused volumes may save more.
- Startup is slow or fragile. Long migrations, large data restores, licence checks, or services that need a hand-run sequence to come up cost engineering time every morning. That time can exceed the compute saved.
- Stateful clusters that do not tolerate restarts. Message brokers or databases that rely on quorum can come back in a bad state after an uncoordinated stop. Keep them running, or move them to managed services, before you schedule the rest.
If several of these apply, consider the alternatives covered later: keep the environment running and right-size it, or delete and recreate it from code if it is fully reproducible.
What keeps nodes alive after pods scale to zero?
On autoscaled pools, scaling Deployments to zero only frees capacity. The bill drops when the autoscaler removes nodes, and it removes a node only when every remaining pod on it can be evicted or ignored. Treat this as a workload-readiness problem. One leftover pod per node is enough to keep the whole pool running.
| Blocker | Why it keeps the node | Typical fix |
|---|---|---|
| Node pool minimum | Autoscaler never shrinks below the configured floor | Set dev pool minimum to zero where the provider allows it |
| PodDisruptionBudget | Eviction would violate minAvailable; a PDB on a single-replica workload blocks indefinitely |
Use maxUnavailable or drop PDBs in dev |
Local storage (emptyDir, hostPath) |
Cluster Autoscaler assumes the data matters | Annotate safe-to-evict if the data is disposable |
| Bare pods (no controller) | Nothing would recreate them | Delete, or wrap in a Deployment or Job |
safe-to-evict: "false" |
Explicit opt-out | Remove it from dev manifests |
| DaemonSets | Don't block removal, but count toward utilization by default | No action needed |
These blockers create false savings. Dashboards show zero replicas while the nodes keep running and billing at full rate. The trap is worse when only system pods remain. Cluster Autoscaler treats kube-system pods without a PDB as blocking by default, so add PDBs for them.
Audit before choosing tooling. Run kubectl get pdb -A, then list pods that have no ownerReferences or that mount emptyDir. After that, scale a namespace down by hand and watch the autoscaler's events and status for the reason each node stays up.
#!/usr/bin/env bash
set -euo pipefail
PODS=$(kubectl get pods -A -o json | jq '[.items[] | select(.status.phase == "Running" or .status.phase == "Pending")]')
echo "== PDBs that currently allow zero disruptions (block eviction)"
kubectl get pdb -A -o json | jq -r '.items[]
| select(.status.disruptionsAllowed == 0)
| "\(.metadata.namespace)/\(.metadata.name)"'
echo "== Bare pods with no controller (nothing would recreate them)"
echo "$PODS" | jq -r '.[]
| select((.metadata.ownerReferences // []) | length == 0)
| "\(.metadata.namespace)/\(.metadata.name) on \(.spec.nodeName)"'
echo "== Pods with local storage (emptyDir or hostPath)"
echo "$PODS" | jq -r '.[]
| select(any(.spec.volumes[]?; .emptyDir != null or .hostPath != null))
| "\(.metadata.namespace)/\(.metadata.name) on \(.spec.nodeName)"'
echo "== Pods that explicitly opt out of eviction"
echo "$PODS" | jq -r '.[]
| select(.metadata.annotations["cluster-autoscaler.kubernetes.io/safe-to-evict"] == "false")
| "\(.metadata.namespace)/\(.metadata.name)"'
echo "== Cluster Autoscaler status (if it publishes the status ConfigMap)"
kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml 2>/dev/null \
|| echo "status ConfigMap not found; check how your autoscaler is deployed"
echo "== Recent events recorded against nodes"
kubectl get events -A --field-selector involvedObject.kind=Node --sort-by=.lastTimestamp | tail -n 20
A read-only bash audit that uses kubectl and jq to list blocking PDBs, bare pods, pods with local storage, and pods annotated safe-to-evict "false", and then shows the autoscaler's status and node events that explain why each node stays up.
How does kube-downscaler work?
kube-downscaler is a single controller that runs a reconciliation loop on a fixed interval. On each pass it lists Deployments, StatefulSets and other supported workload types, works out the schedule that applies to each one, and compares that schedule with the current time.
Schedules are resolved in layers. The controller's own arguments set a cluster-wide default uptime or downtime window. An annotation on a namespace overrides that default for everything in the namespace. An annotation on an individual workload overrides both. With this layering, a platform team can set one office-hours window for the cluster, and a team that works different hours can change its own namespace without touching shared configuration.
When a workload falls outside its uptime window, the controller writes the current replica count into an annotation such as downscaler/original-replicas. It then scales the workload to zero, or to a floor you set with something like downscaler/downtime-replicas. When the window opens again, it reads that annotation back and restores the original count. Because the original count lives on the object, a controller restart in the middle of the night does not lose it.
Exclusions use the same annotations. Set downscaler/exclude: "true" on a namespace to skip all of it, or on a single Deployment to keep only that workload running. Annotation names and accepted values have changed between forks and releases, so check them against the project's documentation for the version you deploy.
The controller also has a cost. It needs cluster-wide permission to read and patch workloads and namespaces. A compromised or misconfigured instance can therefore scale down anything in the cluster, production workloads included if they share the cluster. Limit its scope where the project supports it, and treat its ServiceAccount with the same care as any other credential that can write to the cluster.
apiVersion: v1
kind: Namespace
metadata:
name: team-payments-dev
annotations:
# Overrides the cluster-wide default window for every workload here.
downscaler/uptime: "Mon-Fri 07:00-20:00 Europe/Berlin"
---
apiVersion: v1
kind: Namespace
metadata:
name: shared-ci
annotations:
# Skip this whole namespace; nothing in it is scaled.
downscaler/exclude: "true"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mock-ledger-api
namespace: team-payments-dev
annotations:
# Overnight, keep one replica instead of scaling to zero.
downscaler/downtime-replicas: "1"
# Written by the controller at scale-down time and read back on
# restore. Do not set it by hand.
# downscaler/original-replicas: "3"
spec:
replicas: 3
selector:
matchLabels:
app: mock-ledger-api
template:
metadata:
labels:
app: mock-ledger-api
spec:
containers:
- name: api
image: registry.example.com/mock-ledger-api:1.0.0
Namespace and workload annotations that override the cluster default schedule, exclude a whole namespace, and set a one-replica floor for a single Deployment during downtime.
What are the alternatives to kube-downscaler for off-hours scheduling?
The smallest alternative is a pair of CronJobs: one in the evening, one in the morning. The evening job uses a ServiceAccount bound by a namespaced Role to get, list and patch on Deployments, StatefulSets and their scale subresources. It selects workloads by a label such as off-hours=enabled, writes each current .spec.replicas into an annotation, then runs kubectl scale --replicas=0. The morning job reads the annotation back and restores the original count.
This works on day one, but the edge cases add up quickly:
- Double runs. A retried or double-fired evening run overwrites the saved count with zero unless the script skips workloads that already carry the annotation.
- Partial failure. If the job fails partway through, half the namespace stays down, so the script needs idempotent retries.
- Overnight deploys. A deploy during the night leaves a stale annotation behind.
- HPAs. Autoscalers reconcile against the change.
- Missed schedules. These need
startingDeadlineSecondsand alerting.
Handle all of that and you have rewritten the downscaler without its tests.
Two other options sit at the extremes. You can leave everything running and accept the cost. Or you can delete whole environments and recreate them from GitOps or infrastructure-as-code each morning. Recreation saves the most, but it only works if the environment, including seed data, is fully reproducible.
| Approach | Auditability | Brittleness | RBAC scope | Maintenance |
|---|---|---|---|---|
| kube-downscaler | Annotations plus controller logs | Low; edge cases handled upstream | Often cluster-wide; restrict where possible | Upgrades only |
CronJob + kubectl scale |
Job logs and annotations | High until hardened | One namespace via Role | Ongoing, owned by you |
| Keep running | Nothing to audit | None | None | None |
| Delete and recreate | Git history | Depends on reproducibility | Broad create/delete rights | High upfront |
#!/usr/bin/env bash
set -euo pipefail
NS="${TARGET_NAMESPACE:-dev}"
SELECTOR="off-hours=enabled"
KEY="off-hours/original-replicas"
MODE="${1:?usage: off-hours.sh down|up}"
for kind in deployment statefulset; do
for name in $(kubectl get "$kind" -n "$NS" -l "$SELECTOR" \
-o jsonpath='{.items[*].metadata.name}'); do
saved=$(kubectl get "$kind" "$name" -n "$NS" \
-o jsonpath="{.metadata.annotations.$KEY}")
if [ "$MODE" = "down" ]; then
# Already annotated means a previous or double-fired run got here first.
# Skipping keeps the real count instead of overwriting it with 0.
[ -n "$saved" ] && continue
current=$(kubectl get "$kind" "$name" -n "$NS" -o jsonpath='{.spec.replicas}')
# Save before scaling, so a crash between the two steps loses nothing.
kubectl annotate "$kind" "$name" -n "$NS" "$KEY=$current"
kubectl scale "$kind" "$name" -n "$NS" --replicas=0
else
# No annotation: never scaled down, or someone deployed overnight.
[ -z "$saved" ] && continue
kubectl scale "$kind" "$name" -n "$NS" --replicas="$saved"
# The trailing '-' removes the annotation so tonight's run saves a fresh count.
kubectl annotate "$kind" "$name" -n "$NS" "$KEY-"
fi
done
done
A minimal scale-down and restore script for the evening and morning CronJobs. It stores each workload's replica count in an annotation, skips workloads it has already processed, and leaves comments marking the edge cases it still does not handle.
How do you handle HPAs, GitOps and stateful services during scale-down?
An HPA owns its target's replica count, so any schedule that also writes that field competes with it. When spec.replicas is exactly zero, the HPA controller stops acting on the workload. Any nonzero value below minReplicas is raised back to the floor, though. In the morning, both the HPA and your scheduler try to write the same field. Pick one handling explicitly:
- Lower
minReplicasfor the night and restore it at startup. - Remove or pause the HPA with the same schedule that scales the workload.
- Use tool-level HPA support only if your tool's documentation describes exactly what it changes on the HPA object.
GitOps controllers create a similar conflict. If the manifest in Git declares replicas: 3, the next sync undoes your scale-down. Drop spec.replicas from manifests for workloads that are autoscaled or scheduled, or configure the tool to ignore that field (Argo CD's ignoreDifferences is one example). Operators work the same way. They reconcile the StatefulSet from their custom resource, so scale through the custom resource, or the operator will restore the replicas.
| Component | Overnight decision | Why |
|---|---|---|
| Stateless services | Scale down | Cheap to restart, no ordering |
| Dev databases (StatefulSet) | Scale down if data is reproducible | Volumes persist, so the data survives |
| Message brokers | Keep running or scale as one unit | Quorum and leader election break on partial restarts |
| Shared, heavy state | Move to managed services | Removes ordering risk from the schedule |
Scaling to zero does not stop storage costs. Persistent volumes keep billing overnight. At startup, bring up data stores and brokers first, and wait until leader election finishes before starting the services that depend on them.
How should you design the schedule and make morning scale-up reliable?
Schedule around people, not the cluster clock
Define the schedule in the team's timezone. A UTC cron that suits one office will shut down another team's environment at midday. For distributed teams, set the uptime window to cover the earliest start through the latest finish, or give each team its own namespaces and schedules. Keep weekends off by default, and maintain a holiday list so that idle days are not billed as workdays.
Provide one self-service override, such as a namespace annotation or a documented command, and give it a required expiry timestamp. An override with no expiry tends to stay in place, and the savings disappear one namespace at a time. When an engineer can keep an environment up for an evening without asking anyone, there are no tickets and no manual restarts, and teams have less reason to opt out permanently.
Some workloads must keep running overnight: scheduled jobs, long test suites, receivers for external webhooks, and queue consumers that would let backlogs grow. Exclude them explicitly, or move them to a separate namespace that stays up.
Make morning scale-up boring
Scale-up fails in predictable ways:
- Node provisioning and image pulls. Begin 20–30 minutes before people arrive.
- Readiness probes. Make them reflect real dependencies so traffic waits for warm pods.
- Migrations on startup. Run them in a single job, not in every replica.
- Cold caches. Expect slow first requests.
Stagger scale-up across namespaces so the API server, the registry and any shared database are not hit by every environment at once.
How do you verify the savings and keep the setup safe?
Illustrative estimate (assumed numbers, not a benchmark)
Formula: weekly savings = (nodes at peak − nodes that stay up) × hourly node price × hours off per week
Assumed inputs: 10 worker nodes at $0.40/hour. Off 14 hours each weeknight (70 hours) plus the full weekend (48 hours), for 118 hours off per week. Two nodes stay up for system pods and the downscaler.
(10 − 2) × $0.40 × 118 = $377.60 per week
Persistent volumes, load balancers, any control plane fee and the two remaining nodes still bill overnight, so leave them out of the savings.
Confirm nodes actually left
The schedule only tells you what should happen. Each morning, check what did happen. In the cluster autoscaler's events and its status ConfigMap in kube-system, look for scale-down activity and for nodes it flagged as unremovable, along with the reason. Then graph node count over time from kube-state-metrics or your provider's node metrics. If the line stays flat at night, a pod is pinning a node. Common causes are a strict PodDisruptionBudget, local storage, or a workload nobody excluded.
Guardrails
- Exclude system namespaces, ingress controllers, the downscaler itself and CI runners. If you scale any of these to zero, you lose the tool that brings everything back, or you break overnight pipelines.
- Alert on failed returns. After scale-up, compare desired replicas with available replicas, and page someone if they still differ after a grace period.
- Label ownership. Require labels for team and opt-out reason, so each exclusion has a named owner and someone who can explain it.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to scale dev environment to zero, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.