What does it mean to shut down dev environments at night, and why do nodes matter more than replicas?
Off-hours scaling has two layers. The first is workload replicas: Deployments and StatefulSets set to zero replicas outside working hours. The second is the nodes underneath them. Billing follows the second layer. A node with no application pods costs the same per hour as a full one, so scaling replicas to zero saves nothing until the cluster autoscaler or node-pool configuration removes the empty machines. Changing replica counts is the easy part. Most of the work is getting nodes to drain.
| Layer | What changes at night | Effect on the bill | What usually goes wrong |
|---|---|---|---|
| Replicas | Pods terminate | None by itself | HPAs or GitOps sync restore replicas |
| Nodes | VMs drain and are deleted | Compute cost drops | Pods that cannot be evicted keep nodes alive |
Checklist: can your nodes actually go away?
- Autoscaler scale-down is enabled for the dev node pools, and you know how long a node must sit underutilized before the autoscaler removes it.
- Minimum node-pool sizes are set low enough. If a pool's minimum is 3, the pool never shrinks below three nodes.
- PodDisruptionBudgets: check for any PDB that allows zero disruptions, such as
maxUnavailable: 0or aminAvailableequal to the replica count. These block eviction. - Pods with local storage such as
emptyDir. Depending on autoscaler settings, these can prevent a node from being removed. - System pods, such as those in
kube-system, and bare pods without a controller. Both can pin a node in place. - DaemonSets: they are not rescheduled elsewhere, but they run on every node that survives the night, so they add cost on whatever nodes remain.
Some costs stay the same however far you scale down. These include managed control-plane fees, load balancers, persistent volumes, reserved IPs and NAT gateways.
#!/usr/bin/env bash
# Audit common blockers that keep dev nodes alive after replicas hit zero.
# Requires kubectl access to the dev cluster and jq.
set -euo pipefail
echo '== PDBs currently allowing zero disruptions (block eviction) =='
kubectl get pdb -A -o json | jq -r '
.items[] | select(.status.disruptionsAllowed == 0)
| "\(.metadata.namespace)/\(.metadata.name)"'
echo '== Pods using emptyDir (may block scale-down, depending on autoscaler settings) =='
kubectl get pods -A -o json | jq -r '
.items[] | select(any(.spec.volumes[]?; has("emptyDir")))
| "\(.metadata.namespace)/\(.metadata.name) node=\(.spec.nodeName)"'
echo '== Bare pods with no controller (not recreated elsewhere, can pin a node) =='
kubectl get pods -A -o json | jq -r '
.items[] | select((.metadata.ownerReferences // []) | length == 0)
| "\(.metadata.namespace)/\(.metadata.name)"'
echo '== Pods explicitly marked not safe to evict =='
kubectl get pods -A -o json | jq -r '
.items[]
| select(.metadata.annotations["cluster-autoscaler.kubernetes.io/safe-to-evict"] == "false")
| "\(.metadata.namespace)/\(.metadata.name)"'
echo '== kube-system pods and the nodes they run on =='
kubectl get pods -n kube-system -o wide --no-headers | awk '{print $1, $7}'
echo '== DaemonSets (one pod per surviving node, adds cost on what remains) =='
kubectl get daemonsets -A
echo '== Current node count (compare with node-pool minimums) =='
kubectl get nodes --no-headers | wc -l
A read-only kubectl and jq audit that lists the PodDisruptionBudgets, emptyDir pods, bare pods, non-evictable pods, kube-system pods and DaemonSets that can stop dev nodes from draining at night.
Who should scale dev clusters to zero, and when should you avoid it?
Nightly scale-down pays off most for teams that run dedicated dev or staging clusters, work predictable hours, and deploy workloads that tolerate restarts. If everyone works within a few adjacent time zones, the cluster may sit idle for 12 or more hours a day plus weekends. Those idle hours are what you are reclaiming. Stateless services that start cleanly from config and an image registry are the easiest to stop, because nothing is lost when their pods go away.
Avoid it, or narrow it to specific namespaces, in these cases:
- Globally distributed teams. If someone is always working, there is no real off-hours window. Per-team or per-namespace schedules fit better than a cluster-wide shutdown.
- Fixed costs dominate. If most of the bill is control-plane fees, load balancers and storage, removing nodes saves little and still adds morning startup friction.
- Seeded data is expensive to rebuild. Databases loaded with anonymized production snapshots or long fixture runs may not come back cleanly. Exclude them, or confirm that the data lives on volumes that survive a scale-down.
- Heavy night-time work. Batch jobs, data migrations and load tests often run overnight because nobody else is using the cluster. Move them to a different window or exempt them explicitly.
Go/no-go checklist
- At least an eight-hour window each day with no expected users
- Node compute is a meaningful share of the cluster's cost
- Workloads restart without manual steps
- Stateful data survives scale-down or is cheap to rebuild
- Overnight jobs are rescheduled or excluded
- There is an owner and a documented way to wake the cluster early
How does kube-downscaler-style off-hours scaling work?
The scaler runs as a controller in the cluster. On each loop it reads schedule annotations from namespaces and workloads, compares the current time against them, and adjusts replica counts. In the original kube-downscaler, a namespace-wide default looks like downscaler/uptime: Mon-Fri 07:30-19:00 Europe/Berlin. A workload opts out with downscaler/exclude: "true". A team namespace can carry its own downscaler/uptime value that overrides the cluster default. Key names and precedence rules differ between forks and releases, so check the documentation for the version you deploy. Always give an explicit IANA time zone. If you leave it out, the window is evaluated in UTC or in the controller's local time, and a team in another region loses its afternoon.
Before scaling to zero, the controller writes the current replica count into an annotation such as downscaler/original-replicas. In the morning it reads that value back and restores it.
19:00 scaler sets replicas=0, saves original count
-> pods terminate
-> nodes go idle; cluster autoscaler drains and removes them
07:30 scaler restores saved count
-> pods go Pending
-> cluster autoscaler adds nodes -> pods scheduled
That saved annotation is the weak point. Suppose someone runs kubectl scale during the window. Depending on the version, the controller may treat the new count as the value to restore, or it may scale the workload back down on its next loop. A GitOps reconciler causes a similar problem: it sees zero replicas as drift and scales the workload back up overnight, so the nodes never drain. You can fix this in two ways. Configure the reconciler to ignore spec.replicas on scaled workloads, or leave replicas out of the manifests entirely.
# Namespace-wide default schedule. Key names follow the original
# kube-downscaler; forks and newer releases may use different keys,
# so confirm against the docs for the version you deploy.
apiVersion: v1
kind: Namespace
metadata:
name: team-payments-dev
annotations:
# Explicit IANA time zone; without it the window may be read as UTC.
downscaler/uptime: "Mon-Fri 07:30-19:00 Europe/Berlin"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
namespace: team-payments-dev
spec:
# replicas intentionally omitted so a GitOps reconciler does not
# treat the overnight scale-to-zero as drift and undo it.
selector:
matchLabels:
app: payments-api
template:
metadata:
labels:
app: payments-api
spec:
containers:
- name: api
image: registry.example.com/payments-api:1.0.0
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-db-proxy
namespace: team-payments-dev
annotations:
# Opt this workload out of off-hours scaling entirely.
downscaler/exclude: "true"
spec:
selector:
matchLabels:
app: payments-db-proxy
template:
metadata:
labels:
app: payments-db-proxy
spec:
containers:
- name: proxy
image: registry.example.com/db-proxy:1.0.0
A namespace uptime window with an explicit time zone, a Deployment without spec.replicas so GitOps does not undo scale-down, and a workload opted out via an exclude annotation.
What are the alternatives to kube-downscaler for off-hours scaling?
Three approaches are common, and they control different layers.
| Approach | Layer controlled | How replicas come back | HPA handling | Main failure mode |
|---|---|---|---|---|
| Annotation-driven tool | Workloads (Deployments, StatefulSets) | Original count stored in an annotation | Built-in, usually by adjusting minReplicas |
Controller pod is down or evicted when the schedule fires |
CronJob running kubectl scale |
Workloads you list explicitly | Hardcoded count in the morning job | None unless you script it | Missed job run, or a stale count after someone changes replicas |
| Cloud scheduled node-pool scaling | Nodes | Autoscaler or schedule raises the minimum | Not involved | Pods stay Pending if they come up before the nodes |
The CronJob approach uses two jobs, one for evening and one for morning. Each runs a container with kubectl that scales a label selector to 0 or back to a fixed number. It has no external dependencies. The cost is maintenance: the morning job has no memory of the previous counts, so every replica change has to be copied into it. It also needs RBAC:
- A ServiceAccount.
- A Role in each target namespace granting
getandpatchondeployments/scaleandstatefulsets/scale. getandlistondeploymentsandstatefulsets, so selectors resolve.
Node-pool scheduling is where most of the savings come from, since you pay for nodes. On its own it is a blunt tool. Draining nodes under running pods leaves those pods Pending, and the cluster autoscaler may then add capacity back.
For that reason, many teams combine two approaches:
- An annotation tool scales workloads to zero first.
- Shortly afterwards, the node pool's minimum drops to zero, and the autoscaler removes the empty nodes.
In the morning, run the same steps in reverse: bring nodes back first, then workloads.
apiVersion: v1
kind: ServiceAccount
metadata: {name: night-scaler, namespace: dev}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: night-scaler, namespace: dev}
rules:
# get/list on the workloads lets the -l selector resolve to objects
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets"]
verbs: ["get", "list"]
# the actual change goes through the scale subresource
- apiGroups: ["apps"]
resources: ["deployments/scale", "statefulsets/scale"]
verbs: ["get", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: night-scaler, namespace: dev}
subjects: [{kind: ServiceAccount, name: night-scaler, namespace: dev}]
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: night-scaler}
---
apiVersion: batch/v1
kind: CronJob
metadata: {name: scale-down-evening, namespace: dev}
spec:
schedule: "0 19 * * 1-5" # 19:00 on weekdays
timeZone: "Etc/UTC"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 3600 # still run if the job is up to an hour late
jobTemplate:
spec:
template:
spec:
serviceAccountName: night-scaler
restartPolicy: OnFailure
containers:
- name: kubectl
image: registry.example.com/tools/kubectl:stable # any image that ships kubectl
command: ["/bin/sh", "-c"]
# Morning job: copy this CronJob, set schedule "0 7 * * 1-5" and a fixed --replicas count
args: ["kubectl -n dev scale deployment -l off-hours=enabled --replicas=0 && kubectl -n dev scale statefulset -l off-hours=enabled --replicas=0"]
This manifest defines an evening CronJob that scales labelled Deployments and StatefulSets in the dev namespace to zero, along with the ServiceAccount, Role and RoleBinding it needs to resolve selectors and patch the scale subresource.
Implementation checklist: HPAs, stateful components and exclusions
HPAs
- Find every HPA in scope. An HPA with
minReplicasof 1 or more will scale a Deployment back up after your scaler sets it to zero. Either the HPA wins or the two controllers keep overwriting each other. - Choose one handling strategy. You can pause the HPA, lower its minimum for the night, or use a scaler that explicitly manages HPAs. Do not rely on the HPA being deleted and recreated by hand.
- Confirm how counts are restored. Check where the original replica count and HPA minimum are stored, for example in an annotation. Make sure a manual edit during the night does not overwrite them.
Stateful components
- Decide per component. For each dev database, queue and cache, choose one: stop it, keep it running, or move it to a managed service. Write the decision down so nobody has to guess.
- Check where data lives. Scaling a StatefulSet to zero keeps its PVCs. However, a PersistentVolume with the
Deletereclaim policy is destroyed if its claim is removed, for example during a namespace cleanup. UseRetainfor data you care about. - Check startup order. Pods return in order only under the
OrderedReadypod management policy.Parallelstarts them all at once, which can break clustered databases that need a primary first.
Exclusions
-
Keep these running overnight:
- CI runners
- ingress controllers
- DNS
- monitoring and log shipping
- certificate renewal
- the scaler itself
- any CronJobs expected to run at night
If the scaler shuts itself down, nothing wakes the cluster in the morning.
Schedules
- Use explicit IANA time zones. Do not rely on the controller's local clock.
- Account for daylight-saving shifts. Check how your tool handles the transition days.
- Cover weekends and public holidays.
- Allow per-team windows. Teams in different regions should not share a single off-hours schedule.
#!/usr/bin/env bash
# Pre-shutdown audit for one dev namespace. Read-only unless APPLY=1.
set -euo pipefail
NS="${1:-dev-team-a}"
APPLY="${APPLY:-0}"
# 1. HPAs: any MIN >= 1 will scale its target back up after scale-to-zero.
echo "== HPAs in $NS =="
kubectl get hpa -n "$NS" -o custom-columns=\
NAME:.metadata.name,TARGET:.spec.scaleTargetRef.name,MIN:.spec.minReplicas,MAX:.spec.maxReplicas
# 2. StatefulSets: only OrderedReady brings pods back one at a time, in order.
echo "== StatefulSets in $NS =="
kubectl get statefulset -n "$NS" -o custom-columns=\
NAME:.metadata.name,REPLICAS:.spec.replicas,POD_POLICY:.spec.podManagementPolicy
# 3. PersistentVolumes bound to this namespace: Delete destroys data if the PVC goes.
echo "== PV reclaim policies for $NS =="
kubectl get pv -o jsonpath="{range .items[?(@.spec.claimRef.namespace==\"$NS\")]}{.metadata.name}{' '}{.spec.claimRef.name}{' '}{.spec.persistentVolumeReclaimPolicy}{'\n'}{end}" |
while read -r pv claim policy; do
echo "$pv claim=$claim policy=$policy"
if [[ "$policy" == "Delete" && "$APPLY" == "1" ]]; then
# Switch to Retain so a namespace cleanup cannot wipe dev data.
kubectl patch pv "$pv" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
fi
done
# 4. CronJobs that run at night: exclude them or move their schedule.
# An empty TZ means the schedule follows the controller's local time zone.
echo "== CronJobs in $NS =="
kubectl get cronjob -n "$NS" -o custom-columns=\
NAME:.metadata.name,SCHEDULE:.spec.schedule,TZ:.spec.timeZone,SUSPENDED:.spec.suspend
A kubectl audit script that lists HPA minimums, StatefulSet pod management policies, PersistentVolume reclaim policies and CronJob schedules for a dev namespace. With APPLY=1, it switches Delete-policy volumes to Retain.
How do you make sure everything comes back up correctly in the morning?
Scaling down is the easy half. Most of the engineering effort goes into scale-up, where dozens of services start at once against dependencies that are not ready yet.
- Define startup tiers. Bring up databases, message brokers, and config and secret services first. Start application workloads only after those tiers report ready. Staggered schedules or ordered scale-up steps are easier to debug than one simultaneous burst.
- Rely on readiness probes and retries, not ordering alone. Tiers reduce contention, but they cannot guarantee that a dependency is reachable when a service first connects. Each service should retry its connections with backoff instead of exiting on the first failure.
- Expect crash-loops in the first weeks. These are useful. A service that crash-loops every morning is showing you a missing retry or a hard startup dependency that a long-running cluster had been hiding.
- Account for cold-start delay. Node provisioning, image pulls onto fresh nodes, and cache or JIT warm-up can add significant time before workloads are ready. Waking the cluster before working hours hides this delay, but every hour of pre-wake reduces the savings. Size the lead time against measured startup duration, not guesses.
- Schedule a wake-up check. Run a job shortly before the team starts that lists deployments and statefulsets not at their desired ready count and alerts the owning team. A broken environment should be visible in a channel before anyone tries to use it.
- Provide self-service overrides with expiry. Give engineers an annotation or a small script to keep a namespace up overnight or wake it early. Require an expiry timestamp so exclusions lapse on their own instead of accumulating into a cluster that never sleeps.
How does off-hours scaling affect cost and engineering time?
Verify savings with node count, not replica count. Plot nodes per pool over a week. The overnight line should drop to the floor you expect. Then break billing down by hour of day. If spend at 03:00 is barely lower than at 15:00, something is holding nodes up. Some costs keep running regardless: the control plane, load balancers, persistent volumes, NAT and logging.
Illustrative estimate (assumed numbers, not a benchmark)
- Assumed: 10 nodes at $0.20/hour.
- Downtime: weeknights 19:00–07:00 plus weekends, which is 60 + 48 = 108 hours/week.
- Pre-wake: 1 hour before each weekday start, 5 hours/week total.
- Formula:
saving/week = nodes_removed × hourly_price × (downtime_hours − prewake_hours) - Ideal case: 10 × 0.20 × 103 = $206/week
- Realistic case: a minimum pool size of 2 for system DaemonSets and add-ons, plus 1 node held by a pod the autoscaler won't evict. That gives 7 × 0.20 × 103 = $144.20/week, about 30% below the ideal.
| Engineering time | Up-front | Ongoing |
|---|---|---|
| Startup ordering and readiness fixes | High, one-off | Low |
| Schedules and overrides | Moderate | Small, recurring |
| Manual "please turn it off" requests | — | Removed |
The ordering work pays off during the day as well. Services that tolerate a dependency arriving late also recover better after ordinary restarts.
Periodic review checklist
- List active overrides. Flag any without an expiry date or older than a month.
- Check which namespaces were excluded most often, and ask why.
- Review morning restore failures and confirm each one has a root cause.
- Compare the overnight node floor against the expected minimum.
- Re-check that pre-wake time still matches when the team starts work.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to shut down dev environments at night, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.