How much can you save if you shut down dev environments on a schedule?
A dev-cluster sleep schedule scales workloads to zero outside working hours, then lets the cluster autoscaler remove the now-empty nodes. In the morning both layers come back. Teams do this because dev and staging clusters usually sit idle at night and on weekends while still being billed for every node-hour.
Illustrative estimate (assumed numbers, not a benchmark)
Assumed inputs: 10 nodes at $0.20 per node-hour, working hours of 10 hours × 5 weekdays = 50 hours, and a 168-hour week.
Formula: weekly node cost = nodes × hourly price × hours running
| Schedule | Hours running per week | Weekly node cost |
|---|---|---|
| Always on | 168 | 10 × $0.20 × 168 = $336 |
| Weekday working hours only | 50 | 10 × $0.20 × 50 = $100 |
| Difference | 118 off-hours (~70% of the week) | $236 |
Off-hours make up the majority of the week, which is why the idea is attractive. The table overstates the real savings, though. You only save on capacity you actually release. System pods, the control plane fee and any nodes pinned by DaemonSets or PodDisruptionBudgets may stay up. Persistent volumes, load balancers, static IPs and managed databases or caches keep billing whether pods run or not. Before you trust the result, subtract these fixed costs from your own numbers.
This fits teams with idle dev and staging clusters and predictable working hours. It fits less well when engineers span many time zones, or when overnight jobs and tests depend on the environment. When it fits, development cost moves from fixed to roughly proportional to the hours people actually use the cluster.
How does a two-layer sleep schedule work?
A sleep schedule has two layers, and only the second one saves money. The first layer scales Deployments and StatefulSets to zero replicas, which removes pods. Your cloud provider bills for nodes, not pods, so an empty node costs the same as a full one. The second layer is the cluster autoscaler. It sees nodes whose remaining pods could be evicted or rescheduled and removes them. If the workloads sit at zero but the nodes keep running, you are paying almost the full daytime bill for idle machines.
Node removal fails more often than workload scaling. The usual blockers are:
- Node-group minimums. The autoscaler never shrinks a group below its configured minimum. If the minimum is 3, those three nodes run all night. The minimum has to be zero, or as low as the system workloads that must stay up allow.
- DaemonSet pods. These normally do not block removal, because the autoscaler expects them to disappear with the node. Non-DaemonSet pods in
kube-systemcan block it, depending on configuration. - PodDisruptionBudgets. A PDB that currently allows zero disruptions stops eviction, and that pins the node.
- Local storage. Pods using
emptyDirorhostPathcan make a node ineligible for removal, depending on the autoscaler's settings. Thesafe-to-evictannotation overrides this per pod.
Checklist: what keeps a node alive
- List the pods still running on each surviving node.
- Read the autoscaler's status ConfigMap and its logs for the scale-down reasons it reports per node.
- Check node events and annotations, including
scale-down-disabled. - Look at every PDB's allowed disruptions.
- Confirm the minimum size of each node group.
#!/usr/bin/env bash
# Run after the night schedule has scaled workloads to zero.
# Shows why the cluster autoscaler is not removing the nodes that are still up.
set -euo pipefail
CA_NS=kube-system # namespace where cluster-autoscaler runs (placeholder)
CA_DEPLOY=cluster-autoscaler # deployment name varies by install (placeholder)
for node in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
echo "=== $node ==="
# 1. Pods still on the node. DaemonSet-owned pods normally do not block removal.
# Look at the rest, especially non-DaemonSet pods in kube-system.
kubectl get pods -A --field-selector spec.nodeName="$node" \
-o custom-columns='NS:.metadata.namespace,POD:.metadata.name,OWNER:.metadata.ownerReferences[0].kind'
# 3. Annotations: look for cluster-autoscaler.kubernetes.io/scale-down-disabled=true
kubectl get node "$node" -o jsonpath='{.metadata.annotations}'; echo
# 3. Recent events on the node, including autoscaler scale-down messages
kubectl get events -A --field-selector involvedObject.kind=Node,involvedObject.name="$node"
done
# 2. Autoscaler status ConfigMap (default name) and recent scale-down log lines
kubectl -n "$CA_NS" get configmap cluster-autoscaler-status -o yaml
kubectl -n "$CA_NS" logs deployment/"$CA_DEPLOY" --since=1h | grep -i 'scale.down' | tail -n 50
# 4. PDBs that allow zero disruptions block eviction and keep their nodes up
kubectl get pdb -A \
-o custom-columns='NS:.metadata.namespace,PDB:.metadata.name,ALLOWED:.status.disruptionsAllowed'
# 5. Node-group minimums are set in your cloud provider's node-group config,
# or in the autoscaler's --nodes=MIN:MAX:NAME flags. Check that MIN is 0 where possible.
A read-only script that works through the checklist with kubectl: remaining pods and their owners, node annotations and events, the autoscaler's status ConfigMap and logs, and PDB allowed disruptions.
How do you implement off hours scaling with kube-downscaler?
kube-downscaler is a controller that runs inside the cluster and periodically compares the current time against a schedule. When a workload is outside its uptime window, the controller scales it to zero, or to a configured minimum. When the window opens again, it scales the workload back up. Because it only changes replica counts, the cluster autoscaler from the previous section does the actual saving: empty nodes become eligible for removal.
Declaring the schedule
You set schedules at three levels, and the more specific level wins:
- Cluster default, passed to the controller at startup. This is the baseline for every namespace that has no schedule of its own.
- Namespace annotation, which overrides the default for one team or environment.
- Workload annotation, which overrides the namespace for a single Deployment or StatefulSet.
A schedule is either an uptime window ("run only during these hours") or a downtime window ("stop during these hours"). Each window lists days, a time range and an explicit timezone. Always include the timezone. If you leave it out, the schedule is interpreted in the controller's clock, usually UTC, and a team in another region will find its environment asleep at mid-morning.
Restoring replicas
Before scaling a workload down, the controller writes the original replica count into an annotation on that workload. On wake-up it reads the value back and restores it. Two practical consequences follow:
- Do not strip that annotation during a GitOps sync, or the controller cannot restore the workload correctly.
- If your deploy tooling reasserts
replicaswhile the workload is asleep, the workload will wake up early.
Exclusions
You can exclude whole namespaces with a controller flag. Start with system namespaces and the namespace the controller itself runs in. You can exclude individual workloads with an exclude annotation, which is useful for something like a shared auth stub that other environments depend on.
Annotation names, schedule syntax and the supported resource kinds differ between releases and forks. Check the project's documentation for the version you deploy before you rely on any of them.
# Annotation names follow the kube-downscaler project's documented convention.
# Verify them against the docs for the version you deploy.
# Namespace level: dev workloads run only during working hours in Berlin.
# This overrides the cluster default passed to the controller at startup.
apiVersion: v1
kind: Namespace
metadata:
name: team-a-dev
annotations:
downscaler/uptime: "Mon-Fri 07:30-19:00 Europe/Berlin"
---
# Workload level: this Deployment overrides the namespace schedule.
# It stops only overnight and keeps one replica instead of scaling to zero.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: team-a-dev
annotations:
downscaler/downtime: "Mon-Sun 22:00-06:00 Europe/Berlin"
downscaler/downtime-replicas: "1"
# The controller writes downscaler/original-replicas here while the
# workload is asleep. Keep your GitOps tool from pruning it.
spec:
replicas: 3
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: registry.example.com/team-a/api:1.0.0
---
# Exclusion: a shared database stays up regardless of any schedule.
# metadata:
# annotations:
# downscaler/exclude: "true"
Namespace and workload annotations that set a timezone-aware uptime window, a narrower downtime override with a minimum replica count, and a per-workload exclusion. Check annotation names against the kube-downscaler documentation for your version.
What are the alternatives to kube-downscaler?
The simplest alternative needs no extra component. Two CronJobs run kubectl scale: one sets workloads to zero in the evening, the other restores them in the morning. The job's service account is bound with a Role, not a ClusterRole, in each target namespace. That Role grants get, update and patch on deployments/scale and statefulsets/scale, plus list on deployments and statefulsets so the job can find what to scale. It cannot edit pod templates, read Secrets or touch other namespaces. If the token leaks, the worst outcome is a scaled-down environment.
The weakness is state. With scale-only rights the job cannot record the original replica count on the object. The wake-up job therefore needs fixed counts, kept in Git or a ConfigMap, and those drift from what teams actually run.
| Concern | CronJob + kubectl scale |
Dedicated downscaler |
|---|---|---|
| Original replica counts | Hard-coded in the wake-up job | Stored on the object, restored automatically |
| Per-namespace overrides | One job or argument list per namespace | Annotations on the namespace or workload |
| HPA support | Manual; the HPA fights a zero scale | Usually handled, depending on version |
| Failure visibility | Job status and exit codes | Controller logs; you must alert on them yourself |
A downscaler is a privileged component. It typically needs patch on workloads cluster-wide to write its annotations. Scope it down in three ways:
- Bind it per namespace with RoleBindings instead of one ClusterRoleBinding.
- Grant only the resource kinds you actually schedule.
- Run it in its own namespace, and exclude that namespace and
kube-systemfrom its targets.
# Evening scale-down for one dev namespace. Repeat the Role and RoleBinding per target namespace.
apiVersion: v1
kind: ServiceAccount
metadata: {name: night-scaler, namespace: dev-team-a}
---
# A Role, not a ClusterRole: rights end at this namespace boundary.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: scale-only, namespace: dev-team-a}
rules:
- apiGroups: ["apps"]
resources: ["deployments/scale", "statefulsets/scale"] # replica count only
verbs: ["get", "update", "patch"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets"] # list to find targets; no update on pod templates
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: night-scaler, namespace: dev-team-a}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: scale-only}
subjects: [{kind: ServiceAccount, name: night-scaler, namespace: dev-team-a}]
---
apiVersion: batch/v1
kind: CronJob
metadata: {name: scale-down, namespace: dev-team-a}
spec:
schedule: "0 19 * * 1-5" # weekdays 19:00
timeZone: "Europe/Berlin"
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 2 # repeated failures show up as a failed Job
template:
spec:
serviceAccountName: night-scaler
restartPolicy: Never
containers:
- name: kubectl
image: registry.example.com/tools/kubectl:1.30 # any image with kubectl
command: ["sh", "-c", "kubectl scale deployment,statefulset --all --replicas=0 -n dev-team-a"]
# The morning job is identical but sets fixed counts, e.g. kubectl scale deployment/api --replicas=2.
A namespace-scoped ServiceAccount, a Role limited to the scale subresource plus list, and a weekday-evening CronJob that scales every Deployment and StatefulSet in dev-team-a to zero.
How do you handle HPAs, StatefulSets and managed services while the cluster sleeps?
HPAs
An HorizontalPodAutoscaler owns the replica count of its target. If the downscaler sets a Deployment to zero while its HPA has minReplicas: 1, the HPA scales it back up. You end up paying for pods all night and getting churn in the event log. The schedule therefore has to act on the HPA itself. It can lower minReplicas, or it can scale the target in a way the HPA ignores. Some downscaler implementations handle HPAs and others skip them silently. Check your tool's documentation and test one HPA-backed workload before you trust the schedule.
StatefulSets and volumes
Decide for each StatefulSet whether it should sleep. A dev Postgres that can lose its in-memory state is a good candidate. A Kafka or etcd cluster that runs slow recovery or rebalancing on every start may cost more in morning delays than it saves overnight. Scaling to zero removes the pods but leaves the PersistentVolumeClaims in place. The disks behind them keep billing. Count that storage cost as fixed.
Managed services
A managed database or cache outside the cluster does not notice that your pods are gone. If your provider supports stopping these services, give each one its own stop/start schedule. Align it with the pod schedule so that apps don't wake up before their dependencies are available.
Replica drift and GitOps
If someone deploys or scales manually during the sleep window, the workload wakes up early or wakes to the wrong count. A GitOps reconciler can also revert the change back to the replica count declared in Git within minutes. You can exclude replicas from drift detection, or you can drive the sleep schedule through Git so the reconciler applies it instead of fighting it.
# Argo CD Application for a dev namespace that sleeps overnight.
# Goal: let the downscaler change replica counts without the
# reconciler reverting them to the values declared in Git.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: dev-team-a
namespace: argocd
spec:
project: default
source:
repoURL: <your-git-repo-url>
targetRevision: main
path: envs/dev/team-a
destination:
name: in-cluster
namespace: team-a-dev
# Don't report replica changes made by the scheduler as drift.
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas
# Include only the StatefulSets you decided should sleep;
# their PVCs stay in place and keep billing either way.
- group: apps
kind: StatefulSet
jsonPointers:
- /spec/replicas
# Only needed if your downscaler patches HPA minReplicas.
# Without the HPAScaleToZero feature gate, minReplicas must be >= 1,
# so check how your tool takes an HPA-backed workload to zero.
- group: autoscaling
kind: HorizontalPodAutoscaler
jsonPointers:
- /spec/minReplicas
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
# Also keep the ignored fields untouched during syncs, so a deploy
# in the sleep window doesn't wake workloads at the Git replica count.
- RespectIgnoreDifferences=true
An Argo CD Application that stops treating scheduler-managed replica and HPA minimum fields as drift, so self-heal and syncs leave the overnight scale-down in place.
When should you avoid putting staging to sleep?
A schedule only saves money if nobody needs the environment while it is off. Several situations break that assumption.
Distributed teams. With engineers in Europe, North America and Asia, one team's night is another team's afternoon. A single shared staging cluster then has no window where everyone is offline. Forcing a window means some people wait for wake-ups every day. If teams own separate namespaces, give each one a window in its own time zone instead of a cluster-wide schedule.
Work that runs at night. Nightly end-to-end suites, data backfills and CronJobs often run off-hours because the cluster is quiet. When the downscaler removes their dependencies, those jobs can fail in ways nobody notices. A Job may time out against a missing service, or a CronJob may never start because no node can take it. Either exempt these workloads or move their dependencies into a namespace that stays up.
Cold-start cost. Each wake-up costs time: node provisioning, image pulls, cache warm-up and database migrations. Illustrative estimate: assume 20 engineers each lose 10 minutes per morning. That is roughly 70 engineer-hours a month, which can exceed the compute cost of a small staging cluster.
| Situation | Better choice |
|---|---|
| One team, one time zone, idle nights | Cluster-wide schedule |
| Several teams with separate namespaces | Per-namespace windows |
| Nightly jobs or backfills | Exempt the jobs and their dependencies |
| Slow wake-up, small cluster | No schedule, or right-size instead |
| Staging used for on-call reproduction | No schedule |
How do you make wake-up fast and verify the schedule is working?
Give engineers a documented wake-up path
People will route around a schedule if waking an environment early is hard. Document one override and point to it from the namespace README and the team channel. That override can be setting downscaler/exclude on the namespace, or a manual job that scales a namespace back up. Make the job idempotent so running it twice does no harm.
Tell users how long a cold start takes, and measure it rather than guessing. The wait has three parts: the cluster autoscaler provisioning nodes, image pulls on nodes with empty caches, and readiness probes passing. As an illustrative estimate, if nodes take a few minutes to join and images take another minute or two, a 10-minute wake-up is a realistic expectation to publish. Pre-pulling large images or keeping a small warm node pool can shorten it.
Wake things in dependency order. The downscaler restores all workloads at once, so stateless services can start before their databases or queues are ready and then crash-loop. Fix this in two places:
- In the manual job, scale the data tier first and wait for
kubectl rollout statusbefore scaling the services that depend on it. - In each service, use readiness checks that tolerate a dependency arriving late.
Verify each morning
Check two things every morning:
- Node count dropped overnight. Look at the node count metric over the night. It should show a clear trough.
- Replica counts were restored. Compare current replicas against the
downscaler/original-replicasannotation the downscaler stored.
Alert on three failure cases:
- Any namespace that still has replicas running during the sleep window.
- Any namespace still at zero replicas after the wake window.
- The downscaler pod restarting, or not logging any scaling actions.
Govern exemptions
Make sleeping the default. To stay awake, a namespace needs an annotation with an owner, a reason and an expiry date. Review these exemptions each month and remove the stale ones.
A predictable wake-up, plus alerts that catch problems before people do, means fewer "is staging down?" threads and fewer manual restarts.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to shut down dev environments schedule, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.