How to Shut Down Dev Environments on a Schedule in Kubernetes

Why do idle dev and staging clusters cost so much?

Scheduled shutdown means scaling non-production workloads, and ideally the nodes beneath them, to zero outside working hours, then bringing them back before people start work. Teams use it because dev and staging clusters are usually sized for daytime use but billed around the clock. Cloud compute charges for every hour a node runs, whether or not anyone sends it a request. For most non-production clusters, the largest waste is not oversized pods but nights and weekends.

Illustrative estimate (assumed numbers, not a benchmark)

• Assumed: engineers use the cluster about 50 hours per week (10 hours × 5 weekdays).
• A week has 168 hours.
• Idle fraction = (168 − active hours) ÷ 168 = (168 − 50) ÷ 168 = 118 ÷ 168 ≈ 70%.

Swap in your own active hours. With node autoscaling and no minimum node count, the result is roughly the share of node-hours you could stop paying for. Real savings are smaller because some resources bill whether pods run or not:

The teams that benefit most are platform teams running per-team dev clusters with predictable working hours. In that setup, idle windows are long, easy to predict, and owned by one group that can agree on a schedule.

How does scheduled scale-down work across workloads and nodes?

Scheduled shutdown works in two layers. First, workloads scale to zero: Deployments and StatefulSets have their replica counts set to 0, so their pods terminate. Second, the node pool shrinks. Either the cluster autoscaler notices that nodes are empty and removes them, or a scheduled job sets the node group's size directly. You pay for nodes, not pods, so the second layer is where the money is saved. If you scale every Deployment to zero and the nodes stay up, the bill barely changes.

What keeps nodes alive

A node only drains cleanly if nothing on it refuses to move. The usual blockers:

• DaemonSets run a pod on every node, so a node is never literally empty. Autoscalers generally ignore DaemonSet pods when deciding whether a node is removable, but a homegrown script that checks for "no pods" will never remove anything.
• Pods using local storage, such as emptyDir volumes, can block eviction by default because the data would be lost.
• PodDisruptionBudgets that allow zero disruptions stop the drain.
• System pods in kube-system without a PDB are often treated as unsafe to evict.
• Stray unmanaged pods, created without a controller, have nothing to recreate them, so the autoscaler leaves them in place.
• Node group minimum size sets a floor. A minimum of 3 means 3 nodes stay up no matter what.

The daily sequence

Evening: scale workloads to 0. The autoscaler then drains and removes empty nodes, or a scheduled change lowers the group size to its floor.

Morning: raise the node group size first, or let the autoscaler react to pending pods. Then restore replica counts. Expect startup to take a few minutes while nodes boot and images are pulled.

CopyEVENING: SCALE DOWN                     MORNING: SCALE UP
+----------------------------+          +----------------------------+
| Layer 1: Workloads         |          | Layer 2: Node pool         |
| Deployments, StatefulSets  |          | raise node group size, or  |
| replicas set to 0          |          | autoscaler adds nodes for  |
| pods terminate             |          | pending pods               |
+-------------+--------------+          +-------------+--------------+
              |                                       |
              v                                       v
+----------------------------+          +----------------------------+
| Layer 2: Node pool         |          | Nodes boot, images pulled  |
| autoscaler removes empty   |          | (expect a few minutes)     |
| nodes, or scheduled job    |          +-------------+--------------+
| sets group size to floor   |                        |
+-------------+--------------+                        v
              |                         +----------------------------+
              |                         | Layer 1: Workloads         |
              | drain blocked by:       | restore replica counts     |
              v                         +----------------------------+
+----------------------------------------------------------------------+
| DaemonSet pods: node never looks empty to a naive pod-count script   |
| Local storage (emptyDir) pods: not evicted by default                |
| PodDisruptionBudgets that allow zero disruptions                     |
| kube-system pods without a PDB                                       |
| Unmanaged pods: no controller to recreate them                       |
| Node group minimum size: a floor of N nodes always stays up          |
+----------------------------------------------------------------------+
| If these keep nodes up, pods at 0 save almost nothing                |
+----------------------------------------------------------------------+

Scheduled shutdown in two layers: in the evening, workloads go to zero and then the node pool shrinks unless a blocker holds a node in place; in the morning, nodes come back first and replica counts are restored after.

Which scheduled scaling option should you use: CronJobs, kube-downscaler or cloud node schedules?

Each option draws the line between "what decides to scale" and "what can be scaled" differently, and that line determines how failures show up.

CronJobs suit small setups where you want no extra controller. Scope them tightly. Use a namespaced Role that grants only get, list, patch and update on the deployments/scale and statefulsets/scale subresources. Bind it with a RoleBinding to a dedicated ServiceAccount in that same namespace, and run the CronJob as that ServiceAccount. The example below shows this pattern. Avoid a ClusterRole bound cluster-wide. A bug in the script, a wrong --all-namespaces flag, or a compromised Job pod could then scale production or system workloads to zero.

kube-downscaler fits when many teams need per-namespace windows and exemptions. Annotation names and supported resource types change between releases, so check the project's current documentation before you rely on them.

Node schedules work best as a backstop that runs after workloads have scaled down. They should not be your only mechanism.

Copy# Namespaced identity and permissions: the Job can only touch dev-team-a.
apiVersion: v1
kind: ServiceAccount
metadata: {name: dev-scaler, namespace: dev-team-a}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: dev-scaler, namespace: dev-team-a}
rules:
- apiGroups: ["apps"]   # read-only lookup so kubectl can resolve --all
  resources: ["deployments", "statefulsets"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]   # the only write access: the scale subresource
  resources: ["deployments/scale", "statefulsets/scale"]
  verbs: ["get", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding   # not a ClusterRoleBinding: no reach beyond this namespace
metadata: {name: dev-scaler, namespace: dev-team-a}
subjects: [{kind: ServiceAccount, name: dev-scaler, namespace: dev-team-a}]
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: dev-scaler}
---
apiVersion: batch/v1
kind: CronJob
metadata: {name: scale-down-evening, namespace: dev-team-a}
spec:
  schedule: "0 19 * * 1-5"   # weekdays 19:00 in the controller's time zone
  concurrencyPolicy: Forbid   # never run two scale-downs at once
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          serviceAccountName: dev-scaler
          restartPolicy: Never
          containers:
          - name: kubectl
            image: registry.example.com/tools/kubectl:1.x-pinned   # placeholder; pin a digest
            # Original replica counts are NOT saved here; store them first if you need restore.
            command: ["sh", "-c", "kubectl scale deployment --all --replicas=0 -n dev-team-a && kubectl scale statefulset --all --replicas=0 -n dev-team-a"]

A namespaced ServiceAccount, Role and RoleBinding that grant only lookup and scale-subresource access in dev-team-a, plus a weekday-evening CronJob that uses them to scale that namespace's Deployments and StatefulSets to zero.

How do you handle HPAs, GitOps and stateful services during shutdown?

…drain nodes while pods are still running, or the scheduler will simply recreate them elsewhere.

HPAs and GitOps controllers

A HorizontalPodAutoscaler reconciles replica counts toward its own minReplicas and maxReplicas. If your scaler lowers a Deployment below that minimum, the HPA can raise it again on its next pass. There are two ways to avoid this. You can let the scaling tool lower the HPA minimum, if the tool supports it, or you can pause the HPA. In both cases, restore the HPA and the workload in the same step. If they are restored separately, the HPA can resize the workload before the scale-up finishes.

GitOps controllers create the same conflict. When the manifest in Git specifies replicas, the controller sees the scaled-down Deployment as drift and reverts it. To prevent this, configure the controller to ignore the replica field, or suspend sync for the namespace during the shutdown window.

Store the original replica counts on the objects themselves, for example in annotations. If the scaler keeps them only in memory and crashes mid-run, nobody knows what to restore.

Stateful services

Scaling a StatefulSet to zero stops compute charges, but its PersistentVolumes keep billing. Databases and queues also need a defined startup order on scale-up. Brokers and databases must be ready before the consumers that connect to them, or those consumers will crash-loop.

Clustered systems carry additional risk. Replicas may need to resync after a restart, and stale leader-election state can stall the cluster. If restarts in an environment regularly need manual repair, snapshotting the volumes and destroying the stack is often simpler and cheaper. You can then recreate it from the snapshot when someone needs it.

Copy#!/usr/bin/env bash
# Usage: ./dev-schedule.sh down|up <namespace>
# Original replica counts live in annotations on each object, so a crash
# mid-run never loses them. Pause HPAs and GitOps sync before running this.
set -euo pipefail
MODE="$1"; NS="${2:-dev-team-a}"
KEY="scheduler.example.com/original-replicas"
JP='{.metadata.annotations.scheduler\.example\.com/original-replicas}'

if [[ "$MODE" == "down" ]]; then
  # Stop consumers (Deployments) before the brokers and databases they use.
  for kind in deployment statefulset; do
    for name in $(kubectl get "$kind" -n "$NS" -o jsonpath='{.items[*].metadata.name}'); do
      saved=$(kubectl get "$kind" "$name" -n "$NS" -o jsonpath="$JP")
      # An existing annotation means an earlier run was interrupted. Keep it,
      # because the live count may already be 0.
      if [[ -z "$saved" ]]; then
        current=$(kubectl get "$kind" "$name" -n "$NS" -o jsonpath='{.spec.replicas}')
        kubectl annotate "$kind" "$name" -n "$NS" "$KEY=$current"
      fi
      kubectl scale "$kind" "$name" -n "$NS" --replicas=0
    done
  done
else
  # Start stateful dependencies first and wait for them to be ready, so
  # consumers don't crash-loop against a broker or database that isn't up yet.
  for kind in statefulset deployment; do
    for name in $(kubectl get "$kind" -n "$NS" -o jsonpath='{.items[*].metadata.name}'); do
      saved=$(kubectl get "$kind" "$name" -n "$NS" -o jsonpath="$JP")
      [[ -z "$saved" ]] && continue
      kubectl scale "$kind" "$name" -n "$NS" --replicas="$saved"
      kubectl rollout status "$kind" "$name" -n "$NS" --timeout=10m
      kubectl annotate "$kind" "$name" -n "$NS" "$KEY-"  # clear only after success
    done
  done
fi

A crash-safe scale-down and scale-up script. It records original replica counts in annotations before scaling to zero, and on scale-up it restores StatefulSets before the Deployments that depend on them.

How do you design a shutdown schedule developers will trust?

A schedule only saves money if teams leave it switched on. In practice, developers opt out after a few bad mornings, so design for trust first and savings second.

Start from when people actually work, not from a default 9-to-6. A team spread across time zones has a longer working day than any one office, so take the union of their hours. Weekends and public holidays are usually the largest idle blocks, and you capture them by keeping a holiday calendar in the schedule configuration. Then remove the hours that only look idle: nightly CI runs, integration suites, data loads, and on-call engineers reproducing incidents against staging.

Exceptions are unavoidable, so make them cheap and temporary. A keep-alive override should require an expiry timestamp and revert on its own when that time passes. Without an expiry, "just this week" becomes permanent and you lose the saving without anyone deciding to. For unplanned work, give developers a self-service wake action, such as a chat command or a pipeline button, so a Saturday fix doesn't mean filing a ticket with the platform team.

Cold starts are where trust is won or lost. Measure how long a full wake actually takes: node provisioning, image pulls onto empty nodes, and dependencies like databases, queues, and migrations becoming ready before the services that need them. Schedule scale-up with a comfortable margin before the first person logs on. Test the wake-up path as carefully as the shutdown path, and alert on failed mornings, because one unreliable wake-up costs more engineering time than a week of idle nodes costs in money.

How do you measure whether shutting down dev environments saved money?

Capture a baseline before the schedule goes live. Record at least two full weeks of node-hours per cluster and requested CPU and memory per namespace, so that both weekday and weekend patterns are included. Node-hours are what the cloud bill charges for. Namespace requests show which teams drive that usage. After rollout, collect the same metrics over the same period, then reconcile them against the billing export grouped by cluster tag. Pods can disappear while the cluster autoscaler keeps nodes alive because of a stray DaemonSet-blocking pod or a pod disruption budget, and only the bill shows that gap.

Track drift weekly. Count workloads that carry the exclusion annotation and record who added each one. When that number keeps rising, developers are opting out faster than you are fixing the reasons they opt out, and savings decline without anyone noticing.

Alert on every failed scale-down and scale-up job. A failed scale-down looks normal on dashboards until the invoice arrives.

Finally, compare realized savings against the compute-only estimate, because storage, load balancers and managed services keep billing overnight.

Illustrative estimate (assumed numbers, not a benchmark):

• 10 nodes at $0.20/hour, running 60 of 168 weekly hours, so 108 hours are off
• Always-on costs: $112/week (volumes, load balancers, managed database, control plane)
• Compute saving = nodes × rate × hours off = 10 × 0.20 × 108 = $216/week
• Previous total = (10 × 0.20 × 168) + 112 = $448/week
• Realized saving = 216 / 448 ≈ 48%, compared with the 64% that compute-only math suggests

Report the 48% figure. Reporting 64% sets an expectation the bill will not meet.

When should you avoid scheduled shutdown, and what are the alternatives?

Scheduled shutdown assumes there is a stretch of time when nobody needs the cluster. Several common situations break that assumption.

• Globally distributed teams. If engineers span enough time zones that someone is always working, every off-hours window is someone else's workday. You either block them or carve out so many exceptions that the schedule saves little.
• Heavy off-hours CI. Nightly integration suites, scheduled load tests and dependency rebuilds often run precisely when you would scale down. Shutting the cluster down pushes that work into the morning, where it competes with developers.
• Stateful systems that restart badly. Databases with long recovery, message brokers that rebalance partitions on restart, and search clusters that re-replicate shards can spend a large part of the morning recovering. Repeating that every day also adds risk.
• Hidden over-provisioning. If pods request four times what they use, a schedule cuts the waste for part of the week but leaves the waste in place during working hours. Fix requests first.

For staging, split the estate in two. Keep the shared services that are expensive to rebuild running, such as the database, the broker and the auth stub. Schedule or remove the per-team application environments that depend on them. Most of the cost sits in the second group.

These options can be combined. Right-sizing usually comes first because it lowers the baseline that every other approach builds on.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to shut down dev environments, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.