Where does the money go when a dev cluster sits idle at night?

An off-hours schedule scales a non-production cluster's workloads down to zero outside working hours and brings them back before engineers start. Teams use it because dev clusters are sized for daytime use, and the nodes keep billing overnight and on weekends.

The schedule only saves money on resources that disappear when pods do. Everything else keeps charging.

Cost item Removed by scale-to-zero? Why
Worker node compute Yes, if the autoscaler removes empty nodes Billed per instance-hour
Managed control plane fee No Flat hourly charge per cluster
Load balancers No Provisioned by Service objects that still exist
Persistent volumes No Storage is billed whether or not it is mounted
NAT gateways and egress Partly The hourly gateway charge stays; traffic charges drop
Managed databases No Run outside the cluster

Illustrative estimate

Assumed inputs: 10 worker nodes at $0.20 per node-hour. The cluster is down 12 hours each weekday night and all weekend.

  • Hours per week: 168
  • Downtime: (12 × 5) + 48 = 108 hours, or about 64% of the week
  • Full-week node cost: 10 × $0.20 × 168 = $336
  • Node cost saved: 10 × $0.20 × 108 = $216 per week

This is an illustrative estimate, not a measurement. Real savings depend on how quickly empty nodes are removed and on any nodes that cannot be drained.

Teams whose bill is mostly node compute see the biggest reduction. If most of your spend goes to managed databases, load balancers, or the control plane fee, the 64% applies only to a small part of the bill.

How does scaling to zero actually remove nodes?

Scaling a Deployment to zero removes pods, not machines. The schedule sets replicas to zero, the pods receive SIGTERM and exit, and the nodes sit empty or nearly so. The node autoscaler then decides whether each node can go. It removes a node only when every remaining pod can be evicted and rescheduled. If one pod blocks eviction, the node stays and you keep paying for it.

The common blockers:

  • Pods without a controller. A bare pod started with kubectl run or left over from a debug session has nothing to recreate it. The autoscaler won't delete it.
  • Pods using local storage. emptyDir or hostPath data dies with the node. Many autoscalers treat these pods as unevictable unless you explicitly mark them safe to evict.
  • Restrictive PodDisruptionBudgets. A PDB with maxUnavailable: 0, or a minAvailable equal to the replica count, means no eviction is ever allowed.
  • System pods pinned to nodes. Non-DaemonSet pods in kube-system, such as DNS or metrics components, often block scale-down unless they have a PDB.
  • DaemonSets. The autoscaler normally ignores them when it evicts pods. Check them anyway, because in some configurations their resource requests count toward node utilization and can keep a node above the scale-down threshold.

The node pool's configured minimum size is also a floor that no amount of scaling to zero will get below.

Order matters. Shut down consumers first so they finish and acknowledge in-flight messages. Stop queues next, then databases. Start up in reverse order: databases, then queues, then consumers. If you start consumers before their dependencies, they crash-loop until the dependencies are ready.

After the first night, check the node count, not just the pod count. Zero pods on a full set of nodes saves nothing.

#!/usr/bin/env bash
# Ordered off-hours shutdown/startup for one dev namespace, plus checks for scale-down blockers.
set -euo pipefail
NS=dev-team-a

down() {
  # Consumers first so they finish and ack in-flight messages
  kubectl -n "$NS" scale deployment/order-consumer --replicas=0
  kubectl -n "$NS" wait --for=delete pod -l app=order-consumer --timeout=180s
  # Then the queue, then the database
  kubectl -n "$NS" scale statefulset/rabbitmq --replicas=0
  kubectl -n "$NS" wait --for=delete pod -l app=rabbitmq --timeout=180s
  kubectl -n "$NS" scale statefulset/postgres --replicas=0
  kubectl -n "$NS" wait --for=delete pod -l app=postgres --timeout=180s
}

up() {
  # Reverse order: each dependency must be ready before its clients start
  kubectl -n "$NS" scale statefulset/postgres --replicas=1
  kubectl -n "$NS" rollout status statefulset/postgres --timeout=300s
  kubectl -n "$NS" scale statefulset/rabbitmq --replicas=1
  kubectl -n "$NS" rollout status statefulset/rabbitmq --timeout=300s
  kubectl -n "$NS" scale deployment/order-consumer --replicas=2
  kubectl -n "$NS" rollout status deployment/order-consumer --timeout=300s
}

check() {
  # Bare pods with no controller block node removal
  kubectl get pods -A -o json | jq -r '.items[] | select(.metadata.ownerReferences == null) | "\(.metadata.namespace)/\(.metadata.name)"'
  # PDBs allowing zero disruptions block eviction
  kubectl get pdb -A
  # The number that actually matters: nodes, not pods
  echo "nodes: $(kubectl get nodes --no-headers | wc -l)"
}

"${1:?usage: $0 down|up|check}"

A script that shuts down consumers, queue, and database in order, starts them in reverse order, and lists bare pods, PodDisruptionBudgets, and the node count so you can confirm that nodes, not just pods, were removed.

How do you build a do-it-yourself off-hours schedule with CronJobs?

The simplest setup uses two CronJobs in each dev namespace, or one cluster-scoped pair if you're comfortable with broader permissions. Each runs a small image containing kubectl. Each uses a dedicated ServiceAccount bound to a Role that can only get, list, and patch Deployments and StatefulSets, and patch their scale subresource. Keep the Role this narrow so that a compromised or buggy job can't delete workloads or read Secrets.

The evening job selects workloads by label, for example off-hours=enabled, so teams opt in explicitly. For each match, it writes the current replica count into an annotation such as off-hours/original-replicas, then runs kubectl scale --replicas=0. The morning job reads that annotation, scales back to the stored value, and removes the annotation.

Scaling down rarely goes wrong. Restoring is the fragile half:

  • Lost or overwritten counts. If the evening job runs twice, the second run can record 0 as the original value. Only write the annotation when replicas are above zero.
  • Overnight edits. Someone redeploys at 2 a.m. and sets replicas to 3, but the morning job restores the stale value of 1.
  • GitOps reconciliation. A GitOps controller sees replicas drift from what's in Git and reverts the scale-down within minutes. To prevent this, configure the controller to ignore the replica field, remove replicas from the manifests, or have the scheduler pause sync.

This approach is transparent and easy to audit. However, each new requirement adds more shell logic you have to maintain, such as holidays, per-team time zones, or exclusions.

#!/usr/bin/env bash
# Run by both CronJobs: evening calls "off-hours.sh down", morning calls "off-hours.sh up".
set -euo pipefail
MODE="${1:?usage: off-hours.sh down|up}"
NS="${NAMESPACE:-dev-team-a}"
SELECTOR="off-hours=enabled"   # teams opt in by labeling workloads
KEY="off-hours/original-replicas"

for kind in deployment statefulset; do
  for name in $(kubectl get "$kind" -n "$NS" -l "$SELECTOR" -o name); do
    if [ "$MODE" = "down" ]; then
      replicas=$(kubectl get "$name" -n "$NS" -o jsonpath='{.spec.replicas}')
      # Only record non-zero counts so a repeated run can't overwrite the original with 0
      if [ "${replicas:-0}" -gt 0 ]; then
        kubectl annotate "$name" -n "$NS" --overwrite "$KEY=$replicas"
        kubectl scale "$name" -n "$NS" --replicas=0
      fi
    else
      saved=$(kubectl get "$name" -n "$NS" \
        -o go-template="{{index .metadata.annotations \"$KEY\"}}")
      current=$(kubectl get "$name" -n "$NS" -o jsonpath='{.spec.replicas}')
      # Skip workloads with no saved count (missing annotation renders as "<no value>")
      if ! [[ "$saved" =~ ^[0-9]+$ ]]; then
        echo "skip $name: no saved replica count"; continue
      fi
      # Don't clobber someone's overnight change: only restore if still at zero
      if [ "${current:-0}" -eq 0 ]; then
        kubectl scale "$name" -n "$NS" --replicas="$saved"
      else
        echo "skip $name: replicas changed overnight to $current"
      fi
      kubectl annotate "$name" -n "$NS" "$KEY-"   # trailing '-' removes the annotation
    fi
  done
done

A single script used by the evening and morning CronJobs. It saves each opted-in workload's replica count in an annotation before scaling it to zero, then restores that count in the morning, skipping workloads that were changed overnight or that have no saved count.

How does kube-downscaler handle off-hours scaling?

kube-downscaler packages that logic into a controller so you stop maintaining it as scripts. It runs as a deployment inside the cluster and periodically checks every eligible Deployment, StatefulSet, and similar workload against a time window. You give it a cluster-wide default, either an uptime window (when workloads should run) or a downtime window (when they should not), expressed as weekdays, hours, and a time zone. Outside uptime, it scales the workload to zero. Because it records the original replica count on the object, usually as an annotation, it can restore the exact previous scale when the window reopens. You do not need to keep a separate copy of the "real" replica counts.

Overrides use annotations. A team in another region can annotate its namespace with its own uptime window. A single shared service can be annotated to opt out entirely. A workload can also declare a nonzero replica count to keep during downtime instead of zero. As an illustrative example, a namespace annotated with an uptime of weekdays 07:00–20:00 in its team's time zone would scale down overnight and on weekends, while one deployment inside it with an exclude annotation would keep running. Exact annotation names, flags, and supported resource types differ between forks and versions, so check the documentation for the build you deploy.

The trade-off is operational. This is one more controller to secure, upgrade, and monitor. It also needs RBAC permissions to patch workloads in every namespace it manages, which makes its service account a high-value target. Scope those permissions as narrowly as your namespace layout allows.

# ILLUSTRATIVE ONLY: annotation keys follow the original kube-downscaler
# project's convention. Forks and versions differ, so verify names and
# time-window syntax against the docs for the build you deploy.
apiVersion: v1
kind: Namespace
metadata:
  name: team-emea-dev
  annotations:
    # Override the cluster-wide default for this namespace:
    # run on weekdays during the team's working hours, scale to zero otherwise.
    downscaler/uptime: "Mon-Fri 07:00-20:00 Europe/Berlin"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shared-auth-mock
  namespace: team-emea-dev
  annotations:
    # Opt this one workload out: other teams depend on it overnight.
    downscaler/exclude: "true"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: shared-auth-mock
  template:
    metadata:
      labels:
        app: shared-auth-mock
    spec:
      containers:
        - name: app
          image: registry.example.com/shared-auth-mock:1.0.0
# Every other workload in team-emea-dev scales down outside the uptime window.
# The controller records the original replica count on each object so it can
# restore that count when the window reopens.

An illustrative namespace-level uptime override and a per-workload opt-out using kube-downscaler-style annotations. Check the exact keys against your version's documentation.

What breaks: HPAs, stateful workloads, and cluster-critical components

An HPA and an off-hours scaler both write the same spec.replicas field, so they can conflict. If the scaler sets a count below the HPA's minReplicas, the HPA raises it back on its next sync. If the scaler sets replicas to zero, a standard HPA stops acting on that workload and will not scale it up by itself in the morning. Pick one of three options per workload:

Option How it works Trade-off
Adjust the HPA Lower minReplicas during downtime and restore it in the morning Two objects change, so both must be restored correctly
Suspend the HPA Remove or detach it overnight and recreate it at scale-up Drift if the restore step fails
Exclude the workload Leave HPA-managed workloads out of the schedule Saves less money but nothing can conflict

Scaling a StatefulSet to zero deletes its pods but keeps its PersistentVolumeClaims, so the storage keeps billing overnight. For databases and brokers, scale down in a controlled order so replicas leave cleanly and partitions are not left mid-rebalance. In the morning, start the stateful tier first and wait for quorum before scaling up the services that depend on it. If dependents start first, they crash-loop until the database or broker is ready.

Never schedule these components down:

  • Ingress controllers
  • DNS
  • The downscaler itself
  • In-cluster CI runners
  • Monitoring agents
  • The node autoscaler

Define the schedule's scope by namespace or label, not by listing individual workloads. Make the default opt-in for each namespace. Any opt-out label should record an owner and an expiry date, and a periodic check should report exemptions that have expired.

When should you avoid shutting down dev environments on a schedule?

Even with that discipline, the schedule is not always worth running. Check the bill first. If the control plane fee, load balancers, persistent volumes, NAT gateways and managed databases make up most of the cost, scaling pods to zero saves little. The same applies when pods that block eviction keep most nodes alive: local storage, strict PodDisruptionBudgets, or DaemonSet-heavy node pools.

Night-time work is the next cost. Long test suites, batch jobs, overnight data loads and soak tests get killed mid-run. For distributed teams, someone's working day falls inside the off-hours window.

The morning has costs too. Node provisioning, image pulls, cache warm-up and waiting for dependencies to become ready can take a long time. Every engineer who waits on that pays with their time, which the compute savings don't show.

Scale-up can also fail. Cloud capacity is not guaranteed. Quotas, unavailable instance types or spot interruptions can leave an environment partially up. That is harder to debug than one that is fully down.

A cluster that restarts every night also stops resembling production. Memory leaks, certificate rotation, scheduled jobs that fire at 03:00 and connection pools that degrade over days never get a chance to appear.

Finally, watch the exclusions. Each opt-out is reasonable on its own, but they pile up. A namespace that is excluded "for this release" stays excluded, and the savings shrink without anyone noticing. Give every exclusion an expiry date and enforce it, or the schedule slowly turns into a cluster that runs all night anyway.

How do you run the schedule across time zones without slowing engineers down?

A schedule written as a single UTC window, say 19:00–07:00 UTC, matches one office's evening. For a team eight hours east, that same window starts in the middle of their afternoon. Define each schedule in a named IANA time zone such as Europe/Berlin or America/Chicago, not as a UTC offset. With a named zone, daylight saving transitions are handled by the time zone database. With a fixed offset, the window drifts by an hour twice a year. When one environment serves several regions, set its uptime window to span the earliest start and the latest finish across those regions. Keep holidays in a shared calendar that the scheduler reads, so nobody has to remember to edit the config before a long weekend.

Self-service overrides

Engineers will need to wake an environment early or keep it running late. Give them one documented path for this. Two common options are a temporary annotation with an expiry timestamp, or a manual job that anyone on the team can trigger. Either one avoids a ticket and the wait for someone with cluster-admin rights. Record who requested each override and when it expires, and remove expired overrides automatically.

Make it observable

  • Alert when the morning scale-up fails or when the environment fails its readiness checks by a target time, such as 30 minutes before the workday starts.
  • Record the node count every night. This is how you confirm that nodes are actually being removed, not just pods.
  • Log every scale action with the workload name and its replica counts before and after. When a deployment comes back with zero replicas, this log is the first place to look.
# Nightly scale-down for the dev namespace, scheduled in a named IANA zone
# so daylight saving shifts are handled by the tz database, not a fixed offset.
apiVersion: batch/v1
kind: CronJob
metadata:
  name: offhours-scale-down-berlin
  namespace: dev
spec:
  schedule: "0 19 * * 1-5"        # 19:00 local time, Monday-Friday
  timeZone: "Europe/Berlin"       # named zone, never "UTC+1"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: offhours-scaler   # RBAC: get/patch deployments, deployments/scale
          restartPolicy: Never
          containers:
            - name: scale-down
              image: registry.example.com/tools/kubectl:1.30   # placeholder image with kubectl + sh
              command: ["/bin/sh", "-c"]
              args:
                - |
                  set -eu
                  now=$(date +%s)
                  for d in $(kubectl get deploy -n dev -o name); do
                    # Self-service override: epoch-seconds expiry set by an engineer.
                    keep=$(kubectl get "$d" -n dev -o jsonpath='{.metadata.annotations.offhours/keep-up-until}')
                    if [ -n "$keep" ] && [ "$keep" -gt "$now" ]; then
                      echo "action=skip workload=$d reason=override until=$keep"; continue
                    fi
                    # Expired overrides are removed so they cannot linger.
                    [ -n "$keep" ] && kubectl annotate "$d" -n dev offhours/keep-up-until-
                    before=$(kubectl get "$d" -n dev -o jsonpath='{.spec.replicas}')
                    # Store the count the morning job will restore.
                    kubectl annotate "$d" -n dev --overwrite offhours/restore-replicas="$before"
                    kubectl scale "$d" -n dev --replicas=0
                    echo "$(date -u +%FT%TZ) action=scale-down workload=$d before=$before after=0"
                  done

A CronJob scheduled in the Europe/Berlin time zone that scales dev Deployments to zero, honors and cleans up a time-limited keep-up-until override annotation, saves the original replica count for the morning restore, and logs before and after replica counts for each scale action.

What are the alternatives to scheduled scale-to-zero?

Scheduled scale-to-zero is not the only way to stop paying for idle capacity. Four other approaches cover most setups, and they differ mainly in what gets removed and how hard it is to bring back.

Approach Suits Operational complexity Restore risk Share of bill removed
Shut down whole node pools Small teams with one shared dev cluster Low Medium: pending pods and slow node boot in the morning Most worker compute; control plane and storage remain
Scale node groups in the cloud autoscaling config Teams already managing infrastructure as code Low to medium Medium: config drift if someone edits min/max by hand Same as above, enforced below Kubernetes
Ephemeral per-branch environments Larger teams with mature CI and seed data High Low: environments are rebuilt, not resumed Nearly all idle cost, including storage
Event-driven scale-to-zero on request Bursty, low-traffic services Medium Low to medium: cold-start latency on first request Pod cost during idle periods, nodes only if the cluster autoscaler follows

Shutting down node pools skips workload-level bookkeeping, but it evicts everything at once, including the add-ons you excluded earlier. Ephemeral environments avoid restore problems because nothing is paused. You pay for that with build time and the work of producing realistic test data. Request-driven scaling removes the idle cost of individual services, but it saves nothing on nodes unless pod removal actually lets nodes drain.

Which to choose:

  • Small teams sharing one cluster: use node pool shutdown or workload schedules.
  • Teams with strong CI: move toward per-branch environments.
  • Mostly-idle HTTP services: add request-driven scaling alongside either of the above.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to shut down dev environments schedule, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.