How to Shut Down Staging Environments at Night and Keep Them Off

What scheduled shutdowns are and who they suit

A scheduled shutdown stops or scales down non-production resources, such as dev clusters, staging databases, preview environments and build runners, during hours when nobody uses them, and brings them back before people need them again. The schedule might be "off from 20:00 to 07:00 on weekdays and all weekend," enforced by an automation that scales workloads to zero, stops instances or pauses managed services.

The reason teams do this is simple: most compute is billed by the hour whether or not anyone sends it a request. Staging runs at production-like size so tests mean something, but it sits idle most of the week. As an illustrative estimate, assume an environment is needed 12 hours a day, 5 days a week. That is 60 of 168 weekly hours, so roughly 64% of its running time is idle. Storage, reserved capacity and some managed services keep billing regardless, so real savings will be lower than that figure.

Schedules pay off most for teams with predictable working hours, a small number of time zones, and long overnight or weekend gaps. Teams spread across many time zones, or teams running long soak tests, have shorter idle windows and save less.

Writing the scheduler is rarely the hard part. Schedules usually fail later, for three reasons:

• Overrides that never expire.
• Wake-ups that leave half the stack broken on Monday morning.
• Exceptions that nobody owns.

The rest of this checklist deals with those three problems.

Inventory what stops billing and what does not

Before writing any schedule, sort every resource in the environment into two lists: things that stop billing when stopped, and things that keep billing while idle. Skipping this step is how teams end up with a schedule that runs every night and a bill that barely moves.

Watch for these traps when classifying:

• Stopped databases still pay for storage. Some providers also restart a stopped instance automatically after a fixed period, so a stop is not permanent.
• Scaling a node pool to zero removes the nodes. Persistent volumes and the cluster control plane keep billing.
• Load balancers and NAT gateways bill hourly whether or not traffic flows. Removing them every night is a teardown, not a shutdown, and belongs in a separate decision.

Illustrative estimate (assumed inputs): The team uses staging 12 hours a day on weekdays, which is 60 of 168 weekly hours. That leaves 108 hours idle, about 64%. Assume a monthly non-production bill of $10,000, with $5,000 in the stoppable category. The schedule saves roughly 64% of $5,000, or about $3,200, which is around 32% of the total bill, not 64%.

Share the realistic figure before you start. When a team expects 64% and sees 32%, it often decides the effort failed, even though the savings are real.

Declare the schedule as data on the resource

Schedules that live in a wiki or in a cron script's hard-coded list drift away from reality within weeks. Put the policy on the resource instead, as tags on cloud resources or labels and annotations in Kubernetes. One controller or scheduled job then reads that metadata and acts on it. The environment definition and its shutdown policy sit in the same file in version control, so they are reviewed together, and a new environment gets a schedule when it is created.

A namespace or Deployment might carry a small set of annotations under a prefix your team owns:

• schedule.example.com/owner: payments-team
• schedule.example.com/window: core
• schedule.example.com/timezone: Europe/Berlin
• schedule.example.com/expires: 2025-03-31

Store the time zone as an IANA name such as Europe/Berlin, not a UTC offset. A fixed offset is wrong for half the year wherever daylight saving applies.

Define one shared core window, for example 07:00–20:00 on weekdays, and let most resources reference it by name. When the window changes, you edit one definition and every environment follows. Allow custom windows, but make them visible exceptions.

Treat opt-outs the same way. An annotation like schedule.example.com/opt-out: "true" should be ignored unless it has an owner, a reason, and an expiry date. Without the reason, nobody can later tell whether the opt-out still matters. Without the expiry, a two-day exception for a load test becomes a permanent always-on environment.

Copy# Shared window definitions, read by the scheduler controller.
# Edit 'core' here and every environment that references it follows.
apiVersion: v1
kind: ConfigMap
metadata:
  name: schedule-windows
  namespace: env-scheduler
data:
  core: "Mon-Fri 07:00-20:00"
---
# A typical staging namespace: policy lives beside the environment definition.
apiVersion: v1
kind: Namespace
metadata:
  name: payments-staging
  annotations:
    schedule.example.com/owner: "payments-team"
    schedule.example.com/window: "core"              # name from the ConfigMap, not a copied time range
    schedule.example.com/timezone: "Europe/Berlin"   # IANA name, so daylight saving is handled
    schedule.example.com/expires: "2026-12-31"       # environment is flagged for deletion after this
---
# A temporary opt-out. The controller ignores opt-out unless owner,
# reason and expiry are all present, and resumes the schedule after expiry.
apiVersion: v1
kind: Namespace
metadata:
  name: search-loadtest
  annotations:
    schedule.example.com/owner: "search-team"
    schedule.example.com/window: "core"
    schedule.example.com/timezone: "America/New_York"
    schedule.example.com/opt-out: "true"
    schedule.example.com/opt-out-owner: "search-team"
    schedule.example.com/opt-out-reason: "48h soak test of new indexer"
    schedule.example.com/opt-out-expires: "2026-11-05"

Kubernetes manifests where a shared ConfigMap defines the named core window, and namespaces carry owner, window, IANA time zone and expiry annotations, plus an opt-out that names an owner, a reason and an expiry date.

How to implement Kubernetes scale to zero safely

The CronJob and its permissions

Run two CronJobs per namespace, one for shutdown and one for wake. Each runs a container with kubectl that scales the opted-in Deployments and StatefulSets to zero, or back up. Before scaling down, have the job record each workload's current replica count in an annotation. The wake job then restores the real value instead of a hardcoded guess.

Give the job its own ServiceAccount, bound to a namespaced Role. The Role should grant get, list and patch on deployments, statefulsets and their scale subresources, and nothing else. A leaked token can then resize workloads in one namespace, but it cannot read Secrets, delete resources, or touch other teams.

Let the nodes go too

Scaling pods to zero saves nothing if the nodes stay up. The node pool's minimum size must be zero, so that the cluster autoscaler can remove nodes once they are empty. Check which pods are still running on a node that refuses to drain. The autoscaler skips nodes whose pods it cannot safely evict, such as pods using local storage or pods protected by a restrictive PodDisruptionBudget.

If an event-driven autoscaler already manages these workloads, use it for scale to zero. Running a second controller that fights it over replica counts causes flapping.

Order matters

Shut down in this order:

1. Ingress-facing services
2. Workers
3. Databases and queues

Shutting down the app tiers first lets in-flight requests drain and consumers stop cleanly. On wake, reverse the order. Start the stateful tier and wait for readiness probes to pass. Then start the workers, then the frontends.

Finish with a smoke test, such as one authenticated request through the real ingress. If it fails, alert the owner before the team arrives.

CopyapiVersion: v1
kind: ServiceAccount
metadata: {name: scaler, namespace: staging}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: scaler, namespace: staging}
rules:  # resize workloads only: no Secrets, no delete, no other namespaces
- apiGroups: ["apps"]
  resources: [deployments, statefulsets, deployments/scale, statefulsets/scale]
  verbs: [get, list, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: scaler, namespace: staging}
subjects: [{kind: ServiceAccount, name: scaler, namespace: staging}]
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: scaler}
---
apiVersion: batch/v1
kind: CronJob
metadata: {name: staging-shutdown, namespace: staging}
spec:
  schedule: "0 20 * * 1-5"
  timeZone: "Europe/Berlin"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: scaler
          restartPolicy: Never
          containers:
          - name: kubectl
            image: registry.example.com/tools/kubectl:1.30  # placeholder image
            command: ["/bin/sh", "-c"]
            args:  # app tiers first, data tier last; skip already-zero workloads so a rerun keeps the saved count
            - |
              set -e
              for tier in frontend worker data; do for name in $(kubectl get deployment,statefulset -l scale-to-zero=true,tier=$tier -o name); do
                r=$(kubectl get "$name" -o jsonpath='{.spec.replicas}')
                if [ "$r" -gt 0 ]; then kubectl annotate "$name" staging.example.com/replicas="$r" --overwrite; kubectl scale "$name" --replicas=0; fi
              done; done

A namespaced ServiceAccount and Role limited to get, list and patch on Deployments and StatefulSets and their scale subresources, plus a shutdown CronJob that saves each opted-in workload's replica count in an annotation and then scales the frontend, worker and data tiers to zero in that order.

Wake-ups, overrides and off-hours jobs

A schedule only sticks if breaking it is harder than following it. If waking an environment means filing a ticket and waiting for a platform engineer, people will ask for the schedule to be removed. Give them a faster, controlled path instead.

Self-service wake-ups

• Offer one entry point per team habit. This can be a chat command, a manually triggered CI job, or an annotation on the namespace. All of them should call the same wake-up routine described above, so the startup order and smoke test never vary.
• Require an expiry on every wake-up. Store an "awake until" timestamp on the resource. The scheduler honors it and then resumes normal behavior. Don't accept open-ended overrides. They're how a shutdown quietly stops happening.
• Record who woke it and why. A short reason field costs one line, and it tells you which teams regularly need evening hours.

Off-hours jobs

• Declare fixed windows for predictable work. Nightly integration tests and data refreshes can get an explicit window in the schedule data, such as a two-hour block before dawn.
• Let irregular jobs wake and release. A long migration or a backfill should request a wake-up at the start and release the environment when it exits, including when it fails. Pair that with a maximum duration so a hung job can't hold the environment open indefinitely.
• Alert on missed windows. A test suite that ran against a sleeping environment should fail loudly. A silent pass is the outcome to prevent.

The payoff is fewer interruptions. Engineers stop asking in chat for someone to restart staging, and nobody has to remember to turn it back off.

Copy#!/usr/bin/env bash
# Wrap a long-running off-hours job: wake staging, hold it with an expiring
# annotation, and always release it when the job exits (success or failure).
set -euo pipefail

NS="staging"
MAX_RUNTIME="4h"   # hard cap so a hung job cannot hold the environment open
KEY="sleep-schedule.example.com"

# Expiry is mandatory: even if this script dies before cleanup, the scheduler
# resumes normal behavior after this timestamp. (GNU date syntax.)
AWAKE_UNTIL=$(date -u -d "+4 hours 15 minutes" +%Y-%m-%dT%H:%M:%SZ)

release() {
  # Remove the override (trailing '-' deletes an annotation) so the next
  # scheduler pass can put the namespace back to sleep.
  kubectl annotate namespace "$NS" "$KEY/awake-until-" "$KEY/woken-by-" "$KEY/reason-" || true
}
trap release EXIT

# Record the override, who requested it, and why.
kubectl annotate namespace "$NS" --overwrite \
  "$KEY/awake-until=$AWAKE_UNTIL" \
  "$KEY/woken-by=ci-migration-job" \
  "$KEY/reason=orders-table-backfill"

# Reuse the same wake-up routine as the chat command and CI button, so startup
# order and the smoke test never differ. Fail here rather than run against a
# sleeping environment.
./scripts/wake-env.sh "$NS"

# Run the job under a hard time limit; exit code 124 means it was killed.
set +e
timeout --kill-after=5m "$MAX_RUNTIME" ./scripts/run-backfill.sh
status=$?
set -e

[ "$status" -eq 124 ] && echo "backfill exceeded $MAX_RUNTIME, releasing $NS" >&2
exit "$status"   # the EXIT trap releases the environment either way

A wrapper for an irregular off-hours job. It wakes staging through the shared routine and sets an expiring override with who requested it and why. It caps the job's runtime, and the EXIT trap releases the environment even if the job fails.

Make drift visible and assign ownership

A schedule that works on day one degrades quietly. Someone sets an override for a demo and forgets it. A new namespace gets created without tags. A load test leaves a database scaled up. None of these trigger an alert, because nothing is broken. The environment simply costs money again.

Publish a drift report on a fixed cadence

Generate a report weekly, from the same tags the scheduler reads, so it cannot disagree with enforcement. It should list three things:

• Running outside schedule. Environments with active compute during their declared off-hours, with how many hours they ran and the approximate cost. A cost figure gets more attention than a resource name.
• Active overrides. Each override with who set it, when, the stated reason and its expiry. Overrides with no expiry, or ones renewed several times, deserve a closer look because they are usually permanent exceptions that nobody has admitted to.
• Untagged resources. Anything missing an owner or schedule tag. These are invisible to the scheduler, so they are the most likely source of unexplained spend.

Post the report where teams already look, not in a dashboard nobody opens.

Give the review a named owner

Tooling finds drift, but it does not remove it. Each exception has a reasonable story behind it, and declining someone's request feels costly. Without a person whose job is to ask "does this still need to run?", exceptions pile up until the schedule only exists in config. Assign one engineer per review cycle, rotate the role, and let them close stale overrides by default unless the owner renews them.

When to avoid it, trade-offs and alternatives

Scheduling is not always the cheapest option once restart time is counted. A cold start can mean provisioning nodes, pulling images, warming caches and letting a database replay its logs before it accepts connections. For a small environment, an engineer waiting through that every morning can cost more than the overnight compute saved.

Stateful components carry the most risk. Some managed databases restart on their own after a maximum stopped period, which quietly breaks the schedule. Distributed locks can be left stale, leases expire, and replicas can fall too far behind to catch up without a full resync. Daily restarts also hide bugs that only appear under continuous uptime, such as memory leaks, connection-pool exhaustion and certificate rotation failures, until they reach production.

Networking breaks in less obvious ways. Recreated load balancers or nodes may get new IPs. Clients cache stale DNS records, third-party webhooks fail, and partner allowlists stop matching. The shutdown job also needs permissions to stop or delete resources, so scope it tightly or it becomes a privileged path into production. If your team spans enough time zones that the environment is rarely idle, the savings mostly disappear.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to shut down staging environments at night, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.