What does it mean to shut down dev environments at night, and how much does it save?

Non-production scheduling means scaling dev, staging and preview compute to zero on a predictable timetable, usually evenings and weekends, and bringing it back before people start work. It fits platform and backend teams running always-on environments that sit idle most of the week while billing at full rate.

Illustrative estimate (assumed numbers, not a benchmark)

Assumed inputs: 50 dev nodes, $0.20 per node-hour, currently running 24x7 (168 hours/week), and a target of 12 hours on weekdays (60 hours/week).

  • Hours removed: (168 − 60) / 168 = 108 / 168 ≈ 64%
  • Weekly cost before: 50 × 168 × $0.20 = $1,680
  • Weekly cost after: 50 × 60 × $0.20 = $600
  • Weekly savings: $1,080

General formula: savings = nodes × rate × (hours_now − hours_scheduled)

This only holds if nodes actually go away. Scaling pods to zero saves nothing until the cluster autoscaler or node group scaling removes the empty nodes. Anything that blocks node drain, such as DaemonSet-only nodes or a minimum node count, also blocks the savings.

Resource Stops billing when pods scale to zero? Why
Worker nodes Only if removed The provider bills a node that exists, even when it is empty
Load balancers No Billed while provisioned, regardless of traffic
NAT gateways No Billed per hour plus per GB of data
Persistent disks No Storage persists so state survives the restart

Tooling is the easy part. The hard part is policy: which workloads may stop, who can override the schedule, and how databases, queues and caches come back in a consistent state.

Which workloads are safe to stop, and which should you leave running?

Stateless Deployments are the easiest case. Each pod can be replaced, so setting replicas to zero and back loses nothing except warm caches and in-flight requests. State is the problem: it determines what can stop safely, and stopping compute does not stop the storage bill.

Workload Stop off-hours? Main risk What keeps billing
Stateless Deployments Yes Cold caches, slower first request Nothing significant
StatefulSets / databases With care Unclean shutdown, slow recovery, replica resync Persistent volumes, snapshots
Message brokers With care Queued messages expire under retention limits Persistent volumes
Ingress, cert management, operators, admission webhooks No Breaks other environments or blocks pod creation Not applicable (keep running)
Overnight CI, batch and soak tests Exclude or schedule around Killed mid-run, results lost Not applicable

Shared components need extra care. If an admission webhook's backing service is at zero and the webhook fails closed, the API server rejects new pods. The environment then cannot scale itself back up. Ingress controllers and operators often serve several namespaces, so stopping them for one team breaks others.

Long-running work is the other common failure. A soak test scheduled for 18:00 to 06:00 gets killed when the downscaler runs at 19:00, and nobody notices until morning. Give these workloads an exclusion annotation or run them in a namespace with its own schedule.

The trade-off is direct. Stopping databases and brokers saves the most compute, but it risks data corruption and long recovery on wake. Excluding them keeps environments reliable but reduces the savings. A reasonable default is to stop stateless workloads everywhere and stop stateful ones only where a fresh seed or restore is acceptable.

How does kube-downscaler work in a Kubernetes cluster?

kube-downscaler runs as a single Deployment that wakes on a fixed interval, lists the Deployments, StatefulSets and other scalable workloads it can see, and compares the current time with each workload's uptime window. Outside the window it records the current replica count in the downscaler/original-replicas annotation, then sets replicas to zero, or to the value in downscaler/downtime-replicas if you want one pod kept warm. When the window reopens, it reads the annotation back and restores that count. Because the previous count lives on the object, restarting the downscaler loses nothing.

Configuration is layered, and the most specific setting wins:

  • Cluster: flags such as --default-uptime set the baseline schedule, for example Mon-Fri 07:30-20:00 Europe/Berlin. --exclude-namespaces protects system and shared namespaces. By default it already skips kube-system.
  • Namespace: a downscaler/uptime annotation on the namespace gives one team its own hours without touching anyone else.
  • Workload: downscaler/exclude: "true" permanently opts out a database or message broker. downscaler/exclude-until grants a temporary reprieve for a late debugging session.

Use exclusions to protect anything other environments depend on, such as a shared identity service, an ingress controller or a cluster-wide operator.

The downscaler does not reduce cost by itself. The flow is:

downscaler → replicas: 0 → pods terminate → nodes sit empty → cluster autoscaler drains and removes them

Without a node autoscaler, you have idle pods replaced by idle nodes, and the bill barely moves.

# Namespace level: this team's own schedule overrides --default-uptime
apiVersion: v1
kind: Namespace
metadata:
  name: team-payments-dev
  annotations:
    downscaler/uptime: "Mon-Fri 08:00-19:00 Europe/Berlin"
---
# Workload level: keep one pod warm overnight instead of scaling to zero.
# The downscaler writes downscaler/original-replicas itself; don't set it by hand.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: team-payments-dev
  annotations:
    downscaler/downtime-replicas: "1"
spec:
  replicas: 3
  selector:
    matchLabels: {app: payments-api}
  template:
    metadata:
      labels: {app: payments-api}
    spec:
      containers:
        - name: api
          image: registry.example.com/payments-api:dev
---
# Workload level: the database is never scaled down.
# For a one-off late session, use downscaler/exclude-until: "2026-11-08" instead.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: payments-db
  namespace: team-payments-dev
  annotations:
    downscaler/exclude: "true"
# spec omitted for brevity

Namespace and workload annotations: a team-specific uptime window on the namespace, a Deployment that keeps one replica warm during downtime, and a StatefulSet database that is permanently excluded from downscaling.

How does a VM instance scheduler pattern work, and what are the alternatives?

For VMs and managed databases, the same idea works without an in-cluster controller. Each resource carries its schedule as tags, and a periodic job reconciles actual state against them. A minimal illustrative schema:

  • schedule = office-hours names a schedule defined centrally
  • schedule-tz = Europe/Berlin sets the timezone explicitly
  • schedule-override-until = <ISO timestamp> is optional and keeps the resource running past its window

A cron-driven reconcile loop, running every 10–15 minutes:

  1. List all resources that carry a schedule tag.
  2. Skip any resource whose override has not expired.
  3. Convert the current time to that resource's timezone and decide whether it should be running.
  4. Start or stop only if desired and actual state differ, then log the action.

Because it reconciles instead of firing once at 19:00, a missed run corrects itself on the next tick. The trade-off is that a manual start gets stopped again unless someone sets an override.

The tags matter more than which scheduler you choose. An untagged resource runs 24/7 and nothing reports it. A schedule with no timezone drifts at daylight-saving changes and is wrong for teams in other regions. Require the tags when resources are created, and alert on any resource that lacks them.

Approach Best for Main weakness
kube-downscaler Kubernetes workloads Saves money only if nodes scale down too
Tag-based VM scheduler VMs, managed databases Depends on tag hygiene
Cron + cloud CLI A handful of resources No reconciliation, breaks quietly
Scaling node groups directly Whole dev clusters Abrupt; ignores workload intent
Ephemeral environments Per-branch or per-PR testing Needs fully automated provisioning
# Reconcile loop, run from cron every 15 minutes: */15 * * * * python3 reconcile.py
from datetime import datetime, time, timezone
from zoneinfo import ZoneInfo

# Schedules are defined once, centrally; resources reference them by name via tags.
SCHEDULES = {
    "office-hours": {"days": {0, 1, 2, 3, 4}, "start": time(8, 0), "stop": time(19, 0)},
}

def should_run(tags, now_utc):
    sched = SCHEDULES[tags["schedule"]]
    # A missing timezone raises instead of silently defaulting to UTC.
    local = now_utc.astimezone(ZoneInfo(tags["schedule-tz"]))
    return local.weekday() in sched["days"] and sched["start"] <= local.time() < sched["stop"]

def reconcile(cloud, log):
    now = datetime.now(timezone.utc)
    # list_resources, start and stop are placeholders for your provider's SDK calls.
    for r in cloud.list_resources(tag_key="schedule"):
        tags = r.tags
        if "schedule-tz" not in tags:
            log.warning("%s has no schedule-tz; skipping and flagging", r.id)
            continue
        override = tags.get("schedule-override-until")
        # Override timestamps are expected to include a UTC offset, e.g. 2026-11-10T22:00:00+00:00.
        if override and datetime.fromisoformat(override) > now:
            continue  # someone asked to keep this running; leave it alone
        desired = should_run(tags, now)
        actual = r.state == "running"
        # Act only on drift, so a missed run is corrected on the next tick.
        if desired and not actual:
            cloud.start(r.id)
            log.info("started %s", r.id)
        elif actual and not desired:
            cloud.stop(r.id)
            log.info("stopped %s", r.id)

A cron-driven reconcile loop that reads schedule, schedule-tz and override tags, evaluates each schedule in the resource's own timezone, and starts or stops a resource only when its actual state differs from the desired state, with cloud calls left as placeholders for your provider's SDK.

How do you define schedules and overrides that match how teams actually work?

A single global schedule is easy to audit, but it fails as soon as working hours stop overlapping. If the schedule is set to "08:00–19:00 UTC," engineers in Singapore lose their mornings and engineers in California lose their afternoons. They respond by asking for exemptions, and the exemptions turn into permanent always-on environments.

Per-team schedules fix the fit but add configuration surface. A workable middle ground has three parts:

  • Schedules attach to the owning team, not to each environment. Store each team's timezone and working window in one place, such as namespace labels or a central config, and have environments inherit them. When a team moves timezones, you change one value.
  • Weekends and holidays come from shared calendars. Keep a company calendar plus regional calendars, and resolve the team's own timezone before applying them. Asia-Pacific evening hours are already Saturday in UTC.
  • Distributed teams get the union of their members' hours. Do not try to find an overlap. A wider window still beats running 24/7.

Self-service overrides with expiry

Overrides need to be faster than filing a ticket, or engineers will route around the scheduler. An engineer should be able to keep an environment up tonight or wake it early with one label, annotation or chat command. Every override needs a mandatory expiry timestamp, capped at something like 24 or 72 hours. Reject any override without an expiry. Report active overrides weekly so that renewals that repeat every week become visible.

Overrides with expiry also save engineering time. Engineers no longer file "please don't shut down staging" tickets, and nobody restarts environments by hand at 7 a.m. The platform team stops being the gatekeeper for every late deploy.

from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo

MAX_OVERRIDE = timedelta(hours=72)  # hard cap so overrides cannot become always-on

# Shared calendars: company-wide dates plus regional ones (placeholder dates).
CALENDARS = {"company": {"2026-12-25"}, "SG": {"2026-11-08"}, "US": {"2026-11-26"}}

# Schedules belong to the team; environments inherit via their team label.
TEAMS = {
    "payments": [  # distributed team: union of members' windows, no overlap needed
        {"tz": "Asia/Singapore", "start": 8, "end": 19, "region": "SG"},
        {"tz": "America/Los_Angeles", "start": 8, "end": 19, "region": "US"},
    ],
}

def in_window(now_utc, w):
    local = now_utc.astimezone(ZoneInfo(w["tz"]))  # resolve local time first
    day = local.date().isoformat()
    if local.weekday() >= 5 or day in CALENDARS["company"] | CALENDARS[w["region"]]:
        return False  # weekend or holiday in the team's own timezone
    return w["start"] <= local.hour < w["end"]

def validate_override(expires_iso, now_utc):
    if not expires_iso:
        raise ValueError("override rejected: expiry is required")
    expires = datetime.fromisoformat(expires_iso)
    if expires.tzinfo is None:
        raise ValueError("override rejected: expiry must include a UTC offset")
    if expires - now_utc > MAX_OVERRIDE:
        raise ValueError("override rejected: expiry exceeds 72h cap")
    return expires

def should_run(team, override_expires, now_utc):
    # An unexpired override keeps the env up tonight or wakes it early.
    if override_expires and override_expires > now_utc:
        return True
    return any(in_window(now_utc, w) for w in TEAMS[team])

now = datetime.now(timezone.utc)
print(should_run("payments", validate_override("2026-11-08T06:00:00+00:00", now), now))

A team-level schedule evaluated in each member's timezone, with shared holiday calendars and an override that is rejected unless it carries an expiry within a 72-hour cap.

Why do environments wake up broken, and how do autoscalers and GitOps interfere?

Wake-up order and timing

Most broken mornings come from everything starting at once. Application pods come up before the database, message broker, or cache is accepting connections, fail their first connection attempts, and land in crash-loop backoff. Each restart waits longer than the last, so a service can stay down well after its dependency is healthy. Scale stateful dependencies first, wait for their readiness probes to pass, then scale the services that use them. Make sure readiness probes check real dependencies, not just that the process is alive.

Account for cold start too. If the cluster autoscaler removed nodes overnight, the morning scale-up has to provision nodes and pull images onto empty disks. That can take many minutes for large images. Start the environment early enough to absorb this. A 7:00 wake-up for a 9:00 team costs little and means engineers find a working environment instead of a half-started one.

Controllers that fight the schedule

A HorizontalPodAutoscaler with minReplicas: 2 will scale a deployment back up after your scheduler sets it to zero. Either the downscaler must also manage the HPA's minimum, or the HPA must be suspended during off-hours.

GitOps controllers treat a changed replica count as drift. With auto-sync enabled, they restore the value from Git within minutes. The schedule appears to work, then silently stops. Configure the sync to ignore /spec/replicas, as in the example below, or remove replicas from the manifests so something outside Git owns that field.

Operators that reconcile their own custom resources behave the same way. Scale the custom resource, not the StatefulSet the operator owns. Otherwise the operator and the scheduler take turns overwriting the replica count and the workload flaps.

# Argo CD Application for a dev environment whose replica counts are
# owned by an off-hours scheduler, not by Git.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dev-env
  namespace: argocd
spec:
  project: default
  source:
    repoURL:   # placeholder
    targetRevision: main
    path: overlays/dev
  destination:
    server:
    namespace: dev
  # Do not treat a changed replica count as drift.
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: apps
      kind: StatefulSet
      jsonPointers:
        - /spec/replicas
  syncPolicy:
    automated:
      prune: true
      selfHeal: true   # still heals real drift, just not replicas
    syncOptions:
      # Without this, ignoreDifferences only affects the diff. Any sync
      # triggered by another change would still apply the replica value
      # from Git and undo the overnight scale-down.
      - RespectIgnoreDifferences=true

An Argo CD Application that keeps auto-sync and self-heal enabled but ignores /spec/replicas on Deployments and StatefulSets, so a scheduler can scale the environment to zero overnight without Git restoring the old counts.

When should you avoid off-hours scheduling, and how do you measure whether it works?

Some environments should stay off a schedule entirely:

  • Round-the-clock users. Teams across several time zones, or a QA group that tests overnight, will hit a stopped environment during their working day.
  • Heavy overnight pipelines. Nightly builds, long integration suites and data loads that expect the environment to be up.
  • Shared dependencies that cannot stop. A staging database or message broker that other teams' environments call.
  • Small fleets. With only a handful of instances, writing and maintaining schedules can cost more engineering time than the compute saves.

Treat the scheduler as a dependency in its own right. A failure goes one of two ways: environments stay up and keep costing money, or they stay down and block the morning's work. Alert when a scale-down or scale-up run is missed, not only when one errors. Document a manual fallback that says who can bring an environment back, how they do it, and where the procedure is written down.

Metric What it shows What a bad reading usually means
Non-prod node-hours and spend, before vs. after Whether capacity was actually removed Something keeps nodes alive, such as autoscaler minimums, disruption budgets or excluded workloads
Override requests per week How often the schedule conflicts with real work One team's hours differ from the default
Failed wake-ups How reliably environments come back Startup ordering problems or GitOps drift

Read overrides and failed wake-ups as signs that the schedule is wrong, not as evidence that engineers are misusing it. Fix the schedule before you tighten the rules around it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to shut down dev environments at night, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.