What is a pod disruption budget and what does it actually protect?

A pod disruption budget (PDB) is a namespaced object that caps how many pods matching a label selector can be taken down at once by voluntary disruptions. It only applies to requests that go through the Eviction API. That covers kubectl drain, cluster autoscaler scale-down, and most node upgrade tooling. When an eviction would push the number of healthy pods below the budget, the API server rejects it, and the caller is expected to wait and retry.

Anything that skips the Eviction API also skips the PDB:

Disruption Respects PDB? What actually governs it
kubectl drain, autoscaler scale-down, upgrade tooling Yes The PDB
Deployment rollout No maxUnavailable / maxSurge on the Deployment
Direct kubectl delete pod No Nothing; the pod is deleted
Node crash or kernel panic No Replica count and pod spreading
OOM kill, failing liveness probe No Resource limits and probe tuning
Zone outage No Topology spread across zones

Backend teams need a PDB because a single drain can evict every replica that happens to sit on the node being drained. Without a budget, an API whose pods were all scheduled together can drop to zero capacity during routine maintenance.

The service team should own the PDB, because they know how many replicas are needed to serve traffic. The platform team owns the drain and upgrade process and should require every production workload to have a sensible budget.

A PDB complements multiple replicas, topology spreading, and graceful shutdown. It does not replace any of them. It offers no protection against node failures, zone failures, or bad rollouts.

Who should use a pod disruption budget?

Use a PDB for any workload where losing several replicas at once during maintenance would cause user-visible errors or data risk, and where nodes are drained as a matter of routine. In practice that means:

  • Replicated backend APIs and gRPC services that sit behind a Service or load balancer and need a minimum number of pods to handle traffic. This is the main case this guide covers.
  • Quorum-based or clustered stateful systems run as StatefulSets, where evicting too many members at once breaks quorum. For these, the budget usually allows exactly one member to be down at a time.
  • Workers that consume from queues and need a minimum level of throughput to keep backlog under control.
  • Any production workload on a cluster with the cluster autoscaler or automated node upgrades. These tools drain nodes without a human in the loop, so the PDB is the only thing telling them how fast they may proceed.

A PDB is a poor fit, or unnecessary, in these cases:

  • Single-replica services. Any budget that protects the pod also blocks every drain. See the last section for the options.
  • Batch Jobs and other restartable work where an eviction only costs a retry.
  • Development and ephemeral clusters, where blocked drains cost more time than brief downtime does.

Ownership follows the split described above. The application team writes and tunes the budget alongside the Deployment. The platform team sets the policy that production workloads must have one and that it must allow at least one disruption.

How does a pod disruption budget work?

Two components cooperate: a controller that keeps the budget's status current, and the Eviction API, which enforces it.

  1. The disruption controller computes status. It runs in kube-controller-manager and watches the pods that match each PDB's selector. It then writes four status fields on the PDB:
    • expectedPods: the replica count of the owning controller. Kubernetes reads this from the scale of the Deployment, ReplicaSet, StatefulSet, or ReplicationController.
    • currentHealthy: the matching pods that are Ready.
    • desiredHealthy: the minimum healthy count, resolved from minAvailable or maxUnavailable against expectedPods.
    • disruptionsAllowed: currentHealthy minus desiredHealthy, floored at zero.
  2. Callers request evictions instead of deletions. kubectl drain, the autoscaler, and upgrade tooling create an Eviction object for the pod's eviction subresource. They do not issue a plain DELETE.
  3. The API server checks the budget. If disruptionsAllowed is greater than zero, the API server decrements it, records the pod as disrupted in the PDB status, and deletes the pod gracefully. Recording the pod prevents two concurrent evictions from both spending the same allowance before the controller has observed the first deletion. If disruptionsAllowed is zero, the request fails with HTTP 429, and the caller retries later.
  4. The budget refills as replacements become Ready. The ReplicaSet or StatefulSet controller creates a replacement pod. Once that pod passes its readiness probe, currentHealthy goes back up, the controller recomputes disruptionsAllowed, and the next eviction can proceed.

The rest of this guide follows from this loop. The budget counts only Ready pods. It applies only to callers that use the Eviction API. It needs a controller with a scale so that it can resolve maxUnavailable and percentage values. For pods without such a controller, only an integer minAvailable works.

How does kubectl drain interact with a pod disruption budget?

Draining a node involves several separate steps, and the PDB only takes part in one of them.

  1. Cordon. kubectl cordon <node> marks the node unschedulable, so no new pods land there. kubectl drain cordons the node automatically, but cordoning first lets you confirm the node is out of rotation before anything is evicted.
  2. Evict. kubectl drain <node> --ignore-daemonsets sends a request through the Eviction API for each pod on the node. Add --delete-emptydir-data when pods use emptyDir volumes, because that data is lost on eviction and drain will not proceed without your explicit acknowledgement. DaemonSet pods are skipped because their controller would immediately recreate them on the same node.
  3. Budget check. For each eviction request, the API server checks the matching PDB. If evicting the pod would drop the workload below its budget, the request is refused with HTTP 429 Too Many Requests, and drain keeps retrying it.
  4. Replacement. The ReplicaSet creates a new pod for each evicted one, and the scheduler places it on another node. The next eviction for that workload is allowed only once the replacement is Ready.
  5. Uncordon. After maintenance, run kubectl uncordon <node> to make the node schedulable again.

The PDB counts only Ready pods as healthy. A pod that is Running but failing its readiness probe does not count toward the budget, so it does not free up room for another eviction. A readiness probe that never passes can therefore stall a drain indefinitely. A slow startup does not stall the drain, but it does extend it.

Illustrative estimate (assumed numbers, not a benchmark): drain time ≈ pods on node from the workload × (graceful termination + scheduling + time to Ready) = 3 × (10 s + 5 s + 45 s) = 180 s.

#!/usr/bin/env bash
set -euo pipefail

NODE="worker-node-3"   # placeholder node name
NS="orders-api"        # namespace of the backend API

kubectl cordon "$NODE"
kubectl get node "$NODE"   # STATUS should include SchedulingDisabled

kubectl get pods -n "$NS" -o wide --field-selector spec.nodeName="$NODE"

kubectl get pdb -n "$NS"

kubectl drain "$NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=10m

kubectl uncordon "$NODE"

This script cordons a node, checks the PDB's allowed disruptions, drains the node with the DaemonSet and emptyDir flags and a timeout that bounds a stalled drain, and uncordons the node after maintenance.

Should you use minAvailable or maxUnavailable?

Integer and percentage values behave differently at small replica counts. Kubernetes rounds percentages up for both fields. That rounding is easy to misread when a Deployment runs only a handful of pods.

Spec Replicas Resolved value Evictions allowed when all pods are healthy
minAvailable: 3 3 3 must stay up 0, so the drain blocks
minAvailable: 90% 3 ceil(2.7) = 3 0, so the drain blocks
maxUnavailable: 1 3 1 may be down 1 at a time
maxUnavailable: 10% 3 ceil(0.3) = 1 1 at a time

Illustrative example, not measured. With 3 replicas and minAvailable: 3, the budget leaves no room for any eviction, so kubectl drain retries until it times out. Changing to maxUnavailable: 1 lets the drain evict one pod, wait for its replacement to become Ready, and then evict the next.

For workloads scaled by an HPA, maxUnavailable is usually the better choice. Suppose you set minAvailable: 4 because the service normally runs 6 pods. When the HPA scales down to 4 overnight, the budget allows zero disruptions. maxUnavailable is calculated against the current expected replica count, so it permits roughly the same number of evictions at any scale.

Replica count alone is the wrong input. The question is how many pods can absorb peak traffic without breaching latency targets. If 5 of 6 pods are needed at peak, maxUnavailable: 1 is the ceiling, even though 2 would drain nodes faster. A higher value shortens drains and upgrades, but it reduces headroom. If a pod crashes or traffic spikes during a drain, the remaining pods have less spare capacity to absorb the extra load.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api-pdb
  namespace: backend
spec:
  # Assumption: peak load needs 5 of 6 pods, so only one pod may be down at a time.
  # A value of 2 would drain nodes faster but leave no headroom at peak.
  maxUnavailable: 1
  # Avoid `minAvailable: 3` when the Deployment runs 3 replicas.
  # It resolves to zero allowed disruptions, so `kubectl drain` blocks.
  selector:
    matchLabels:
      app: orders-api
---

This PodDisruptionBudget uses maxUnavailable: 1 so a drain evicts one API pod at a time at any HPA scale, and the comments show how to check allowed disruptions before running kubectl drain.

How do you implement a pod disruption budget for a backend API?

Start with label alignment. The PDB's spec.selector must match the labels on the Deployment's pod template, not the Deployment's own metadata labels. A selector that matches nothing produces a PDB that reports zero expected pods and protects nothing. Use one dedicated label such as app: orders-api, and make sure no second PDB selects the same pods. The eviction API refuses to evict pods covered by more than one budget.

Graceful shutdown

The budget limits how many pods leave at once. It does nothing about requests in flight when a pod leaves. On eviction, the kubelet runs the preStop hook, sends SIGTERM, and waits up to terminationGracePeriodSeconds before sending SIGKILL. Endpoint removal happens in parallel, so load balancers may keep routing to the pod for a few seconds. A short preStop sleep keeps the process serving while it drops out of rotation. After that, the SIGTERM handler should stop accepting connections, finish in-flight requests, and exit.

Illustrative estimate (assumed numbers, not a benchmark): grace period ≥ preStop delay + longest request + margin. With 5 s + 20 s + 5 s, the grace period is 30 s.

A satisfiable budget

With minAvailable: 2, run at least three replicas. Spread them with topology spread constraints or pod anti-affinity on kubernetes.io/hostname, so a single drain never takes two replicas at once. Keep spare capacity or autoscaler headroom so evicted pods can reschedule. If they cannot, the next node's drain stalls.

Once this is in place, drains need no supervision. Nobody has to watch dashboards or cordon nodes by hand during upgrades.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3                        # minAvailable 2 + 1 pod that may be evicted
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api              # the PDB selects these pod-template labels
    spec:
      terminationGracePeriodSeconds: 30  # 5s preStop + 20s longest request + 5s margin
      topologySpreadConstraints:     # avoid stacking replicas on one node
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: orders-api
      containers:
        - name: api
          image: registry.example.com/orders-api:1.4.2
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "5"]  # keep serving while the pod leaves endpoints; SIGTERM follows
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
spec:
  minAvailable: 2                    # drains evict at most one replica at a time
  selector:
    matchLabels:
      app: orders-api                # must match the pod template labels exactly

A three-replica Deployment spread across nodes, with a preStop delay and a 30-second grace period, paired with a PodDisruptionBudget whose selector matches the pod template labels exactly.

Why is a pod disruption budget blocking a drain, and how do you debug it?

Start with kubectl get pdb -A. The column that matters is ALLOWED DISRUPTIONS. If it reads 0, every eviction against pods that PDB selects will be refused, and kubectl drain will keep retrying until it times out or you interrupt it. Then run kubectl describe pdb <name> and compare the current healthy and desired healthy counts. Allowed disruptions is roughly current healthy minus desired healthy, floored at zero. If you are already at or below the desired count, the budget has nothing left to give.

Selector hygiene

Most blocked or useless PDBs come from the label selector:

  • Matches nothing. The expected pods count is 0. This PDB never blocks a drain, but it also protects nothing, so the outage it was meant to prevent can still happen. Look for this after a Deployment's labels are renamed.
  • Matches the wrong pods. A broad selector such as tier: backend can capture a single-replica worker. With minAvailable: 1, that worker can never be evicted.
  • Overlapping PDBs. If two budgets select the same pod, the eviction API refuses the request outright instead of choosing one. Run kubectl get pods -l <selector> for each PDB to find the overlap.

Never-Ready pods

A pod that is crashlooping or failing its readiness probe does not count as healthy. It lowers the current healthy count and can drive allowed disruptions to zero. Under the default behavior, the broken pod itself may also be unevictable, which stalls the drain on a pod that serves no traffic. Some clusters support an unhealthy-pod eviction policy on the PDB spec that lets those pods be evicted. Check that your cluster version and API actually expose this field before you rely on it. Otherwise, fix or delete the failing pod first.

#!/usr/bin/env bash
set -euo pipefail
NS=payments-api        # placeholder namespace
PDB=orders-api-pdb     # placeholder PDB name

kubectl get pdb -A

kubectl get pdb "$PDB" -n "$NS" -o go-template='expected={{.status.expectedPods}} current={{.status.currentHealthy}} desired={{.status.desiredHealthy}} allowed={{.status.disruptionsAllowed}}{{"\n"}}'
kubectl describe pdb "$PDB" -n "$NS"

for p in $(kubectl get pdb -n "$NS" -o name); do
  sel=$(kubectl get "$p" -n "$NS" -o go-template='{{range $k,$v := .spec.selector.matchLabels}}{{$k}}={{$v}},{{end}}')
  echo "== $p selector: ${sel%,}"
  kubectl get pods -n "$NS" -l "${sel%,}" -o wide
done

kubectl get pods -n "$NS" -o go-template='{{range .items}}{{.metadata.name}} {{range .status.conditions}}{{if eq .type "Ready"}}Ready={{.status}}{{end}}{{end}}{{"\n"}}{{end}}'

kubectl explain poddisruptionbudget.spec.unhealthyPodEvictionPolicy

A bash debugging pass that shows PDB status fields, lists the pods each budget selects so you can spot empty, wrong, or overlapping selectors, finds never-Ready pods, and checks whether the cluster's API exposes the unhealthy-pod eviction policy field.

How do you run cluster upgrades without breaking PDB-protected services?

Before the upgrade

List every PDB in the cluster and check the ALLOWED DISRUPTIONS column from kubectl get pdb --all-namespaces. Any budget showing zero will block a drain before you start. Contact those owners first: the service may be under-replicated, a rollout may be stuck, or the budget may be set as strictly as the replica count allows. Resolving this a day early is cheaper than finding out mid-upgrade.

During the drain

Drain one node at a time. Wait for the evicted pods to be rescheduled and Ready before you move on. A successful drain only shows that evictions passed the budget. It does not show that the API stayed healthy. Watch request error rates and p99 latency on the services you are disrupting. Remaining pods can be Ready while saturated, and a cold cache or connection-pool warmup on the new pods can still push latency up. If the metrics degrade, pause the upgrade even if pod counts look fine.

When a drain hangs

Decide the policy before the upgrade: how long you wait (the --timeout flag on kubectl drain puts a limit on it), who you page, and who can approve overriding a budget. If you make these decisions under pressure at 2 a.m., people tend to delete PDBs.

Why conventions matter

Agree on a shared rule, such as "every production API runs at least three replicas with maxUnavailable: 1", and publish a report of zero-disruption budgets. Upgrades then become routine. Platform teams can plan maintenance windows without negotiating with each app team, and app owners know what is expected of them. Because each upgrade follows the same steps, it takes less engineering time to plan and causes fewer incidents.

How can a pod disruption budget save engineering time?

Without a PDB, the only safe way to drain a node that hosts critical APIs is to coordinate by hand. Someone checks which replicas sit on the node, asks the owning team whether losing them is acceptable, drains, watches the pods come back, and only then moves to the next node. A correct budget turns that coordination into a rule the API server enforces. The time saved comes from several places:

  • Unattended drains. kubectl drain, the cluster autoscaler, and upgrade tooling pace themselves against the budget. Nobody needs to sequence evictions per service or watch replica counts while nodes roll.
  • Fewer maintenance meetings. The budget records the service team's answer to "how many pods can we lose?" in the cluster itself. The platform team no longer has to ask before each upgrade.
  • Fewer maintenance-caused incidents. A drain that cannot take every replica of an API at once cannot cause the zero-capacity outage described at the start of this guide. Each avoided incident saves the time spent on paging, mitigation, and the postmortem.
  • Faster diagnosis when something does stall. The ALLOWED DISRUPTIONS column and the PDB status fields point straight at the workload that is blocking. Engineers do not have to work out from scratch why a node will not empty.
  • Autoscaler savings without the risk. Scale-down can consolidate underused nodes safely, so teams can let it run instead of disabling it on nodes that host sensitive workloads.

Illustrative estimate (assumed inputs, not a measured result): assume a cluster with 30 nodes, an upgrade every quarter, and 10 minutes of manual coordination and watching per node when no budgets exist. That comes to 30 × 10 = 300 minutes, or 5 engineer-hours, per upgrade. Over four upgrades a year, that is 20 engineer-hours. With correct budgets, the remaining manual work is the pre-upgrade check for zero-disruption PDBs plus monitoring error rates and latency. Assume 1 hour per upgrade for that work, or 4 hours a year. Substitute your own node count, upgrade frequency, and per-node effort. The saving grows with cluster size and with how often nodes are recycled.

These savings depend on budgets that can be satisfied. A PDB that allows zero disruptions turns the saved time back into manual intervention, as the next section explains.

When should you avoid a strict PDB, and what are the alternatives?

A budget that permits zero voluntary disruptions, such as minAvailable equal to the replica count or maxUnavailable: 0, does not make a service safer. It stops every drain until someone intervenes. Under upgrade deadlines, that person usually bypasses the budget completely, and the service ends up less protected than a looser PDB would have left it. A single-replica service has the same problem. Any PDB that actually protects it will block drains, so the realistic options are to add replicas or to accept a short outage during maintenance.

Escape hatches and their cost

  • kubectl drain --disable-eviction deletes pods directly instead of going through the Eviction API, so PDBs are skipped. Use it only when you have confirmed the workload can tolerate the loss.
  • --timeout on drain, followed by force-deleting stuck pods, is for nodes that are already failing. Force deletion removes the API object without waiting for the kubelet to confirm the containers have stopped, so StatefulSet pods with stable identities can end up running twice.
  • Upgrade tooling may stop respecting a blocking PDB after a grace period, because node replacement cannot wait forever. If your budget blocks drains indefinitely, the tooling will eventually decide when your pods are evicted.

Alternatives and complements

Approach What it fixes Limitation
More replicas Lets a PDB allow evictions while capacity stays up Costs more
Topology spreading Prevents one drain from taking out most of your replicas Needs enough nodes or zones
Graceful shutdown Lets each evicted pod finish in-flight requests Doesn't limit how many pods are evicted at once
Rollout maxUnavailable/maxSurge Controls disruption during deploys No effect on node drains
Maintenance windows Makes accepted downtime predictable Downtime still happens

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to pod disruption budget, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.