What are Kubernetes topology spread constraints and why do high availability pods need them?

Topology spread constraints are a field in the pod spec, topologySpreadConstraints, that tells the scheduler how evenly a group of matching pods must be distributed across a topology domain. A domain is defined by a node label. The common choices are topology.kubernetes.io/zone for availability zones and kubernetes.io/hostname for individual nodes. You state how much imbalance you tolerate and whether exceeding it should block scheduling or only lower a node's score.

The problem they solve is easy to miss. A Deployment with replicas: 3 guarantees three running pods. It does not guarantee that those pods run in three zones. The scheduler does apply a default spreading behavior, but that default is a soft preference. It is one scoring signal among several, and resource fit, node affinity, image locality, and capacity can outweigh it. Suppose one zone has the most free capacity after a scale-up, or the other zones are full during a rollout. In that case, all three replicas can land in the same zone. When that zone fails, the service goes down even though it looked redundant.

You should care if you run replicated, stateless or quorum-based services and have a multi-zone cluster. Examples include API backends, ingress controllers, and consensus systems. Explicit zone spreading is warranted when losing one zone must not take the service down, or when replicas sit behind a load balancer that assumes zonal redundancy. It also matters when a PodDisruptionBudget only protects you if replicas are actually spread.

How do topology spread constraints work?

Each entry in topologySpreadConstraints names a topologyKey, a node label whose distinct values define the domains. With topology.kubernetes.io/zone, every zone is a domain. With kubernetes.io/hostname, every node is a domain. The labelSelector decides which pods are counted. It usually matches the workload's own pods, so replicas spread relative to each other rather than to unrelated pods.

maxSkew bounds imbalance. For each candidate node, the scheduler counts matching pods in that node's domain, adds the pod being placed, and subtracts the global minimum count across eligible domains. If the result exceeds maxSkew, the node fails the constraint.

whenUnsatisfiable sets what happens on a violation. DoNotSchedule filters the node out. ScheduleAnyway only lowers the node's score.

Worked example: 5 replicas, 3 zones, maxSkew: 1

  • 2/2/1 is allowed. The largest gap between zones is 1.
  • 3/1/1 is rejected. It reaches a gap of 2.
  • 3/2/0 is rejected. It also reaches a gap of 2.

In practice, the fifth pod may land in any zone, but the sixth must go to the zone holding one pod.

Eligible domains

Only nodes carrying the topologyKey label count toward skew. By default, the pod's nodeSelector and required node affinity also narrow the set of nodes. This prevents a zone the pod could never run in from holding the minimum at zero. Two fields control this behavior:

  • nodeAffinityPolicy (default Honor) controls whether node affinity and nodeSelector narrow the domains.
  • nodeTaintsPolicy (default Ignore) controls whether nodes with untolerated taints are excluded.

minDomains applies only with DoNotSchedule. If fewer eligible domains exist than this value, the global minimum is treated as zero. That leaves pods Pending, which can signal an autoscaler to add nodes in new zones.

These fields arrived at different times. Check that your cluster version supports them before relying on them.

 +---------------------------+     +------------------------------------+
 | New pod (6th replica)     |---->| 1. Eligible nodes                  |
 | labelSelector: app=web    |     | - carry the topologyKey label      |
 | maxSkew: 1                |     | - nodeAffinityPolicy=Honor:        |
 | whenUnsatisfiable:        |     |   nodeSelector/affinity narrow     |
 |   DoNotSchedule           |     | - nodeTaintsPolicy=Ignore:         |
 +---------------------------+     |   untolerated taints not used      |
                                   +-----------------+------------------+
                                                     |
                                                     v
 +----------------------------------------------------------------------+
 | 2. Pods matching labelSelector, grouped by zone (topologyKey)        |
 |   +-------------+      +-------------+      +-------------+          |
 |   | zone-a      |      | zone-b      |      | zone-c      |          |
 |   | [P] [P]     |      | [P] [P]     |      | [P]         |          |
 |   | count = 2   |      | count = 2   |      | count = 1   |          |
 |   +------+------+      +------+------+      +------+------+          |
 |          |                    |                    |                 |
 +----------------------------------------------------------------------+
            v                    v                    v
 +----------------------------------------------------------------------+
 | 3. skew = (domain count + 1) - global min across domains (= 1)       |
 |   3 - 1 = 2 > 1        3 - 1 = 2 > 1        2 - 1 = 1 <= 1           |
 |   FILTERED             FILTERED             ALLOWED                  |
 +----------------------------------------------------------------------+
   minDomains (DoNotSchedule only): if eligible domains < minDomains,
   the global min is treated as 0.

With five replicas spread 2/2/1 across three eligible zones and maxSkew: 1, the scheduler computes skew per zone for the sixth pod and filters out zone-a and zone-b, leaving zone-c as the only allowed placement.

How do you implement zone and node spreading in a Deployment?

A common default for a stateless Deployment uses two constraints. The first sets topologyKey: topology.kubernetes.io/zone, maxSkew: 1 and whenUnsatisfiable: DoNotSchedule. Losing a zone is the failure you are designing for, so the scheduler should leave a pod Pending rather than stack replicas in one zone. The second sets topologyKey: kubernetes.io/hostname with ScheduleAnyway. You still want replicas spread across nodes, but a hard per-node rule can block scheduling during node drains or when a zone has few nodes. As a soft rule, it only adjusts scoring.

The labelSelector in each constraint must match the labels in the pod template, usually the same labels as the Deployment's spec.selector. The scheduler counts every pod in the namespace that matches the selector. Mistakes here produce no errors:

  • Mismatched selector. The scheduler counts zero pods in every domain, so the constraint is always satisfied and has no effect.
  • Overly broad selector, such as tier: backend. The scheduler balances your pods against unrelated workloads, so this Deployment can end up skewed while the combined set looks even.

During a rolling update, old and new ReplicaSets both match the selector, so skew is calculated across the two revisions together. Adding matchLabelKeys: [pod-template-hash] limits counting to pods from the same revision. Support for this field depends on your cluster version.

Spread constraints apply only at scheduling time. They do not stop kubectl drain or other voluntary evictions from removing every replica in a zone. Add a PodDisruptionBudget with the same selector and a maxUnavailable value that keeps enough replicas running while pods are evicted.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 6
  selector:
    matchLabels: {app: web}
  template:
    metadata:
      labels: {app: web}   # every labelSelector below must match these labels
    spec:
      topologySpreadConstraints:
        # Hard: zone loss is the failure we design for, so stay Pending rather than stack a zone
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels: {app: web}   # not a broad label like tier: backend
          matchLabelKeys: [pod-template-hash]   # count only this revision's pods; depends on cluster version
        # Soft: prefer different nodes, but only affect scoring so drains and small zones don't block
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: {app: web}
          matchLabelKeys: [pod-template-hash]
      containers:
        - name: web
          image: registry.example.com/web:1.4.2
---
# Spread rules apply only at scheduling time; this limits voluntary evictions such as kubectl drain
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  maxUnavailable: 1   # evict at most one replica at a time
  selector:
    matchLabels: {app: web}   # same selector as the Deployment

A Deployment with a hard zone spread constraint and a soft hostname constraint, both scoped to the pod template's labels and the current revision. It is paired with a PodDisruptionBudget that limits how many replicas voluntary evictions can remove at once.

How do you verify pod placement and debug Pending pods?

Start by confirming where the replicas actually landed. kubectl get pods -o wide with your app's label selector shows the node for each pod. Then run kubectl get nodes -L topology.kubernetes.io/zone to add a zone column to the node list. Matching the two outputs gives you a per-zone count. Check this count directly. A Deployment that reports all replicas ready can still have every pod in one zone if the constraint is soft or misconfigured.

When pods stay Pending, run kubectl describe pod on one of them and read the Events section. The scheduler records a FailedScheduling event that totals the reasons nodes were rejected. A message such as "node(s) didn't match pod topology spread constraints" means a hard constraint blocked placement. Other reasons in the same message, such as taints, insufficient CPU or a node affinity mismatch, tell you why the remaining nodes were ruled out.

The usual causes:

  • A zone with no eligible nodes. Taints, node affinity or exhausted capacity can leave a zone with nothing the scheduler can use. A strict maxSkew of 1 then caps the other zones as well, so new pods have nowhere to go.
  • Selector mismatch. If the constraint's labelSelector doesn't match the pod's own labels, the scheduler counts the wrong set of pods. It may count zero pods, or pods from another workload, so spreading either does nothing or blocks placement for no clear reason.
  • Rolling updates. Old and new ReplicaSet pods share the app label, so during a rollout they are counted together. The surge pods can look like skew, or the old pods can hide a lopsided new ReplicaSet. Adding pod-template-hash to matchLabelKeys scopes the count to each revision.

Put the placement check in a script or CI step and run it after each rollout. That way you catch skew right after the change that caused it, before an incident forces you to investigate.

#!/usr/bin/env bash
# Post-rollout placement check: per-zone pod counts plus scheduler reasons for Pending pods.
set -euo pipefail
NS="${NS:-default}"
SELECTOR="${SELECTOR:-app=web}"

# Raw views. The pod-template-hash column separates old and new ReplicaSet pods.
kubectl get pods -n "$NS" -l "$SELECTOR" -o wide -L pod-template-hash
kubectl get nodes -L topology.kubernetes.io/zone

# Build a node -> zone map. Nodes without the label show up as <none>.
declare -A zone_of
while read -r node zone; do
  zone_of["$node"]="${zone:-<none>}"
done < <(kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}{end}')

# Count scheduled pods per zone. Unscheduled pods have no nodeName and are skipped.
declare -A count
while read -r node; do
  [[ -z "$node" ]] && continue
  z="${zone_of[$node]:-<unknown>}"
  count["$z"]=$(( ${count[$z]:-0} + 1 ))
done < <(kubectl get pods -n "$NS" -l "$SELECTOR" -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}')

echo "Pods per zone:"
for z in "${!count[@]}"; do echo "  $z: ${count[$z]}"; done

# For any Pending pod, print the Events section that contains FailedScheduling reasons.
pending=$(kubectl get pods -n "$NS" -l "$SELECTOR" --field-selector=status.phase=Pending -o name)
if [[ -n "$pending" ]]; then
  first=$(echo "$pending" | head -n 1)
  echo "Pending pods found. Scheduler events for $first:"
  kubectl describe -n "$NS" "$first" | sed -n '/^Events:/,$p'
  exit 1  # Fail the CI step so a blocked rollout is not reported as healthy
fi

A repeatable bash check that maps pods to zones, prints per-zone counts with the ReplicaSet hash visible, and shows the FailedScheduling events for a Pending pod, failing the run if any pod is Pending.

Topology spread constraints vs pod anti-affinity: what are the alternatives?

Pod anti-affinity says "not together." Required anti-affinity on topology.kubernetes.io/zone allows at most one matching replica per zone. On a three-zone cluster, the fourth replica stays Pending no matter how much capacity is free. Preferred anti-affinity only adds a scoring weight. It never limits imbalance, so under resource pressure the scheduler can still stack most replicas in one zone. Spread constraints say "evenly, within a tolerance." maxSkew limits the difference between zones rather than the absolute count, so the same constraint works at 3 replicas or 300.

Mechanism Expresses Caps imbalance Scales with replicas Typical use
Required pod anti-affinity Never co-locate Only as one per domain No Strictly one per node or zone, e.g. quorum members
Preferred pod anti-affinity Avoid co-locating No Yes, but no guarantee Soft separation hints
Scheduler default spreading Loose, soft spread No, scoring only Yes Baseline for workloads with no constraints
Cluster default constraints Platform-chosen spread Only if set to DoNotSchedule Yes Org-wide policy
Per-workload constraints Explicit spread and tolerance Yes, with DoNotSchedule Yes Services with availability targets

Cluster-level defaults are set in the PodTopologySpread plugin's arguments in the scheduler configuration. They apply only to pods that define no constraints of their own, and they derive the selector from the owning Service or controller. Treat them as a safety net for workloads nobody tuned. Making them hard would leave pods Pending for teams that never opted in. For any service with an availability target, require explicit per-workload constraints and enforce that with admission policy or CI. That way the tolerance is visible in the manifest and reviewed alongside the code it protects.

When should you avoid hard zone spreading?

Hard spreading turns an availability preference into a scheduling precondition. That works when capacity is plentiful and balanced. It backfires in these situations:

  • Zone outage. With DoNotSchedule and a small maxSkew, the failed zone still counts as a domain holding few or no matching pods. Replacement pods in the surviving zones would exceed the allowed skew, so they stay Pending during the incident the constraint was meant to protect against.
  • Soft fallback. ScheduleAnyway only adds a scoring penalty. Under pressure, the scheduler can place most or all replicas in one zone, and nothing moves them back after capacity returns.
  • Unbalanced capacity. If node groups have different sizes per zone, or the autoscaler cannot add nodes in the specific zone the constraint requires, hard constraints block scheduling even when the cluster has free capacity elsewhere.
  • Zonal volumes. A StatefulSet pod bound to a zonal persistent volume is pinned to that zone by volume node affinity. Spreading only influences the first placement.
  • Stacked constraints. Combining zone, node, and custom-label constraints with affinity rules makes Pending events hard to interpret, because any one rule can produce the rejection.
Scenario DoNotSchedule ScheduleAnyway
Zone outage Replacements may stay Pending Replacements land in surviving zones
Capacity shortage in one zone Pods Pending until that zone grows Pods placed elsewhere, skew grows
Autoscaler can't target zone Pending indefinitely Scheduled, unevenly spread
Zonal PV already bound Volume affinity wins; may conflict Volume affinity wins
Steady state, balanced nodes Even spread guaranteed Usually even, not guaranteed

What happens after scheduling, and what does zone spreading cost?

Constraints only act at scheduling time

The scheduler evaluates spread constraints when it places a pod. It does not move running pods. A node drain, a cluster autoscaler scale-down, or a rolling update that lands during a zone capacity shortage can leave a workload at three replicas in one zone and one in another, and nothing corrects it automatically.

There are two realistic remedies. A rolling restart (kubectl rollout restart) recreates the pods so the scheduler places them again, but the result depends on capacity at that moment. The community descheduler project can evict pods that violate topology spread constraints. It only evicts them, so the scheduler still decides where replacements go. Pair it with PodDisruptionBudgets so evictions cannot take down too many replicas at once.

The ongoing cost

Spreading replicas puts callers and callees in different zones. Every cross-zone hop adds network latency. A chatty service that makes many sequential calls per request accumulates that latency. Some cloud providers also bill for cross-zone data transfer.

Illustrative estimate (assumed numbers, not a benchmark): monthly cost = cross-zone GB × per-GB rate. If you assume 5,000 GB per month at $0.01/GB, the result is $50 per month for one service. Replace both inputs with your own traffic and pricing.

Engineering time also adds up. Each team that writes its own constraints spends time on Pending-pod debugging and skew investigations. Standardizing a vetted default template, or setting cluster-level defaults through the scheduler's PodTopologySpread configuration where your platform allows it, removes most of that repeated work.

# Cluster-level default spread rules for the kube-scheduler.
# Pass this file to kube-scheduler with --config. Managed control planes
# often do not expose it; in that case, use a shared workload template instead.
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    pluginConfig:
      - name: PodTopologySpread
        args:
          # Applies only to pods that define no topologySpreadConstraints
          # of their own. Pods with their own constraints replace these.
          defaultConstraints:
            # Prefer an even spread across zones, but never block scheduling.
            # This avoids Pending pods during a zone capacity shortage.
            - maxSkew: 1
              topologyKey: topology.kubernetes.io/zone
              whenUnsatisfiable: ScheduleAnyway
            # Spread across nodes inside each zone as a softer secondary goal.
            - maxSkew: 2
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: ScheduleAnyway
          # Default constraints must not set labelSelector. The scheduler
          # derives the selector from the pod's Service, ReplicaSet,
          # StatefulSet or ReplicationController.
          # "List" uses the constraints above instead of the built-in defaults.
          defaultingType: List
# Like any spread constraint, these act only when a pod is placed.
# They do not rebalance running pods after drains or scale-down.

A KubeSchedulerConfiguration that sets soft zone and node spread defaults for every pod without its own constraints, so teams do not each write and debug their own constraints.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes topology spread constraints, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.