What are Kubernetes topology spread constraints and which fields matter?

topologySpreadConstraints is a list in the pod spec, usually set in a Deployment's pod template, that tells the scheduler how evenly to place matching pods across groups of nodes. Each group is a topology domain: every node that shares the same value for a given label. This guide is for backend and platform engineers running stateless API Deployments who want zone spreading and high availability pods without hand-tuned pod anti-affinity rules.

Each constraint uses four core fields:

Field What it controls Typical values for an API
topologyKey The node label that defines a domain topology.kubernetes.io/zone, kubernetes.io/hostname
maxSkew Largest allowed difference in matching pod count between the most and least loaded domains 1
whenUnsatisfiable Whether the constraint is a hard filter or a soft preference DoNotSchedule (hard), ScheduleAnyway (soft)
labelSelector Which pods are counted in each domain The Deployment's own app label

topologyKey sets the failure boundary you care about. Zones protect against losing a zone. Hostnames protect against losing a single node.

maxSkew must be at least 1. A value of 1 is the tightest spread you can ask for.

whenUnsatisfiable decides what happens when the constraint can't be met:

  • DoNotSchedule removes nodes that would break the skew, so the pod stays Pending if no node qualifies.
  • ScheduleAnyway only lowers the score of those nodes, so the pod still runs.

labelSelector is the field people get wrong most often. If it is too broad, pods from other workloads distort the counts. If it is too narrow, for example by including pod-template-hash, pods from the old ReplicaSet are not counted during a rollout.

Who should use topology spread constraints?

Topology spread constraints fit any workload that runs more than one replica and needs to survive the loss of a node or a zone. The most common users are:

  • Teams running stateless services, such as HTTP APIs, gRPC services and queue consumers, with several replicas behind a Service. These workloads tolerate some imbalance but should not have every replica in one zone or on one node.
  • Platform engineers who own shared clusters and want a consistent placement policy. They can publish a standard constraint block for service teams, or set cluster-level defaults in the scheduler configuration.
  • Teams running on multi-zone clusters with autoscaling, where node counts change often and fixed one-per-node rules become a source of Pending pods.

They are a weaker fit in a few cases:

  • Single-replica workloads. There is nothing to spread.
  • Single-zone or very small clusters. A zone constraint has only one domain to work with, and a node constraint on two or three nodes gives little room for balance.
  • Quorum systems and workloads that saturate a node-local resource. These usually need the strict exclusion of required pod anti-affinity, covered later in this guide, sometimes combined with a soft zone spread.

Spreading also assumes your nodes carry accurate topology labels. On managed cloud clusters the zone label is normally populated for you. On self-managed or on-premises clusters, confirm the labels exist with kubectl get nodes -L topology.kubernetes.io/zone before relying on a zone constraint.

How do topology spread constraints work?

Topology spread constraints are implemented by the scheduler's PodTopologySpread plugin. They act only at scheduling time, when the scheduler picks a node for a new or replacement pod. The process for one pod looks like this:

  1. Group nodes into domains. For each constraint, the scheduler reads the topologyKey label on every eligible node and groups nodes by its value. All nodes with topology.kubernetes.io/zone=zone-a form one domain, for example.
  2. Count matching pods per domain. The scheduler counts existing pods that match the constraint's labelSelector in each domain.
  3. Filter (hard constraints). For each DoNotSchedule constraint, the scheduler works out what the skew would be if the pod landed on each candidate node. Nodes that would push skew above maxSkew are removed from consideration. If every node is removed, the pod stays Pending and a FailedScheduling event is recorded.
  4. Score (soft constraints). For each ScheduleAnyway constraint, nodes that would increase imbalance get a lower score. That score is combined with other scoring plugins, such as those that consider resource usage, so a soft constraint is a preference and can be outweighed.
  5. Bind. The scheduler places the pod on the highest-scoring node that survived filtering.

When a pod has several constraints, all hard constraints must pass at the same time, and all soft constraints contribute to the score. The constraints are evaluated independently. A hostname constraint and a zone constraint each compute their own domains and skew.

Two consequences follow from this design. First, the scheduler never moves running pods, so balance can drift after drains, scale-down or outages. A later section covers rebalancing. Second, the result depends on which nodes count as eligible, which the next section explains in detail.

How does the scheduler calculate skew?

Take three zones, five replicas and maxSkew: 1. Skew is the difference between the most-populated and least-populated eligible domain. A 3/1/1 split has a skew of 2, and 3/2/0 has a skew of 3. The only layout that stays within 1 is 2/2/1, in any order.

Now the sixth pod arrives with zones at A=2, B=2, C=1. For each candidate node, the scheduler takes the matching pods in that node's zone, adds one for the incoming pod, and subtracts the global minimum across eligible zones:

  • A node in zone A gives 3 − 1 = 2, which exceeds maxSkew. Zone B gives the same result.
  • A node in zone C gives 2 − 1 = 1. The pod lands there and the zones end at 2/2/2.

With whenUnsatisfiable: DoNotSchedule, nodes in A and B are removed during filtering. If C has no node with room, the pod stays Pending. With ScheduleAnyway, A and B stay in the running but score lower, so a resource shortage in C can still push the pod into A.

Domains are built only from nodes that are eligible for the pod. If the pod's nodeSelector or node affinity excludes every node in zone C, then C is not counted at all. The scheduler computes skew across A and B only, and you lose zone coverage without any warning.

Three fields change this behavior:

  • nodeAffinityPolicy controls whether node affinity affects which nodes count.
  • nodeTaintsPolicy controls whether taints affect which nodes count.
  • minDomains treats the global minimum as zero until enough domains exist. This forces spreading onto new zones or nodes as they appear.

Check your cluster's documentation to confirm these fields are available and to find their defaults.

              +------------------------------+
              | Pod 6 arrives (maxSkew: 1)   |
              | current: A=2  B=2  C=1       |
              | global min (eligible) = 1    |
              +---------------+--------------+
                              |
         +--------------------+--------------------+
         v                    v                    v
+------------------+ +------------------+ +------------------+
| Zone A node      | | Zone B node      | | Zone C node      |
| (2+1) - 1 = 2    | | (2+1) - 1 = 2    | | (1+1) - 1 = 1    |
| 2 > maxSkew      | | 2 > maxSkew      | | 1 <= maxSkew     |
+--------+---------+ +--------+---------+ +--------+---------+
         |                    |                    |
         +---------+----------+                    |
                   v                               v
+----------------------------------+ +------------------------------+
| DoNotSchedule:  filtered out     | | Passes. Pod lands in C       |
| ScheduleAnyway: kept, lower score| | Result: A=2  B=2  C=2        |
+----------------------------------+ | If C full: DoNotSchedule ->  |
                                     | Pending; ScheduleAnyway ->   |
                                     | may fall back to A or B      |
                                     +------------------------------+

Domains = zones with eligible nodes only (nodeSelector, node affinity,
taints; tuned by nodeAffinityPolicy / nodeTaintsPolicy).
If no node in C is eligible, C drops out: skew is computed over A and B.
minDomains treats the global minimum as 0 until enough domains exist.

How the scheduler evaluates the sixth pod across zones A=2, B=2, C=1 with maxSkew 1: nodes in A and B would produce a skew of 2 and are filtered out under DoNotSchedule or scored lower under ScheduleAnyway, while zone C yields a skew of 1 and receives the pod.

How do you implement a hard zone and soft node spread for an API Deployment?

The recommended default places two entries in the same topologySpreadConstraints list on the Deployment's pod template. The first uses topologyKey: topology.kubernetes.io/zone, maxSkew: 1 and whenUnsatisfiable: DoNotSchedule. Losing a zone is the failure you are designing for, so zone balance is a hard requirement. The second uses topologyKey: kubernetes.io/hostname with whenUnsatisfiable: ScheduleAnyway. With that setting, the scheduler prefers spreading replicas across nodes, but it can still stack them when it has no other option.

A hard node constraint is risky on small or autoscaled pools. Suppose you run six replicas on four nodes with maxSkew: 1. Every node must hold one or two replicas. If you drain a node, or the autoscaler removes one, pods can go Pending even though the remaining nodes have plenty of CPU. The soft rule keeps the preference for spreading without turning it into an availability problem.

Both constraints depend on a correct labelSelector. It must match the pod template labels, and it should identify the app, such as app: orders-api. It should not include a version label or pod-template-hash. Each mistake has a different failure mode:

  • Too broad. A selector like tier: backend counts unrelated workloads, so your replicas get pushed away from zones that hold someone else's pods.
  • Too narrow. Selecting on a version label makes each ReplicaSet balance only against itself during a rollout. The scheduler cannot see the old ReplicaSet's pods, so new pods can pile into one zone while old pods are still running elsewhere.

Constraints set on the pod replace any cluster-level defaults from the scheduler configuration. The two sets are not merged. If you need the platform's default rules as well, restate them in the pod spec.

The engineering-time benefit is that this one declarative block, copied across services with only the app label changed, replaces per-service tuning of affinity weights and anti-affinity terms.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 6
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
        version: v2            # changes per release; keep it out of spread selectors
    spec:
      # Pod-level constraints REPLACE scheduler-config defaults; restate any you still need.
      topologySpreadConstraints:
        # Hard rule: losing a zone is the failure we design for.
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: orders-api  # app identity only: not version, not pod-template-hash
        # Soft rule: prefer one-per-node, but allow stacking during drains or scale-down.
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: orders-api  # same selector so both rules count the same pods
      containers:
        - name: orders-api
          image: registry.example.com/orders-api:2.0.0
          ports:
            - containerPort: 8080

A Deployment with a hard zone spread (maxSkew 1, DoNotSchedule) and a soft node spread (ScheduleAnyway). Both constraints use a labelSelector that matches the app label and leaves out the version label, so old and new ReplicaSets are counted together during a rollout.

How can topology spread constraints save engineering time?

The time savings come from replacing ad hoc, per-service placement work with one policy that the scheduler enforces automatically. The main sources are:

  • One reusable template. The two-constraint block above works for most stateless services with only the app label changed. Teams copy a known-good pattern instead of designing and reviewing affinity rules for each service.
  • Less tuning of affinity weights. Preferred anti-affinity gives you a weight, not a balance target, so teams often adjust weights by trial and error until placement looks right. maxSkew states the target directly, which removes that loop.
  • Fewer Pending-pod investigations after scaling. Required anti-affinity fails as soon as replicas outnumber nodes, which tends to surface during scale-ups and node pool changes. A soft node constraint avoids that class of incident, so fewer people get pulled into debugging it.
  • Central defaults for a whole cluster. Platform teams can set default constraints in the scheduler configuration, so workloads that declare nothing still get a reasonable spread. Service teams only need to act when their requirements differ.
  • Fewer outages caused by poor placement. Replicas concentrated in one zone or on one node turn a routine node failure into a service outage. Preventing that removes the incident response, postmortem and follow-up work that would come with it.
  • Shorter reviews. A declarative constraint is easy to check in code review: the reviewer confirms the topologyKey, maxSkew, whenUnsatisfiable and that the labelSelector matches the pod template. Custom placement logic or scripts take longer to verify.

How much time this saves depends on how many services you run and how often placement problems come up. A simple way to estimate it is to count the placement-related tickets, incidents and review comments from a recent period and ask which of them a standard spread policy would have prevented. Treat any figure produced this way as an illustrative estimate based on assumed inputs, not a measured result.

The savings are not free. You still need to verify placement after rollouts, rebalance after drift, and keep the labelSelector correct as labels change. The later sections cover that work.

What are the alternatives to topology spread constraints, and when is pod anti-affinity still the right choice?

Pod anti-affinity is the older tool for keeping replicas apart, and it answers a different question. Required anti-affinity with topologyKey: kubernetes.io/hostname is a binary exclusion: no two matching pods may share a node. It cannot express "roughly even." On a cluster with four schedulable nodes, a fifth replica stays Pending. Preferred anti-affinity avoids that failure, but it is only a weighted score added to each node. The scheduler can still stack replicas when other scoring factors outweigh it, and nothing bounds how uneven the result can get. Spread constraints state an explicit balance target through maxSkew.

Cost is another reason to move away from anti-affinity. Inter-pod affinity and anti-affinity have historically been expensive to evaluate, because the scheduler must check existing pods across topology domains for every candidate node. The upstream documentation has long advised caution in clusters with many hundreds of nodes.

Mechanism Expresses Failure mode
Required anti-affinity Exactly zero co-located peers Pending once replicas exceed nodes
Preferred anti-affinity A scoring nudge Silent stacking
Topology spread Bounded imbalance Pending (hard) or skew (soft)

Keep required anti-affinity for cases where co-location is itself the fault. One case is quorum members, such as etcd or ZooKeeper, where losing one node must cost at most one vote. Another is workloads that saturate a node-local resource, such as local NVMe or NIC bandwidth.

Avoid stacking a hard zone constraint, a hard node constraint and required anti-affinity. Each one removes candidate nodes. On a cluster with three zones and two nodes per zone, cordoning a single node can leave a replica with nowhere valid to land.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zookeeper
spec:
  serviceName: zookeeper
  replicas: 3
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      affinity:
        podAntiAffinity:
          # Binary exclusion: losing one node costs at most one vote.
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: zookeeper
              topologyKey: kubernetes.io/hostname
      topologySpreadConstraints:
        # Zone balance is a soft goal. Making it DoNotSchedule on top of the
        # hard node rule can leave a replica Pending after one cordon.
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: zookeeper
      containers:
        - name: zookeeper
          image: zookeeper:3.9
          ports:
            - containerPort: 2181

A ZooKeeper StatefulSet uses required anti-affinity for the strict one-per-node rule and a soft zone spread constraint, so the hard rules don't stack into an unsatisfiable placement.

When should you avoid hard zone constraints?

A hard zone constraint has the same weakness. DoNotSchedule holds pods Pending whenever placing them would exceed maxSkew. During a zone outage or a capacity shortage, that is exactly when you need replacement replicas to start somewhere. Some teams switch the zone constraint to ScheduleAnyway for this reason. They accept a temporary imbalance in exchange for pods that keep running.

The hard constraint also protects less than it appears to. Skew is calculated only across domains that have eligible nodes. If a zone loses all its nodes, it drops out of the calculation. The scheduler can then pack every replica into the two remaining zones and still report the constraint as satisfied. Setting minDomains, which works only with DoNotSchedule, closes this gap. When fewer eligible zones exist than the configured minimum, the scheduler treats the global minimum as zero, so pods wait instead of piling up.

Replica count sets a floor on how even the spread can be. Two replicas across three zones always leave one zone empty. Rounding up to a multiple of the zone count fixes the layout, but you pay for that capacity whether or not traffic needs it.

Spreading also has a cost in locality. An API pod in one zone that calls a database or cache in another zone adds network latency to each request. On most clouds it also adds cross-zone data transfer charges.

Finally, the cluster autoscaler may add a node in a zone that does nothing to reduce the skew. Zone-balanced node groups make this less likely, but the behavior depends on your provider and how the node groups are configured.

Why do pods drift out of balance, and how do you rebalance them?

Topology spread constraints are evaluated only when the scheduler places a pod. Once a pod is running, nothing in Kubernetes moves it to restore balance. The skew you see at any moment reflects the cluster as it was when each pod was scheduled.

Several routine events leave a Deployment unbalanced. A node drain evicts pods, and their replacements land wherever capacity exists at that moment, often in the zones that were not drained. Cluster autoscaler scale-down removes underused nodes without regard to your spread, so replicas can concentrate in whichever zone kept its nodes. After a zone outage, replacement pods schedule into the surviving zones. When the zone comes back, those pods stay where they are, and the recovered zone runs few or none of your replicas until something triggers rescheduling.

Rebalancing options

kubectl rollout restart replaces every pod in the Deployment, and the scheduler applies the constraints to each new pod. This method is simple and uses machinery you already trust. It is coarse, though, because it restarts every replica even when only one zone is skewed.

The descheduler is a separately maintained project that runs periodically, finds pods that violate configured policies such as topology spread, and evicts them so the scheduler can place them again. It does not choose where pods go. It only removes them.

The cost

Both approaches add pod churn: dropped connections, cold caches, and readiness delays. The descheduler uses the Eviction API, which honors PodDisruptionBudgets. A missing or overly permissive PDB lets it evict too many replicas at once, so the attempt to fix the imbalance can cause an outage.

#!/usr/bin/env bash
set -euo pipefail

NS=shop          # placeholder namespace
APP=web          # placeholder app label / Deployment name

kubectl -n "$NS" get pods -l app="$APP" \
  -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' |
while read -r node; do
  kubectl get node "$node" \
    -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}'
done | sort | uniq -c

kubectl -n "$NS" get pdb "$APP-pdb" >/dev/null 2>&1 ||
  kubectl -n "$NS" create pdb "$APP-pdb" \
    --selector=app="$APP" --max-unavailable=1

kubectl -n "$NS" rollout restart deployment/"$APP"
kubectl -n "$NS" rollout status deployment/"$APP" --timeout=10m

A shell script that counts a Deployment's replicas per zone, adds a PodDisruptionBudget if one is missing, and then runs kubectl rollout restart so the scheduler applies the spread constraints to fresh pods.

How do you verify pod spread and debug Pending pods?

Start by checking the actual distribution instead of trusting the manifest. kubectl get pods -o wide -l app=api shows which node each replica landed on, and kubectl get nodes -L topology.kubernetes.io/zone maps those nodes to zones. Joining the two outputs gives per-zone and per-node counts you can compare against your maxSkew. Do this after every rollout and after node pool changes, because the scheduler only enforces skew at placement time.

Reading FailedScheduling events

For a Pending pod, kubectl describe pod lists FailedScheduling events with a per-node tally, such as "3 node(s) didn't match pod topology spread constraints" alongside counts for taints, insufficient CPU, or node affinity. That tally tells you spreading is the blocker. It does not tell you which constraint failed when a pod has several. A hard zone constraint and a hard node constraint are reported under the same reason, so you have to work out the skew for each topologyKey yourself.

Also check that labelSelector matches the pods you expect. A selector that also catches replicas from an old ReplicaSet produces counts you can't reconcile with what you see. Soft constraints (ScheduleAnyway) never cause Pending, so look elsewhere if only those are set.

The rest of the availability picture

Spreading controls where pods land, not how many can be removed at once. A node drain, cluster upgrade, or descheduler run can evict replicas from one zone back to back. Pair spreading with:

  • A PodDisruptionBudget that caps how many replicas voluntary evictions can take down at the same time.
  • Replica counts that divide evenly across zones. Six replicas across three zones keeps balance. Five replicas leaves one zone short by design.
  • Readiness probes, so a rescheduled pod receives traffic only once it can serve it.

Verify these properties in staging before an incident. During an outage, you want to already know your spread holds rather than be decoding scheduler events under pressure.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes topology spread constraints, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.