What topology spread constraints are and how the scheduler evaluates them

Topology spread constraints tell the scheduler to keep matching pods evenly distributed across failure domains. For stateless API Deployments running in several zones, they are the right default for zone spreading. Backend and platform engineers use them for one reason: if a zone fails, enough replicas should survive in the other zones to keep serving traffic. They have one key limitation. They are enforced only at scheduling time. The scheduler checks them when it places a pod and never revisits the decision afterward.

How the scheduler evaluates a constraint

  1. Build domains. The topologyKey names a node label, usually topology.kubernetes.io/zone. Every distinct value of that label is one domain. Nodes without the label are ignored. Nodes the pod cannot run on, because of its node selector or affinity, do not count as eligible.
  2. Count matching pods. In each eligible domain, the scheduler counts the pods selected by labelSelector. Normally this is the Deployment's own pod labels.
  3. Compute skew. For each candidate placement, skew is the count in the most populated domain minus the count in the least populated domain, assuming the new pod lands in the candidate's domain.
  4. Apply whenUnsatisfiable. If skew would exceed maxSkew, DoNotSchedule filters that node out. With no valid node left, the pod stays Pending. ScheduleAnyway lets the placement through but scores it lower.

Example: maxSkew: 1, five replicas running, scheduling the sixth

Zone Current pods Pods if sixth lands here Resulting skew Allowed?
zone-a 2 3 3 − 1 = 2 No
zone-b 2 3 3 − 1 = 2 No
zone-c 1 2 2 − 2 = 0 Yes

Only zone-c keeps skew within 1, so the sixth pod goes there and every zone ends up with two replicas.

New pod #6 (app=api), maxSkew: 1, whenUnsatisfiable: DoNotSchedule
                               |
                               v
+--------------------------------------------------------------+
| kube-scheduler (runs once, at scheduling time only)          |
| domains = values of node label topology.kubernetes.io/zone   |
| count pods matching labelSelector app=api in each domain     |
+--------------------------------------------------------------+
        |                      |                      |
        v                      v                      v
+----------------+     +----------------+     +----------------+
| zone-a         |     | zone-b         |     | zone-c         |
| [api] [api]    |     | [api] [api]    |     | [api]          |
| current: 2     |     | current: 2     |     | current: 1     |
+----------------+     +----------------+     +----------------+
        |                      |                      |
        v                      v                      v
+----------------+     +----------------+     +----------------+
| a/b/c: 3/2/1   |     | a/b/c: 2/3/1   |     | a/b/c: 2/2/2   |
| skew 3-1 = 2   |     | skew 3-1 = 2   |     | skew 2-2 = 0   |
| 2 > 1: REJECT  |     | 2 > 1: REJECT  |     | 0 <= 1: ALLOW  |
+----------------+     +----------------+     +----------------+
                                                      |
                                                      v
                       +---------------------------------------+
                       | Pod 6 bound in zone-c: now 2 / 2 / 2  |
                       | skew 0; not rebalanced later          |
                       +---------------------------------------+

With maxSkew: 1 and current counts of 2/2/1 across three zones, placing the sixth pod in zone-a or zone-b would raise skew to 2, so the scheduler filters those nodes and binds the pod in zone-c, and it does not revisit that placement afterward.

Implementing zone spreading for an API Deployment

The Deployment below puts two entries under topologySpreadConstraints in the pod template's spec. The first is the hard rule. It sets topologyKey: topology.kubernetes.io/zone, maxSkew: 1 and whenUnsatisfiable: DoNotSchedule. Under this rule the scheduler leaves a pod Pending rather than place it where it would push one zone more than one replica ahead of another. That is the guarantee you want for an API: losing one zone should never take out a disproportionate share of capacity.

The second entry uses topologyKey: kubernetes.io/hostname with whenUnsatisfiable: ScheduleAnyway. This makes node-level spreading a scoring preference rather than a filter. Inside each zone, the scheduler favours nodes that hold fewer replicas, which reduces the impact of a single node failure. If a zone has only one suitable node, the pod still schedules, because the zone rule is the one that matters.

Both constraints carry a labelSelector, and it decides which pods get counted. It must match the labels in the Deployment's own pod template, such as app: orders-api together with a component or tier label. If you use a broad selector such as app: orders, and a worker Deployment shares that label, the scheduler counts the worker pods as well. Your API replicas can then cluster in one zone while the skew calculation reports balance, and nothing in the events tells you the count is wrong.

Also note that setting topologySpreadConstraints on a pod replaces any cluster-level default spread constraints for that pod. Your explicit list becomes the only spreading rule the scheduler applies to it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 6
  selector:
    matchLabels:
      app: orders-api
      tier: api
  template:
    metadata:
      labels:
        app: orders-api   # specific to this Deployment, not shared with workers
        tier: api
    spec:
      # Explicit constraints replace cluster-level defaults for these pods.
      topologySpreadConstraints:
        # Hard rule: zones may differ by at most one replica; otherwise stay Pending.
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: orders-api   # must match the template labels above
              tier: api
        # Soft rule: within a zone, prefer nodes with fewer replicas.
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: orders-api
              tier: api
      containers:
        - name: api
          image: registry.example.com/orders-api:1.0.0
          ports:
            - containerPort: 8080

A Deployment with a hard zone spread constraint and a soft per-node constraint, both using a labelSelector that matches only this Deployment's pod labels.

Alternatives: pod anti-affinity versus spread constraints

Before spread constraints existed, the usual way to keep replicas apart was pod anti-affinity against the Deployment's own labels. It solves a different problem, and the difference matters once you scale.

Required anti-affinity with topologyKey: topology.kubernetes.io/zone means "never place this pod in a zone that already runs a matching pod." In practice that allows one replica per zone. With three zones, the fourth replica stays Pending, and so does the extra pod a rolling update creates during surge. Preferred anti-affinity avoids the hard cap, but it only adds a weight to the scheduler's node scoring. That weight competes with resource-balance and other scores, so it does not guarantee an even result.

Spread constraints express "roughly even" directly through maxSkew. They work the same way at 3 replicas or 30.

Mechanism Rule it expresses Replicas above zone count Under zone loss or capacity shortage
Required anti-affinity (zone) At most one matching pod per zone Pending Pods stay Pending
Preferred anti-affinity (zone) Weighted hint to avoid co-location Scheduled, balance not guaranteed Scheduled anywhere
Spread, DoNotSchedule Skew must stay ≤ maxSkew Scheduled evenly Pending if skew would be violated
Spread, ScheduleAnyway Prefer nodes that reduce skew Scheduled, mostly even Scheduled, skew may grow

Anti-affinity is still the right tool for a strict one-per-node requirement, such as a workload binding a host port or needing a node to itself. For that case, use topologyKey: kubernetes.io/hostname.

Even spreading also does not make traffic zone-local. A Service still sends requests across zones unless you configure topology-aware routing, and that is a separate decision.

How spreading interacts with rollouts, node selection and autoscaling

Rolling updates

During a rollout, pods from the old and new ReplicaSets carry the same app labels, so both match the constraint's labelSelector. The scheduler counts them together. New pods land wherever the combined count is lowest, and when the old pods terminate, the new ReplicaSet can be lopsided even though skew looked fine at every step. Adding matchLabelKeys: ["pod-template-hash"] fixes this. The scheduler then adds each pod's own template hash to the selector, so it computes skew only against pods from the same ReplicaSet.

Which nodes count

A zone counts as a domain only if it contains eligible nodes. By default, nodeAffinityPolicy is Honor, so nodes that fail the pod's nodeSelector or node affinity are excluded. nodeTaintsPolicy defaults to Ignore, so tainted nodes the pod cannot tolerate still count. That default can make a zone look available when the pod can never run there. Set nodeTaintsPolicy: Honor if you taint nodes by zone or workload. minDomains, which applies only with whenUnsatisfiable: DoNotSchedule, tells the scheduler how many zones you expect. That prevents it from treating two visible zones as a balanced spread when you planned for three. Support for these fields depends on your cluster version, so check its documentation before you rely on them.

Autoscaling

A hard constraint can require a node in one specific zone. The cluster autoscaler can add that node only if a node group can provision in that zone. If no node group covers the zone, or the cloud provider has no capacity there, the pod stays Pending. Watch for this during scale-up.

In very large clusters, measure how spread constraints affect scheduling latency instead of assuming the overhead is negligible or prohibitive.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 6
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          # Hard constraint: the pod stays Pending rather than skewing zones.
          # If no node group can provision in the required zone, the
          # cluster autoscaler cannot fix this, so watch Pending pods on scale-up.
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: api
          # Count only pods from the same ReplicaSet so old pods
          # don't distort skew during a rolling update.
          matchLabelKeys:
            - pod-template-hash
          # We plan for three zones; don't treat two visible zones as balanced.
          # Only valid with DoNotSchedule.
          minDomains: 3
          # Ignore nodes that fail this pod's nodeSelector/affinity (the default).
          nodeAffinityPolicy: Honor
          # Ignore tainted nodes this pod can't tolerate (default is Ignore).
          nodeTaintsPolicy: Honor
      containers:
        - name: api
          image: registry.example.com/api:1.0.0

A Deployment whose zone spread constraint uses matchLabelKeys, minDomains, nodeAffinityPolicy and nodeTaintsPolicy to keep skew accurate during rollouts and count only nodes the pod can run on. Check that your cluster version supports each field.

Sizing replicas and pairing with a PodDisruptionBudget

Choose a replica count that is a multiple of the zone count. With maxSkew: 1, a multiple divides evenly, so every zone holds the same number of pods and no single zone loss costs more than its share. With any other count, some zone holds an extra pod, and losing that zone takes out more capacity than the others would.

Illustrative estimate (assumed numbers, not a benchmark). Assume 3 zones, 6 replicas, maxSkew: 1 and a peak load of 1,200 requests per second.

  • Each zone holds 2 pods. Losing one zone leaves 4 of 6 replicas.
  • Surviving fraction = 1 − 1/Z = 1 − 1/3 ≈ 0.67.
  • Required per-pod capacity = peak load / (R × (1 − 1/Z)) = 1,200 / (6 × 0.67) = 300 rps.
  • In normal operation each pod serves 200 rps. Under these assumed numbers, size each pod for about 300 rps and keep normal utilization near two-thirds.

Compare this with 2 replicas across 3 zones. One zone always has no pod. If a zone that does hold a pod fails, you lose half your capacity, and the remaining pod must absorb the full load alone.

Spread constraints only control where pods land. To limit how many can be evicted at once during node drains or cluster upgrades, add a PodDisruptionBudget with the same selector as the Deployment and maxUnavailable: 1. Prefer maxUnavailable over minAvailable, because it stays correct as the autoscaler changes the replica count. The budget counts pods and has no concept of zones. It applies only to evictions that go through the eviction API, so it offers no protection against an actual zone outage. Covering that case is the job of the spread and the capacity headroom.

# Deployment: 6 replicas across 3 zones -> 2 pods per zone with maxSkew 1.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 6            # multiple of the zone count (3)
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: api
      containers:
        - name: api
          image: registry.example.com/api:1.0.0
---
# PDB: limits voluntary evictions (drains, upgrades) to one pod at a time.
# It counts pods only, not zones, and does not help in a real zone outage.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  maxUnavailable: 1      # stays correct as the replica count changes
  selector:
    matchLabels:
      app: api           # same selector as the Deployment

A Deployment with 6 replicas spread evenly across 3 zones using maxSkew 1, paired with a PodDisruptionBudget that allows at most one pod to be evicted at a time during voluntary disruptions.

No automatic rebalancing: restoring balance and verifying placement

Topology spread constraints are evaluated only when a pod is scheduled. Once a pod is bound to a node, nothing re-checks the constraint, and nothing moves the pod.

How replicas become skewed

Zone outage. If one zone fails, its pods are rescheduled into the surviving zones. When the zone recovers, those pods stay where they are, and the recovered zone remains empty until new pods are created.

Scale-down. The ReplicaSet controller decides which pods to delete without consulting your spread constraints. Removing several replicas can leave one zone carrying most of the remaining load.

Restoring balance

Rollout restart. The simplest fix is kubectl rollout restart deployment/<name>. This replaces every pod, and each replacement goes through the scheduler again. While the rollout is running, the old pods still count toward skew, so the final placement may not be perfectly even. It is usually much closer than before.

Descheduler. The descheduler is an add-on that evicts pods violating spread constraints so the scheduler can place them again. It is another component to run, configure, and monitor, and it removes running pods by design. Its evictions go through the eviction API, so a PodDisruptionBudget limits how fast it works.

Verifying placement

Run kubectl get pods -o wide -l app=<label> to see which node each pod is on. Then run kubectl get nodes -L topology.kubernetes.io/zone to map each node to its zone. Count the pods per zone and compare the difference against your maxSkew.

Diagnosing Pending pods

If pods stay Pending under whenUnsatisfiable: DoNotSchedule, run kubectl describe pod on one of them. The FailedScheduling event reports how many nodes failed each filter, including how many did not match the pod topology spread constraints. That count tells you whether the cause is the constraint itself, a node selector, or a lack of capacity in the zone the scheduler needs to fill.

When to use it, when to avoid it, and the impact on cost and engineering time

Use a hard zone constraint (whenUnsatisfiable: DoNotSchedule) when the workload is a stateless API Deployment, it runs enough replicas to cover every zone, and each zone has its own capacity. Use ScheduleAnyway, or skip the constraint, when spare capacity is tight or replica counts are very low. A hard rule can leave pods Pending during the zone outage it was meant to survive. When a zone disappears, the surviving zones may not satisfy maxSkew, so the replacement pods wait instead of running.

Situation Recommended setting Why
Stateless API, replicas ≥ zones, per-zone node groups DoNotSchedule Even spread holds, and the autoscaler can add a node in the zone that needs it
Tight capacity or shared node pools ScheduleAnyway A running pod in the wrong zone beats a Pending pod
One or two replicas ScheduleAnyway or none There is too little to spread, and a hard rule only adds Pending risk
Per-zone node groups too costly ScheduleAnyway A single multi-zone group cannot reliably add a node in a specific zone

Cost. Spreading can force extra nodes per zone and adds a fixed per-zone node group minimum. It also creates cross-zone traffic, which adds latency and may incur transfer charges.

Illustrative estimate (assumed numbers, not a benchmark): replicas needed to keep capacity C after losing one of Z zones is R = ⌈C × Z / (Z − 1)⌉. With C = 6 and Z = 3, R = 9, so you pay for 3 more pods than C. If traffic routes randomly, roughly (Z − 1) / Z of requests cross zones, about two-thirds here.

Engineering time. One declarative rule replaces hand-tuned anti-affinity. Outage behavior becomes predictable, and placement failures can be diagnosed from scheduler events rather than guesswork.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes topology spread constraints, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.