What are topology spread constraints and why do multi-AZ clusters need them?
Spreading nodes across three availability zones does not mean your pods are spread across them. The scheduler places each pod on whichever feasible node scores best at that moment. Scores depend on free resources, affinity rules, image locality and other factors. The default scheduler does apply some built-in spreading, but only as a soft scoring preference, and other signals can outweigh it. Several common events can leave replicas concentrated in one zone: a node pool that scaled up in one zone first, a rollout during a capacity crunch, or nodes drained for maintenance.
As an illustrative estimate, take a Deployment with six replicas across zones A, B and C. Suppose four replicas land in zone A. If zone A goes down, you lose two-thirds of serving capacity, and the remaining two pods take the full load until replacements schedule. With an even 2/2/2 split, the same outage costs one-third of capacity.
Topology spread constraints let you state how uneven placement may become. You name a topology domain, such as the well-known topology.kubernetes.io/zone node label. You set a maximum allowed difference in matching pod counts between domains. You also choose whether the scheduler must enforce that limit or only prefer it.
They fit backend and platform engineers running stateless or replicated services across zones. They are worth adding when two conditions hold:
- The replica count is at least the number of zones. Below that, some zone always has zero pods.
- The service has a real availability requirement. Losing one zone should not take down most of the service.
Deployment: 6 replicas | nodes in zones A, B, C | outage: zone A
1) Default scheduling only (soft spreading score) -> 4 / 1 / 1
+------------------+ +------------------+ +------------------+
| Zone A (DOWN) | | Zone B | | Zone C |
| [pod] [pod] | | [pod] | | [pod] |
| [pod] [pod] | | | | |
+------------------+ +------------------+ +------------------+
|
v
4 of 6 pods lost -> 2/3 of capacity gone, 2 pods take full load
2) topologySpreadConstraints -> 2 / 2 / 2
topologyKey: topology.kubernetes.io/zone maxSkew: 1
whenUnsatisfiable: DoNotSchedule (must) or ScheduleAnyway (prefer)
+------------------+ +------------------+ +------------------+
| Zone A (DOWN) | | Zone B | | Zone C |
| [pod] [pod] | | [pod] [pod] | | [pod] [pod] |
+------------------+ +------------------+ +------------------+
|
v
2 of 6 pods lost -> 1/3 of capacity gone, 4 pods take full load
Illustrative estimate: with six replicas, a 4/1/1 placement loses two-thirds of capacity when zone A fails, while a 2/2/2 placement enforced by a zone topology spread constraint loses one-third.
How do topology spread constraints work in the scheduler?
The four fields
topologyKeyis the node label that defines a domain.topology.kubernetes.io/zonegroups nodes by zone;kubernetes.io/hostnamemakes every node its own domain.labelSelectorpicks which pods are counted. It must match the Deployment's own pod labels, or the scheduler balances the wrong set.maxSkewis the largest allowed difference between the most and least populated eligible domains.whenUnsatisfiabledecides what happens when placing a pod would exceedmaxSkew.
Only domains with eligible nodes count. Nodes filtered out by the pod's nodeSelector or required node affinity are excluded by default. Tainted nodes are still counted by default unless nodeTaintsPolicy: Honor is set. A zone with no eligible nodes is not a domain at all. A zone whose nodes exist but are full is still a domain, and its pod count of zero sets the minimum.
Illustrative estimate
Assume 6 replicas, 3 zones, and maxSkew: 1. With enough capacity everywhere, the result is 2/2/2. Now assume zone C's nodes are full but still eligible:
whenUnsatisfiable |
Placement (A/B/C) | Skew | Pending |
|---|---|---|---|
DoNotSchedule |
1/1/0 | 1 | 4 |
ScheduleAnyway |
3/3/0 | 3 | 0 |
Under DoNotSchedule, a second pod in zone A would make A=2 against C=0. That gives a skew of 2, so the scheduler rejects it. The constraint is a hard filter.
Under ScheduleAnyway, skew only adds a scoring penalty. The scheduler picks the least-skewed node that fits and accepts that balance gets worse.
How do you build a zone-balanced Deployment step by step?
Start with the Deployment's pod template and add a topologySpreadConstraints list. The first entry targets zones: topologyKey: topology.kubernetes.io/zone, maxSkew: 1, whenUnsatisfiable: DoNotSchedule. This is the guarantee you care about. Losing a zone should never take out more than its share of replicas.
The second entry targets nodes: topologyKey: kubernetes.io/hostname, maxSkew: 1, whenUnsatisfiable: ScheduleAnyway. Node spread is usually soft because node counts drift constantly. Autoscalers add and remove nodes, and some nodes are cordoned or full. A hard hostname constraint leaves pods Pending whenever one node can't take another replica, even if the zone has room elsewhere.
Per-zone capacity decides whether the zone constraint can be hard. If every zone reliably has headroom for its share, make it hard. If one zone is often short of capacity, a hard constraint stalls rollouts there. In that case, soften it or fix the capacity first.
The labelSelector defines which pods are counted. Match labels unique to this workload, such as app: checkout plus a component label. If the selector is too broad, it counts other apps' pods and skews placement for reasons unrelated to your service. If it is too narrow, it skips some of your own replicas, and the spread you see differs from the spread the scheduler enforces.
Rollouts add a further problem. During a rolling update, old and new ReplicaSet pods share the selector. The scheduler balances the combined set, so new pods can end up clustered while old pods drain. Where your cluster version supports it, matchLabelKeys: [pod-template-hash] scopes counting to the incoming pod's revision. Check your version's documentation before relying on it.
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
spec:
replicas: 6
selector:
matchLabels:
app: checkout
component: api
template:
metadata:
labels:
app: checkout
component: api # with app, unique to this workload
spec:
topologySpreadConstraints:
# Hard: a zone outage must not take more than its share of replicas.
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels: {app: checkout, component: api}
# Version-dependent: count only pods from the incoming revision.
matchLabelKeys: [pod-template-hash]
# Soft: node counts drift, so prefer spread but never block.
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels: {app: checkout, component: api}
matchLabelKeys: [pod-template-hash]
containers:
- name: api
image: registry.example.com/checkout-api:1.4.2
resources:
requests: {cpu: 250m, memory: 256Mi}
A Deployment with a hard zone-level spread constraint and a soft hostname-level constraint, both counting only this workload's pods, with matchLabelKeys scoping the count to the current rollout revision on cluster versions that support it.
How can you verify pod placement and debug Pending pods?
Run the same three checks in the same order each time. Each one rules out a different class of problem.
1. Confirm the zone labels
Start with kubectl get nodes -L topology.kubernetes.io/zone. This adds a column showing each node's zone. A blank value means the node has no zone label. The scheduler can't count that node under a zone-keyed constraint, and it is treated as ineligible for placement. Typos in a custom topologyKey fail silently in the same way, so check the labels before you check anything else.
2. Map pods to nodes
Next, run kubectl get pods -l app=<your-app> -o wide. The NODE column, read against the zone column from step 1, gives you the actual per-zone counts. Use the same label selector your constraint uses. If they differ, the scheduler is counting a different set of pods than the one you are looking at.
3. Read the scheduling events
For a Pending pod, run kubectl describe pod <name> and find the FailedScheduling event. The message tallies the reason each node was rejected, in a form like "0/6 nodes are available: 3 node(s) didn't match pod topology spread constraints, 3 Insufficient cpu."
Each phrase points to a different fix:
- "didn't match pod topology spread constraints": placing the pod on that node would push skew past
maxSkew. Either the other zones can't take the pod, or they hold too few eligible nodes. - "Insufficient cpu" or "Insufficient memory": this is a capacity problem. Loosening the spread won't help.
- "had untolerated taint": the pod needs a toleration, or the node needs a different taint.
Mixed messages are common. A zone whose nodes are all full or tainted can't absorb a pod, and that is the condition that creates the skew violation in the first place. Fix the resource or taint cause first, then re-check the spread.
#!/usr/bin/env bash
# Repeatable placement check for a workload using zone-keyed topology spread.
# Run the steps in order: labels, then pod-to-node mapping, then events.
set -euo pipefail
NS="shop" # placeholder namespace
SELECTOR="app=web" # must match the constraint's labelSelector
# 1. Confirm every node carries a zone label.
# A blank ZONE column means the node has no topology.kubernetes.io/zone label.
kubectl get nodes -L topology.kubernetes.io/zone
# 2. Map the selected pods to nodes. Read NODE against the ZONE column
# from step 1 to get per-zone counts. Pending pods show <none> as NODE.
kubectl get pods -n "$NS" -l "$SELECTOR" -o wide
# 3. Inspect each Pending pod's FailedScheduling events.
# The message tallies node rejections by reason, for example:
# "0/6 nodes are available: 3 node(s) didn't match pod topology spread
# constraints, 3 Insufficient cpu."
for pod in $(kubectl get pods -n "$NS" -l "$SELECTOR" \
--field-selector=status.phase=Pending -o name); do
echo "=== $pod"
# Print only the Events section of describe output.
kubectl describe -n "$NS" "$pod" | sed -n '/^Events:/,$p'
done
# Namespace-wide view of recent scheduling failures, oldest first.
kubectl get events -n "$NS" \
--field-selector reason=FailedScheduling \
--sort-by=.lastTimestamp
# Reading the reasons:
# "didn't match pod topology spread constraints" -> skew would exceed maxSkew
# "Insufficient cpu" / "Insufficient memory" -> capacity; fix this first
# "had untolerated taint" -> add a toleration or change the taint
A shell script that runs the three checks in order: confirm node zone labels, map the selected pods to nodes, and print FailedScheduling events for Pending pods so you can see which rejection reason applies.
Where do the guarantees stop: rebalancing, rollouts and autoscaling
Topology spread constraints are evaluated by the scheduler only when it places a pod. Once a pod is bound to a node, nothing checks the constraint again. If a zone loses its nodes, the replacement pods land in the surviving zones, which is the behavior you want during an outage. When the zone recovers, those pods stay where they are. Scale-down has a similar effect: the ReplicaSet controller chooses which pods to delete without considering your spread rules, so it can leave the remaining pods skewed. The imbalance persists until pods are recreated by a rollout, a restart, or an external process such as the Kubernetes descheduler, which can evict pods that violate a spread constraint so the scheduler can place them again.
Rolling updates can briefly distort spread. Old and new pods usually share the labels your labelSelector matches, so terminating old pods still count toward skew while new pods are being placed. The new revision can therefore end up unevenly distributed even though the combined set looked balanced. Using matchLabelKeys with the pod template hash, where your cluster supports it, scopes the calculation to a single revision.
Autoscalers create a different gap. A cluster autoscaler adds nodes to a node group, not to a zone. If one node group spans several zones, the new node may come up in a zone that does not resolve the skew, and the pod stays Pending. Use one node group per zone so the autoscaler can add capacity in the zone the constraint requires.
Low replica counts also limit what spreading can do. Two replicas across three zones always leave one zone empty. Three replicas give you one per zone, so losing a zone removes a third of your capacity at once.
When should you avoid hard spread constraints, and what are the alternatives?
Hard constraints (whenUnsatisfiable: DoNotSchedule) work against you in three common situations. On tightly packed clusters, a pod that would fit on a node in an already-busy zone stays Pending rather than raise the skew. On single-zone dev clusters, a spread that assumes several zones either has no effect or fails in confusing ways, depending on how nodes are labelled. During a zone outage, the failed zone's nodes can still count as a domain until they are removed. Replacement pods then cannot land in the surviving zones without breaking maxSkew, so you lose capacity at the moment you most need it. ScheduleAnyway keeps the preference for balance but lets the scheduler place the pod somewhere when balance is not possible.
| Mechanism | What it expresses | Fits when |
|---|---|---|
Spread, DoNotSchedule |
Bounded imbalance, enforced | Ample per-zone capacity, and the replica count is a multiple of the zone count |
Spread, ScheduleAnyway |
Bounded imbalance, preferred | Most stateless services |
| Required anti-affinity | At most one matching pod per domain | Quorum members, one per zone or per node |
| Preferred anti-affinity | Weighted avoidance of co-location | Simple "try not to stack" rules |
| Cluster default constraints | Spread rules applied to every workload without one, set in the scheduler configuration | Platform-wide baseline |
| Node selector / node affinity | Which nodes a pod may use | Pinning to hardware or a zone, not balancing |
Even spread is only one part of high availability. You also need enough spare capacity in each zone to absorb a failed zone's load, PodDisruptionBudgets to stop voluntary evictions such as node drains from emptying a zone, zone-aware load balancing, and stateful dependencies like databases and caches that survive the same failure.
How do topology spread constraints affect engineering cost and time?
The main cost is capacity, not the YAML. Even spread only protects you if the surviving zones can absorb the lost zone's pods. Otherwise the scheduler correctly reports "no room," and the rescheduled pods sit Pending during the outage.
Illustrative estimate (assumed numbers, not a benchmark)
Assume a service needs 30 pods at peak and runs across Z = 3 zones.
- Per-zone pods to survive one zone loss = peak ÷ (Z − 1) = 30 ÷ 2 = 15
- Total provisioned = 15 × 3 = 45 pods, against 30 without zone-loss headroom
- Overhead = 1 ÷ (Z − 1) = 50%. With four zones it drops to about 33%.
Plug in your own peak and zone count. More zones cost less headroom each but add cross-zone traffic and operational surface.
Other costs:
- Autoscaling setup. A node group that spans zones lets the autoscaler add a node in whichever zone it picks, which may not be the zone the Pending pod needs. You usually need one node group per zone.
- Debugging time. Engineers must learn to read scheduler events and
maxSkewinteractions with taints and affinity.
The savings offset this. Declarative placement replaces hand-pinning pods to nodes. Cluster-level default constraints remove the boilerplate each team would otherwise repeat in its own manifests. Selectors that are validated before an incident mean an outage is not where you find out that half your replicas shared a zone.
Adoption checklist
- Confirm nodes carry consistent zone labels
- Size per-zone capacity using the formula above
- Create one autoscaling node group per zone
- Set cluster defaults, then override them per workload
- Verify label selectors match pod labels
- Test with a zone drain before relying on it
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to topology spread constraints, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.