Why does Kubernetes create cross-AZ data transfer cost by default?
Cross-AZ data transfer cost is the per-gigabyte charge a cloud provider bills when traffic moves between availability zones in the same region. Traffic that stays inside one zone is usually free or much cheaper, so the zone a packet lands in determines what it costs.
A default ClusterIP Service has no zone awareness. kube-proxy, or whatever dataplane implements Services, picks a backend from all ready endpoints. It does not check whether the caller and the backend share a zone. If pods are spread evenly across N zones, a call lands in another zone with probability (N-1)/N. In a three-zone cluster, that is about two-thirds of calls.
Each hop in a call chain makes this choice independently, so the cost compounds:
zone-a zone-b zone-c
ingress ──────► api (cross)
│
▼
orders (same)
│
└──────────────► inventory (cross)
For a chain of k hops, about k·(N-1)/N hops cross a zone boundary. The chance that a request stays entirely in one zone is (1/N)^k, which is roughly 4% for three hops across three zones.
Illustrative estimate (assumed numbers, not a benchmark).
Formula: daily cost = GB per hop × hops × (N-1)/N × price per GB × directions billed.
Assume:
- 2,000 GB/day per hop
- 3 hops
- N = 3
- $0.01/GB
- charged in both directions
Then 2,000 × 3 × 2/3 × 0.01 × 2 = $80/day, or about $29,000/year, for traffic that never leaves the region.
┌─ zone-a ─────────────┐ ┌─ zone-b ─────────────┐ ┌─ zone-c ─────────────┐
│ │ │ │ │ │
│ ┌────────────────┐ │ │ │ │ │
│ │ ingress │──┼─┼──┐ hop 1: cross-AZ │ │ │
│ └────────────────┘ │ │ ▼ │ │ │
│ │ │ ┌────────────────┐ │ │ │
│ │ │ │ api │ │ │ │
│ │ │ └───────┬────────┘ │ │ │
│ │ │ │ hop 2: │ │ │
│ │ │ │ same zone │ │ │
│ │ │ ▼ │ │ │
│ │ │ ┌────────────────┐ │ │ │
│ │ │ │ orders │──┼─┼──┐ hop 3: cross-AZ │
│ │ │ └────────────────┘ │ │ ▼ │
│ │ │ │ │ ┌────────────────┐ │
│ │ │ │ │ │ inventory │ │
│ │ │ │ │ └────────────────┘ │
└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
hops crossing a zone here: 2 of 3 (expected: k·(N-1)/N = 3 × 2/3 = 2)
P(entire request stays in one zone) = (1/N)^k = (1/3)^3 ≈ 4%
With a default ClusterIP Service, each hop independently picks a backend in any zone, so in this three-zone, three-hop chain two of the three hops cross a zone boundary and are billed as cross-AZ traffic.
How much does inter-zone traffic actually cost?
Most cloud providers bill traffic between availability zones in the same region per GB. Some charge only the sender. Others charge both the sending and the receiving side, which doubles the effective rate. Before you size the problem, check your provider's pricing page and confirm whether inter-zone transfer is billed in one direction or both. Getting this wrong halves or doubles your estimate.
For a whole cluster, the monthly estimate is:
monthly cost = east-west GB × (N−1)/N × rate × directions billed
The (N−1)/N term comes from how Kubernetes routes by default. If pods are spread evenly across N zones and a Service picks endpoints without regard to zone, a request lands outside the caller's zone (N−1)/N of the time. With three zones, that is two out of every three requests.
Illustrative estimate (assumed numbers, not a benchmark)
| Input / step | Assumed value | Result |
|---|---|---|
| East-west traffic per month | 200 TB | 200,000 GB |
| Zones (N) | 3 | (N−1)/N = 2/3 |
| Traffic crossing zones | 200,000 × 2/3 | ≈ 133,333 GB (~133 TB) |
| Rate per GB | $0.01 | ≈ $1,333 |
| Directions billed | 2 | ≈ $2,667, roughly $2,700/month |
Substitute your own measured east-west volume, zone count and provider rate. The east-west volume should exclude internet egress and traffic that already stays local, such as sidecars or node-local caches.
Whether this matters depends on its share of your bill. If inter-zone transfer is a material line item next to compute, the rest of this article is worth acting on. If it is a rounding error, the operational risk of changing routing likely outweighs the savings.
How do you find which workloads generate cross-zone traffic?
Start with billing, not with the cluster. Your provider's cost allocation or billing reports break data transfer out by usage type, and inter-zone traffic usually appears as its own line. Filter it to the accounts or tags that host your clusters and check the trend over the last few months. That number is your ceiling: no routing change can save more than this.
Next, attribute the traffic to workloads. Billing tells you how much but not who. VPC flow logs record source and destination IPs and byte counts. Network observability tooling inside the cluster can go further and resolve those IPs to pods and Services. To classify a flow as cross-zone, you need each endpoint's zone, which comes from the topology.kubernetes.io/zone label on the node running the pod. Running kubectl get nodes -o wide -L topology.kubernetes.io/zone lists each node's internal IP next to its zone. That is enough to join against flow records when your tooling doesn't enrich them for you.
Aggregate the cross-zone bytes by client workload and destination Service, then sort. Inter-zone traffic is usually concentrated in a short list of chatty pairs, such as an API tier calling a cache, a service reading from a message broker, or replication between stateful pods. That ranked list becomes your rollout order. Enable zone-local routing on the top pairs first, then re-measure against the billing line to confirm the savings are real.
This step keeps the work proportionate. Without the list, teams tend to change routing on every Service and absorb the risk everywhere. With it, you change only the few pairs that account for most of the cost.
#!/usr/bin/env bash
# Build IP-to-zone lookup tables to join against flow log records.
set -euo pipefail
# Human-readable view: each node's internal IP next to its zone label.
kubectl get nodes -o wide -L topology.kubernetes.io/zone
# Node internal IP -> zone (tab-separated), for flows that use node IPs.
kubectl get nodes -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\t"}{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}{end}' \
> node-ip-zone.tsv
# Node name -> zone, used to place each pod in its node's zone.
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}{end}' \
> node-name-zone.tsv
# Pod IP, namespace/name and node for every scheduled pod.
kubectl get pods -A --field-selector=status.phase=Running \
-o jsonpath='{range .items[*]}{.status.podIP}{"\t"}{.metadata.namespace}/{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}' \
> pod-ip-node.tsv
# Join pods to zones: output is pod IP, pod, zone.
# hostNetwork pods report the node IP, so node-ip-zone.tsv covers them too.
awk -F'\t' 'NR==FNR { zone[$1]=$2; next } { print $1 "\t" $2 "\t" zone[$3] }' \
node-name-zone.tsv pod-ip-node.tsv > pod-ip-zone.tsv
# Pod IPs are reused over time, so take this snapshot close to the flow
# log window you are analysing, or capture it on a schedule.
head pod-ip-zone.tsv
This script lists nodes with their topology.kubernetes.io/zone label and builds IP-to-zone lookup files for nodes and pods, so you can tag each flow log record's source and destination as same-zone or cross-zone.
How does topology aware routing work in Kubernetes?
Kubernetes has three built-in ways to keep Service traffic close to the client. They differ in how strict they are and in what happens when local capacity runs out.
Topology Aware Routing is enabled by setting the service.kubernetes.io/topology-mode: Auto annotation on the Service. The EndpointSlice controller then writes a zone hint onto each endpoint, and kube-proxy routes only to endpoints hinted for the client's zone. To prevent a small zone from being overloaded, the controller assigns hints in proportion to each zone's allocatable CPU. It only does this when every zone has enough endpoints to take its share. If that condition fails, the controller removes all hints and routing reverts to cluster-wide. This can happen after a scale-down or a node failure. Nothing errors when hints are dropped, so the only sign is that your cross-zone bytes go back up.
trafficDistribution: PreferClose is a field in the Service spec rather than an annotation. It expresses a same-zone preference without the proportional-capacity calculation, and it falls back to other zones only when no local endpoint exists. It is newer than the annotation, so confirm that your cluster version and your data plane honor it. Some kube-proxy replacements implement it differently or not at all.
internalTrafficPolicy: Local restricts traffic to endpoints on the same node. It suits DaemonSet agents such as log shippers and metrics collectors. If the node has no local endpoint, traffic is dropped rather than rerouted.
| Option | Fallback behavior | Capacity requirement | Intended use |
|---|---|---|---|
topology-mode: Auto |
Hints dropped, so routing becomes cluster-wide | Endpoints proportional to zone CPU | Large, evenly spread Deployments |
trafficDistribution: PreferClose |
Other zones used only when the local zone has no endpoints | None calculated; you must size each zone yourself | Simple zone preference |
internalTrafficPolicy: Local |
Traffic dropped | An endpoint on every client node | Node-local agents |
# Option 1: Topology Aware Routing (annotation).
# Zone hints are added only when every zone has enough endpoints
# for its share of allocatable CPU; otherwise routing is cluster-wide.
apiVersion: v1
kind: Service
metadata:
name: checkout-api
annotations:
service.kubernetes.io/topology-mode: Auto
spec:
selector:
app: checkout-api
ports:
- port: 80
targetPort: 8080
---
# Option 2: declarative same-zone preference, no capacity math.
# Falls back to other zones only when no local endpoint exists.
# Confirm your cluster version and data plane honor this field.
apiVersion: v1
kind: Service
metadata:
name: inventory-api
spec:
selector:
app: inventory-api
trafficDistribution: PreferClose
ports:
- port: 80
targetPort: 8080
---
# Option 3: node-local only, meant for DaemonSet agents.
# Traffic is dropped if this node has no ready endpoint.
apiVersion: v1
kind: Service
metadata:
name: log-shipper
spec:
selector:
app: log-shipper
internalTrafficPolicy: Local
ports:
- port: 24224
targetPort: 24224
Three Service manifests that show the topology-mode annotation, trafficDistribution: PreferClose, and internalTrafficPolicy: Local, each with comments that describe its fallback behavior.
Implementing zone-local routing safely: capacity prerequisites and rollout
Zone-local routing only works if every zone has enough capacity for the traffic that originates there. Fix capacity first, then change routing.
1. Make capacity symmetric
- Spread pods evenly. Add a
topologySpreadConstraintsentry to each Deployment, keyed ontopology.kubernetes.io/zone, withmaxSkew: 1andwhenUnsatisfiable: DoNotSchedule. Pods then stay within one replica of even across zones. - Set node pools to span every zone. Each pool needs enough headroom in every zone. If one zone runs out of nodes, its pods stay Pending, and traffic routed to that zone falls on fewer replicas.
- Account for how the HPA scales. The HPA scales on average utilization across all pods. It does not know which zone is busy. A hot zone can saturate while the average still looks healthy. Set
minReplicasto a multiple of the zone count, and use a lower utilization target so each zone has slack. - Watch scale-down. Spread constraints apply only when pods are scheduled. Scale-down can leave zones unbalanced. A descheduler, or periodic checks, corrects this drift.
2. Roll out one Service at a time
Use your traffic data to find the highest-volume client/server pairs. Enable topology-aware routing on that Service only.
3. Verify
- Check for zone hints. Run
kubectl get endpointslices -l kubernetes.io/service-name=<svc> -o yamland confirm each endpoint hashints.forZones. - Check for hint removal. Run
kubectl get events --field-selector involvedObject.name=<svc>. If the controller withdrew hints, an event here explains why. - Compare before and after. Look at per-zone request rates, per-zone p99 latency, and inter-zone transfer volume. Move on to the next Service only once these numbers are stable.
# Deployment: keep replicas within one of even across zones.
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout
spec:
replicas: 6 # multiple of the zone count (3 zones here)
selector:
matchLabels:
app: checkout
template:
metadata:
labels:
app: checkout
spec:
topologySpreadConstraints:
- maxSkew: 1 # at most 1 replica difference between zones
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule # stay Pending rather than skew
labelSelector:
matchLabels:
app: checkout
containers:
- name: checkout
image: registry.example.com/checkout:1.0.0
resources:
requests:
cpu: 250m
memory: 256Mi
# Enable topology-aware routing on this one Service only:
# kubectl annotate service checkout service.kubernetes.io/topology-mode=Auto
# Confirm each endpoint carries hints.forZones:
# kubectl get endpointslices -l kubernetes.io/service-name=checkout -o yaml
# Check whether the controller withdrew hints, and why:
# kubectl get events --field-selector involvedObject.name=checkout
# Check the actual zone spread of running pods:
# kubectl get pods -l app=checkout -o wide
A Deployment that spreads replicas evenly across zones with topologySpreadConstraints, followed by the kubectl commands to enable topology-aware routing on one Service, inspect its EndpointSlice zone hints, and check for hint-removal events.
When should you avoid topology aware routing?
Topology aware routing assumes that client traffic per zone roughly matches endpoint capacity per zone. When that assumption fails, the savings come with real risks.
Load imbalance. Hints allocate endpoints in proportion to each zone's allocatable CPU, not to where requests originate. If one zone receives most of the ingress, perhaps because a load balancer favours it, its local endpoints absorb that traffic while endpoints in the other zones sit idle.
Silent fallback. During scale-downs, node loss, or rollouts, the controller can remove hints when it can no longer allocate endpoints safely. Traffic spreads across zones again with no error and no alert. Unless you track hint presence and inter-zone transfer volume, the savings can disappear for weeks before anyone notices.
Rougher failover. Zone-local routing keeps sending traffic to degraded local pods as long as they pass readiness checks. Readiness probes must reflect real health, not just whether the process is running.
| Situation | Better choice |
|---|---|
| Fewer than ~3 replicas per zone | Leave default routing |
| Mesh or client-side load balancing | Configure the mesh's own locality settings |
| Low-traffic cluster | Skip it; savings are small |
| Skewed per-zone ingress | Fix ingress balance first |
Small services often lack enough endpoints per zone to qualify. Adding replicas so they qualify can cost more in compute than the transfer it saves. Service meshes and gRPC client-side balancing bypass kube-proxy, so hints have no effect on that traffic. Never apply internalTrafficPolicy: Local to general services. Any node without a local endpoint drops the traffic, which causes outages.
For low-traffic clusters, the engineering time, monitoring, and capacity headroom can outweigh the transfer bill you would save.
What are the alternatives for inter-zone traffic Kubernetes routing does not fix?
Topology aware routing only changes how kube-proxy picks endpoints for Service traffic. Many large cross-zone flows never pass through kube-proxy, so they need their own fixes.
Cloud load balancers with cross-zone balancing enabled spread requests evenly across all backends, so a request that enters in one zone often lands in another. Managed database primaries and caches live in one zone, so every write from the other zones crosses a boundary. NAT or egress gateways placed in a single zone pull all outbound traffic through that zone. Replication in databases and message brokers is cross-zone by design, because the redundancy only helps if copies live in different failure domains.
A service mesh can act as a separate routing layer. Its locality-aware load balancing prefers same-zone endpoints and fails over to other zones when local endpoints are unhealthy. This covers cases that kube-proxy hints do not handle well.
| Traffic source | Mitigation | Trade-off |
|---|---|---|
| Load balancer cross-zone balancing | Disable it and run balanced backends in every zone | Uneven zone capacity causes hot spots |
| Single-zone database primary or cache | Route reads to same-zone replicas | Replica lag means reads can be stale |
| Single-zone NAT/egress gateway | Run one gateway per zone | Each gateway adds fixed cost |
| Broker consumers | Use rack/zone-aware fetching from a same-zone replica | Requires broker and client support |
| Replication | Compress the data and tune the replication factor | Fewer replicas reduce durability |
Collapsing workloads into fewer zones removes the cost entirely. It also removes your protection against a zone outage. Make that choice explicitly as a reliability decision, not as a cost tweak.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to cross az data transfer cost, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.