What is cross-AZ data transfer cost and why does Kubernetes generate so much of it?

On AWS, traffic that stays inside a single availability zone over private IPs is generally not billed for data transfer. Traffic between two AZs in the same region is billed per GB, and it is charged in each direction: the sending side pays for data out and the receiving side pays for data in. A gigabyte that crosses a zone boundary is therefore billed twice. Rates change and vary by region, so check the current figures on the provider's pricing page rather than relying on a number quoted in an article.

Kubernetes produces a lot of this traffic because, by default, Service routing does not consider zones. kube-proxy programs rules that spread connections across all ready endpoints of a Service, no matter which zone they run in. A client pod has no preference for a backend on its own node or in its own zone.

Take a Service whose backends are spread evenly across three zones. A client in zone A picks among them roughly uniformly:

Client Endpoint Endpoint zone Crosses zone boundary?
pod in zone A backend-1 A No
pod in zone A backend-2 B Yes
pod in zone A backend-3 C Yes

Only one of the three possible paths stays local, so about two-thirds of requests cross a zone. Responses come back over the same path, so they cross the zone boundary too. The effect also compounds: every hop in a chain of microservices repeats the same odds.

+------- Zone A -------+  +------- Zone B -------+  +------- Zone C -------+
|                      |  |                      |  |                      |
|  +----------------+  |  |                      |  |                      |
|  |   client pod   |  |  |                      |  |                      |
|  +-------+--------+  |  |                      |  |                      |
|          | kube-proxy|  |                      |  |                      |
|          +-----------+--+----------+-----------+--+----------+           |
|          |           |  |          |           |  |          |           |
|          v ~1/3      |  |          v ~1/3      |  |          v ~1/3      |
|  +----------------+  |  |  +----------------+  |  |  +----------------+  |
|  |   backend-1    |  |  |  |   backend-2    |  |  |  |   backend-3    |  |
|  +----------------+  |  |  +----------------+  |  |  +----------------+  |
|                      |  |                      |  |                      |
|  same AZ: not billed |  |  cross-AZ: billed    |  |  cross-AZ: billed    |
|                      |  |  in both directions  |  |  in both directions  |
+----------------------+  +----------------------+  +----------------------+

 kube-proxy spreads connections over ALL ready endpoints, ignoring zone.
 Result: ~2/3 of requests, and their responses, cross a zone boundary.

With default Service routing, a client pod in zone A reaches backends in zones A, B and C about equally often, so roughly two of every three requests are billed as cross-AZ traffic in both directions.

How does it work? The approach end to end

Cutting inter-zone traffic in Kubernetes is a measure-then-route process, not a single setting. It has five steps, each covered in detail in the sections below:

  1. Size the problem. Use the cloud bill to find out how large the inter-zone data transfer line is and whether it is growing. If it is small, stop here.
  2. Attribute the bytes. Join VPC flow logs with pod-to-node and node-to-zone mappings so each cross-zone flow is credited to a source and destination workload. This produces a ranked list of paths.
  3. Change routing for the top paths. For the high-volume Service-to-Service paths, opt the Service in to topology aware routing. The EndpointSlice controller then attaches zone hints to endpoints, and kube-proxy on each node programs only the endpoints hinted for that node's zone. Clients keep calling the same Service name. Only the endpoint selection changes.
  4. Keep each zone able to serve its own traffic. Because traffic now stays in-zone, every zone needs enough replicas and node capacity. Topology spread constraints and per-zone autoscaling keep the distribution even.
  5. Verify and handle the rest. Confirm the hints were actually applied, compare byte counts and billing before and after, and watch per-zone saturation. Traffic that Service routing cannot touch, such as replication, load balancer hops and NAT egress, needs separate fixes or a deliberate decision to keep paying for it.

Who should use it?

This approach is worth the effort for teams that meet most of these conditions:

  • They run multi-zone clusters on a provider that bills inter-zone traffic. A cluster confined to a single zone has no cross-zone Service traffic to reduce.
  • The inter-zone data transfer line on the bill is large enough to matter compared with the engineering time a fix needs. The estimate in the next section shows how to check this.
  • They have chatty, high-volume Service-to-Service paths, such as API tiers calling backends, request fan-out across microservices, or services that return large responses.
  • Those services already run several replicas in every zone, and their callers are spread across zones roughly in proportion to those replicas.

The work usually falls to platform or infrastructure engineers who own cluster configuration, often together with whoever owns cloud cost reporting. Service owners need to be involved too, because they must accept the replica-count and scaling constraints that zone-local routing places on their workloads.

It is a poor fit for clusters with a small inter-zone line, for low-replica services where extra pods would cost more than the transfer saved, and for workloads whose clients are concentrated in one zone. The section on when to avoid topology aware routing covers these cases.

How much does inter-zone traffic actually cost? An illustrative estimate

Illustrative estimate (assumed numbers, not a benchmark). Every input below is an assumption. Replace each one with figures from your own cluster and your provider's current price list.

Assumptions:

  • Three zones, with workloads spread evenly across them.
  • With zone-unaware routing, two-thirds of service-to-service traffic crosses a zone.
  • An assumed rate of $0.01 per GB, charged in each direction. The sending side and the receiving side are both billed, so the effective rate per GB that crosses a zone is $0.02.

The formula:

monthly cost = monthly GB × cross-zone fraction × per-GB rate × 2

Traffic category Assumed monthly volume Cross-zone fraction Cross-zone GB Estimated monthly cost
Chatty microservices (requests and responses) 100,000 GB 2/3 ~66,700 ~$1,333
Stateful replication (each replica in a different zone) 20,000 GB 1.0 20,000 $400
Log and metric shipping to a collector in one zone 15,000 GB 2/3 10,000 $200
Total 135,000 GB ~96,700 ~$1,933

The categories behave differently, and that changes what is worth fixing:

  • Microservice traffic dominates the total. Routing changes can reduce it without making the system less resilient.
  • Replication traffic crosses zones by design. Zone redundancy is the reason the replicas exist, so reducing it usually means accepting weaker durability.
  • Log and metric shipping can be reduced with compression or per-zone collectors. In this example, the most it could save is $200 a month.

For each path, compare the estimated savings with the engineering time the fix needs. If a fix takes two engineer-weeks and saves $200 a month, it pays back slowly. Testing and ongoing maintenance stretch that payback further. A fix aimed at the $1,333 microservice line will usually pay for itself much sooner. Rank fixes by savings per hour of work, not by total traffic volume.

How do you find which services cause the most inter-zone traffic?

Ranking fixes requires data on where the bytes go. Without it, teams usually tune whichever service they suspect, and that often is not the one driving the bill.

Start with the bill. In AWS Cost Explorer, filter to the regional data transfer usage type to see the size of the inter-zone line and its trend. If node groups or instances carry cost allocation tags for team or workload, you can split that line further. It will show which clusters and node pools are expensive. It will not show which service-to-service calls cause the cost.

For that, enable VPC flow logs on the cluster subnets. Each record carries source IP, destination IP, and byte count. Pod and node IPs mean nothing on their own, so you need to enrich them:

  • kubectl get pods -A -o wide maps each pod IP to its node.
  • kubectl get nodes -L topology.kubernetes.io/zone maps each node to its zone.

Join the two outputs into an IP-to-pod-to-zone table. Then attach it to each flow record and keep only the records where the source zone differs from the destination zone. Roll those up by owning workload or Service, and you have per-Service cross-zone byte counts. If your CNI uses an overlay, flow logs see node IPs, not pod IPs, so attribution stops at the node unless you also collect pod-level metrics.

This approach has two costs. First, flow logs and the enrichment pipeline add their own storage and query charges, so sample, or run them for a bounded window. Second, pod IPs get reused. If pods churn faster than you snapshot metadata, bytes get credited to the wrong workload. Take snapshots at least as often as your deployment cadence.

The output you want before changing anything is a ranked list of source-to-destination paths by cross-zone volume.

#!/usr/bin/env bash
set -euo pipefail
FLOWS="${1:-flows.txt}"

kubectl get nodes -L topology.kubernetes.io/zone --no-headers \
  | awk '{print $1, $NF}' > node_zone.txt

kubectl get pods -A --no-headers \
  -o custom-columns=IP:.status.podIP,NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName \
  | awk '$1 != "<none>"' > pod_node.txt

awk 'NR==FNR {zone[$1]=$2; next}
     {owner=$3; sub(/-[a-z0-9]+-[a-z0-9]+$/, "", owner)
      print $1, $2 "/" owner, zone[$4]}' node_zone.txt pod_node.txt > ip_map.txt

awk 'NR==FNR {owner[$1]=$2; zone[$1]=$3; next}
     $13 == "ACCEPT" && ($4 in zone) && ($5 in zone) && zone[$4] != zone[$5] {
       bytes[owner[$4] " -> " owner[$5]] += $10
     }
     END {for (k in bytes) print bytes[k], k}' ip_map.txt "$FLOWS" \
  | sort -rn | head -20

A shell sketch that joins kubectl pod-to-node and node-to-zone mappings with VPC flow log records and outputs a ranked list of cross-zone workload-to-workload paths by bytes.

How does Kubernetes topology aware routing work?

With the ranked list of source-to-destination paths in hand, the main lever inside the cluster is how Service traffic picks its endpoints. Topology aware routing changes that choice.

Hints on EndpointSlices. When a Service opts in, the EndpointSlice controller adds a hint to each endpoint, under endpoints[].hints.forZones, naming the zone that should consume it. The controller hands out endpoints to zones in proportion to each zone's share of allocatable CPU. This way a zone with more nodes receives more endpoints, and simple pod counts do not decide the split.

kube-proxy filtering. kube-proxy reads those hints and programs only the endpoints hinted for its own node's zone. The traffic stays zone-local because the client-side proxy never routes to the other zones. The application does nothing differently.

Opt-in. The Service has to request this behaviour, usually through an annotation such as service.kubernetes.io/topology-mode: Auto. To check whether hints were actually set, run kubectl get endpointslices -l kubernetes.io/service-name=<service> -o yaml and look for hints.

When hints are withheld or removed. The controller declines to set hints, or strips existing ones, when:

  • a zone has too few endpoints to receive a proportional share
  • capacity is so uneven that a balanced allocation isn't possible
  • nodes lack zone labels

kube-proxy also falls back to cluster-wide routing if any endpoint in the Service is missing a hint. Because of these fallbacks, enabling the feature does not mean it is working. Verify it.

Version caveat. The annotation name, its accepted values, the newer trafficDistribution field, and the allocation rules have all changed across Kubernetes releases. Confirm the mechanism in the documentation for your cluster's version. Do not copy configuration from another cluster or article.

apiVersion: v1
kind: Service
metadata:
  name: checkout
  namespace: shop
  annotations:
    # Ask the EndpointSlice controller to add per-zone hints
    service.kubernetes.io/topology-mode: Auto
spec:
  selector:
    app: checkout
  ports:
    - name: http
      port: 80
      targetPort: 8080
---

A minimal Service that opts in to topology aware routing with the 1.27+ annotation, plus a kubectl command that lists each endpoint's zone and zone hint so you can confirm the hints were actually applied.

How do you implement zone-local routing without creating hot spots?

Apply this only to the high-volume, chatty service-to-service paths you identified during measurement. Most workloads move too little data across zones to justify the operational constraints.

Prerequisites

Every node needs the topology.kubernetes.io/zone label. Managed node pools on major clouds usually set it automatically. Self-managed nodes may not have it, so confirm with kubectl get nodes -L topology.kubernetes.io/zone before relying on it. Without the label, zone-aware routing has nothing to route on.

Keep replicas even

Zone-local routing sends each zone's traffic to that zone's endpoints. If one zone holds two replicas and another holds six, the smaller zone saturates. Add a topologySpreadConstraints entry to the Deployment's pod template with these settings:

  • topologyKey: topology.kubernetes.io/zone
  • maxSkew: 1
  • a labelSelector that matches the pods
  • whenUnsatisfiable: DoNotSchedule, if even placement matters more than scheduling every pod

Keep capacity balanced as you scale

The scheduler enforces spread only when it places a pod. Scale-down can leave zones uneven, so check the distribution after scaling events. Set replica counts and HPA minimums to multiples of the zone count. Configure node autoscaling so each zone can add capacity independently. If one zone runs out of nodes, its pods stay pending while traffic keeps arriving there.

Verify

Compare inter-zone byte counts and AWS data transfer billing line items before and after the change, over equivalent traffic windows, such as the same weekday and hours. During rollout and afterward, track latency, CPU saturation, and error rates per zone, not just as service-wide averages. An overall average can hide a single overloaded zone.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  # Multiple of the zone count (3 zones here) so each zone can hold an equal share
  replicas: 6
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      topologySpreadConstraints:
        # Spread across zones using the node label set by the cloud provider
        - topologyKey: topology.kubernetes.io/zone
          # Zone replica counts may differ by at most one
          maxSkew: 1
          # Leave a pod Pending rather than place it unevenly
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: orders-api
      containers:
        - name: orders-api
          image: registry.example.com/orders-api:1.0.0
          resources:
            # Requests let the scheduler and autoscalers size each zone correctly
            requests:
              cpu: 250m
              memory: 256Mi

A Deployment that uses topologySpreadConstraints keyed on topology.kubernetes.io/zone with maxSkew 1 to keep replicas even across three zones, so zone-local routing does not overload a single zone.

When should you avoid topology aware routing?

Zone-local routing assumes that clients and serving capacity are spread across zones in roughly the same proportions. When that assumption breaks, the savings come with costs that may matter more than the bill.

Uneven client traffic. Hints allocate endpoints by each zone's capacity, not by where requests come from. If most callers run in one zone, for example because a batch job is pinned there or the ingress tier is skewed toward it, that zone's endpoints take a disproportionate share while the others sit idle. You pay less for transfer but get saturated pods and higher tail latency. Autoscaling on service-wide CPU may not respond, because the average still looks healthy.

Capacity loss. When a zone loses nodes, the control plane may remove hints entirely, or traffic may shift abruptly to the remaining endpoints instead of tapering off. Test this directly: drain one zone's nodes in a staging cluster and measure what the surviving endpoints receive. Don't assume failover degrades gracefully.

Replica inflation. Zone-local routing needs enough replicas in every zone. A service that two pods could handle may need three or more to cover three zones safely. For low-volume services, that extra compute can cost more than the transfer you save.

Decision checklist

  • Traffic volume: Is this service's cross-zone traffic large enough to show up on the bill?
  • Client distribution: Are callers spread across zones roughly in proportion to endpoints?
  • Replica count: Does the service already run enough replicas per zone, without adding pods just to satisfy routing?

If any answer is no, keep the default routing.

How can it save engineering time?

The main saving is in spending effort where it pays back, not in the routing change itself. Several parts of the approach cut wasted work:

  • Measuring first avoids tuning the wrong service. The ranked list of cross-zone paths replaces guesswork. Teams spend their time on the few paths that drive the bill, not on whichever service someone suspects.
  • Ranking by savings per hour of work filters out low-value fixes early. A path that would take weeks to change for a small monthly saving can be dropped before anyone starts on it, as the illustrative estimate above shows.
  • The routing change needs no application code. Topology aware routing is a Service-level opt-in handled by the EndpointSlice controller and kube-proxy. Service owners do not have to add zone-aware client logic, custom discovery, or per-zone Service names, and they do not have to maintain that code afterwards.
  • The measurement pipeline can be reused. The flow log join and the IP-to-workload mapping built for the first investigation can be run again after each change to confirm the effect, and later to catch regressions when new services or traffic patterns appear. Each review does not have to start from scratch.
  • Verifying hints directly shortens debugging. Checking EndpointSlice hints and controller events shows within minutes whether a Service is actually routing zone-locally. Without this check, a team may wait a full billing cycle to find that the controller withheld hints.
  • The decision checklist prevents work that would have to be undone. Skipping services with skewed clients or too few replicas avoids hot-spot incidents, rollbacks, and the on-call time they consume.

The trade-off is ongoing upkeep. Spread constraints, replica minimums that are multiples of the zone count, and per-zone monitoring all need to be maintained. Include that recurring effort when you compare a fix's cost with its savings.

What are the alternatives for inter-zone traffic outside Service routing?

Service routing only covers pod-to-pod traffic through ClusterIP. Several other paths cross zones, and they often cost more, because they carry replication streams or all external traffic.

Source Mitigation Cost trade-off Resilience trade-off
Stateful replication (databases, Kafka-style brokers, consensus stores) Usually keep it. Reduce volume through compression or fewer replicas only where durability targets allow Saves bytes, but each replica removed lowers redundancy Replicas in other zones are what let the system survive a zone outage
Clients reading from a leader in another zone Follower or rack-aware reads, where the system supports them Removes read traffic. Writes still go to the leader Follower reads may return slightly stale data
Load balancers forwarding to nodes in other zones Target pods directly instead of node ports, or keep the load balancer's traffic in-zone Removes the extra node-to-pod hop Zones with few healthy pods can be overloaded if in-zone traffic is enforced
NAT gateway in another zone One NAT or egress path per zone, with route tables kept per zone Adds fixed hourly cost for each gateway Removes a single-zone dependency for egress
Chatty, tightly coupled services Co-locate them with pod affinity, or merge them No new infrastructure Pods packed into one zone fail together

Cross-zone replication usually exists on purpose for durability. Do not remove it for cost reasons alone. Running everything in a single AZ removes most of these charges, but the workload no longer survives a zone outage. That is a business risk decision, not a tuning change. Per-zone NAT and load balancer setups add fixed hourly costs and more infrastructure to operate. Compare those costs against the transfer savings you measured.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cross az data transfer cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.