What is cross-AZ data transfer cost in Kubernetes, and why does it happen by default?

Cross-AZ data transfer cost is what your cloud provider charges when bytes move between availability zones in the same region. On AWS, inter-AZ traffic is generally billed per GB, and the charge can apply on both the sending and the receiving side, so one transfer may be billed twice. Rates vary by region and change over time, so check current regional pricing rather than relying on a figure you remember.

Kubernetes does not consider zones by default. kube-proxy programs each node so that ClusterIP traffic is spread across all ready endpoints of a Service, wherever those endpoints run. Suppose a Service has replicas spread evenly across three zones and endpoints are chosen at random. A caller in any zone then lands in one of the other two zones about two-thirds of the time.

Where the hidden cross-AZ bytes come from

  • Service-to-service calls: every ClusterIP hop can cross a zone.
  • Load balancer to node: the load balancer can send traffic to a node in a zone other than the one where it was received.
  • NodePort forwarding: the receiving node can forward the packet to a pod on a node in another zone.
  • CoreDNS lookups: CoreDNS pods are often in a different zone from the pod making the query.
  • Stateful replication: database replicas and Kafka partition replicas copy every write across zones by design.
  • Log and metrics shipping: collectors send data to aggregators that may run in another zone.

A single request can pay for several of these hops:

                 Load balancer (AZ-a)
                        |
        +---------------+ (1) LB -> node in AZ-b
        v
 [AZ-a]          [AZ-b]  Node 2          [AZ-c]
                   | (2) NodePort -> pod in AZ-c
                   +-------------------> Pod (app)
                                           | (3) DNS lookup
 CoreDNS <---------------------------------+
   ^  (AZ-a)
   +-- (4) app calls a Service; endpoint chosen in AZ-a
                          +--------------------+
                          |   Load balancer    |
                          | entry point: AZ-a  |
                          +---------+----------+
                                    | (1) LB forwards to a node in AZ-b
                                    v
+----------------------+  +---------+------------+  +----------------------+
| AZ-a                 |  | AZ-b    |            |  | AZ-c                 |
|                      |  |  +------v-------+    |  |                      |
|                      |  |  | Node 2       |    |  |                      |
|                      |  |  | NodePort     |    |  |                      |
|                      |  |  +-------+------+    |  |  +--------------+    |
|                      |  |          +--------------->| Pod (app)    |    |
|                      |  |  (2) to pod in AZ-c  |  |  +-------+------+    |
|  +--------------+    |  |                      |  |          |           |
|  | CoreDNS      |<-------------------------------------------+           |
|  +--------------+    |  | (3) DNS query -> AZ-a|  |                      |
|                      |  |                      |  |                      |
+----------------------+  +----------------------+  +----------------------+

One request in a three-zone cluster can cross zone boundaries three times: the load balancer forwards to a node in AZ-b (1), the NodePort hop lands on a pod in AZ-c (2), and the pod's DNS lookup reaches a CoreDNS pod in AZ-a (3).

How do you measure which Kubernetes flows cross availability zones?

Start with the bill, because it sets the ceiling on what any optimization can save. In the Cost and Usage Report, filter on usage types containing DataTransfer-Regional-Bytes. This is the charge for traffic between availability zones in the same region. Group it by resource ID and by day. If the total is small compared to compute, stop here. If it is large, the report still cannot tell you which service caused it, so you need flow data.

VPC Flow Logs provide that. Each record has source and destination addresses, bytes, and the subnet it was captured on. Build a lookup table from subnet ID to AZ, plus each subnet's CIDR. With it, you can tag both endpoints of a flow with a zone and keep only the records where the two zones differ. Use the pkt-srcaddr and pkt-dstaddr fields where available. Without them, traffic that is NATed or proxied appears as the intermediate hop instead of the real endpoint.

To attribute bytes to workloads, join pod IPs to pod metadata. kubectl get pods -A -o wide lists each pod's IP and node. kubectl get nodes -L topology.kubernetes.io/zone maps nodes to zones. Pod IPs are recycled within minutes, so a single snapshot will misattribute traffic. Record pod IP assignments as intervals (IP, pod, start, end), for example from the pod watch API. Then join each flow on IP and timestamp. If your CNI uses an overlay, flow logs show node IPs, not pod IPs, and you cannot attribute below the node level.

This pipeline costs money too: log ingestion and storage, query compute, and engineering time to maintain it. Sample traffic or run the pipeline for a fixed window instead of indefinitely.

#!/usr/bin/env bash
set -euo pipefail

OUT=pod_ip_zone.tsv
TS=$(date -u +%Y-%m-%dT%H:%M:%SZ)

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}{end}' \
  > /tmp/node_zone.tsv

kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.nodeName}{"\t"}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.podIP}{"\t"}{.spec.hostNetwork}{"\n"}{end}' \
  > /tmp/pods.tsv

awk -F'\t' -v ts="$TS" 'BEGIN { OFS="\t" }
  NR == FNR { zone[$1] = $2; next }
  $4 != "" && $5 != "true" {
    print ts, $4, $2 "/" $3, $1, zone[$1]
  }' /tmp/node_zone.tsv /tmp/pods.tsv >> "$OUT"

A shell snapshot that records each pod's IP, node and zone with a timestamp, producing rows that can be collapsed into IP-assignment intervals and joined to VPC Flow Log records on both IP and time.

How much does inter-AZ traffic cost? A worked illustrative estimate

Once you have per-flow volumes, the cost math is simple. The inputs are what matter, so here they are stated plainly.

Illustrative estimate (assumed numbers, not a benchmark)

  • One client service calls one backend. Both are spread evenly across three zones.
  • Combined request and response volume: 2,000 GB/day (assumed).
  • Transfer is billed at $0.01/GB in each direction (assumed). That makes $0.02 per cross-zone GB.
  • The client picks endpoints at random, so about two-thirds of traffic lands in another zone.

Monthly cost = daily GB × (2/3) × per-GB rate × 2 directions × 30 days

2,000 × 0.667 × 0.01 × 2 × 30 ≈ $800/month.

Zone-local routing will not remove all of it. Endpoints fall back to other zones during rollouts, scale events, and zone imbalance. Assume it removes 90%, which saves about $720/month.

Keeping traffic local also has a price. Every zone needs enough backend capacity to absorb its own load spikes. In practice that means extra replicas and headroom, assumed here at $80/month. There is also a one-off engineering cost to configure and verify routing, plus ongoing attention during incidents.

Volume/day Cross-AZ cost/month Savings (90%) Extra replica cost Net/month
2,000 GB $800 $720 $80 +$640
200 GB $80 $72 $80 −$8
20 GB $8 $7.20 $80 −$72.80

The fixed cost of per-zone capacity does not shrink with traffic, but the savings do. Below a break-even volume, which is about 220 GB/day under these assumptions, optimizing loses money. Rank your flows by cross-zone GB and start with the few at the top.

Who should use zone-local routing?

Zone-local routing is worth the effort for teams that meet most of these conditions:

  • You run a multi-zone cluster on a cloud that bills inter-zone traffic. If the cluster lives in one zone, or your provider does not charge for traffic between zones, there is nothing to save.
  • Cross-AZ transfer is a visible line item. The DataTransfer-Regional-Bytes total from the measurement step is large enough that a reduction matters next to compute and storage. If it is a rounding error, stop.
  • A few flows carry most of the bytes. Chatty service pairs, log and metrics pipelines, and Kafka consumers are typical candidates. A handful of high-volume paths is easier to fix than traffic spread thinly across hundreds of services.
  • Traffic arrives roughly evenly across zones. Zone-local routing assumes each zone's backends can absorb that zone's own load. If one zone receives most requests, you will overload it.
  • Services run enough replicas to place several in every zone. A Service with two replicas in a three-zone cluster cannot have a local endpoint everywhere, and the hints controller will refuse to set hints.

In practice, the work usually falls to the platform or infrastructure team that owns the cluster, Service defaults, and ingress configuration, together with the owners of the few high-volume services. Application teams with low-volume services can usually stay on default routing.

How does it work? Topology-aware routing in Kubernetes

Kubernetes splits the work between two components. The EndpointSlice controller decides which endpoints should serve each zone and writes that decision into each endpoint as a hints.forZones entry. kube-proxy on each node reads those hints and programs only the endpoints hinted for its own zone. The routing rules on a node in zone A then never include pods in zone B, so the cross-zone hop disappears.

Routing is only half of it. Keeping traffic in zone end to end combines three layers: placement puts enough replicas in every zone (topology spread constraints, per-zone node groups), routing sends each request to a local endpoint (topology hints, externalTrafficPolicy: Local, IP-mode load balancer targets), and application-level settings keep stateful reads local (rack-aware consumers, per-zone read replicas). Each layer is covered below.

To opt a Service in, use an annotation or a field. Older releases used the service.kubernetes.io/topology-aware-hints annotation, later releases renamed it to service.kubernetes.io/topology-mode: Auto, and newer releases add a spec.trafficDistribution field. These options have changed between releases and do not behave identically, so check the documentation for your cluster's version before you choose one.

In annotation mode, the controller assigns endpoints to zones in proportion to each zone's allocatable CPU. This keeps any single zone from being overloaded. The controller declines to set hints, or removes hints it already set, in these cases:

  • There are fewer endpoints than zones.
  • Nodes lack the topology.kubernetes.io/zone label.
  • Per-zone capacity is too uneven to give each zone a fair share.

When this happens, routing falls back to cluster-wide routing. The Service keeps working, so nothing signals the change, but your cross-zone bill returns.

To catch this, check whether the Service's EndpointSlices still carry hints. List them with the kubernetes.io/service-name label selector. Also look for controller events on the Service that report hints being disabled. Run this check in CI or as a periodic alert, not only once at rollout. Scaling events and node replacements can drop hints at any time.

#!/usr/bin/env bash
set -euo pipefail

NS=shop
SVC=checkout

kubectl -n "$NS" annotate service "$SVC" \
  service.kubernetes.io/topology-mode=Auto --overwrite

kubectl -n "$NS" get endpointslices \
  -l kubernetes.io/service-name="$SVC" \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .endpoints[*]}{"  "}{.addresses[0]}{" zone="}{.zone}{" hints="}{.hints.forZones[*].name}{"\n"}{end}{end}'

kubectl -n "$NS" get events \
  --field-selector involvedObject.kind=Service,involvedObject.name="$SVC"

MISSING=$(kubectl -n "$NS" get endpointslices \
  -l kubernetes.io/service-name="$SVC" -o json \
  | jq '[.items[].endpoints[]? | select(.hints == null)] | length')

if [ "$MISSING" -gt 0 ]; then
  echo "ALERT: $MISSING endpoint(s) of $NS/$SVC have no zone hints" >&2
  exit 1
fi
echo "OK: all endpoints of $NS/$SVC carry zone hints"

Enables topology-aware routing on a Service with an annotation, lists each endpoint's zone hints from its EndpointSlices, shows controller events for the Service, and fails a CI or cron check when hints have been silently dropped.

How do you implement zone-local placement that actually keeps traffic local?

Routing hints only help when every zone has enough local endpoints, so placement comes first.

Spread replicas evenly

Add a topology spread constraint keyed on topology.kubernetes.io/zone with maxSkew: 1. Use whenUnsatisfiable: DoNotSchedule for services that depend on locality, so the scheduler does not pile replicas into one zone. Spreading alone does not survive load spikes. Give each zone headroom, or run one node group per zone so the autoscaler can add capacity exactly where pods are pending. If one zone runs out of endpoints, traffic falls back to crossing zones.

Keep DNS local

Every service call starts with a lookup. If CoreDNS pods sit in other zones, that is a cross-zone round trip for each cache miss. A node-local DNS cache answers repeated queries on the node itself.

Fix the ingress path

With cross-zone load balancing and instance-mode targets, traffic takes two hops. The load balancer sends a request to any node's NodePort, and kube-proxy then forwards it to a pod that may be in another zone. IP-mode targets register pod IPs directly, which removes the second hop. For NodePort or LoadBalancer Services, externalTrafficPolicy: Local keeps traffic on the receiving node and preserves the client IP. The tradeoff is that nodes without a ready pod fail health checks and stop receiving traffic, so uneven pod counts lead to uneven load.

Stateful and messaging systems

Configure rack or zone awareness so that:

  • consumers fetch from a replica in their own zone, and
  • read replicas serve local clients.

Durability replication still crosses zones. That traffic is the price of surviving a zone failure, so leave it in place.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 6
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      topologySpreadConstraints:
        - maxSkew: 1                               # zones differ by at most one replica
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule         # stay Pending rather than pile into one zone
          labelSelector:
            matchLabels:
              app: checkout
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.0.0
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: checkout
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local   # no kube-proxy forward to a pod on another node or zone
  selector:
    app: checkout
  ports:
    - port: 80
      targetPort: 8080

A Deployment that uses a zone topology spread constraint to keep replica counts balanced across zones, and a LoadBalancer Service with externalTrafficPolicy: Local so external traffic is not forwarded to pods on other nodes.

When should you avoid zone-local routing?

Zone-local routing removes a safety margin that cross-zone balancing gave you for free. Before enabling it, check whether your traffic and capacity are spread evenly enough to cope without that margin.

The main risk is skew. If one zone receives most of the requests, for example because a large client fleet or an ingress path is concentrated there, its pods saturate while replicas in other zones sit idle. Failover also gets weaker. A pod that is degraded but still passes its readiness probe keeps receiving local traffic, because routing stays in zone instead of spreading the load around it. And not every flow is worth chasing. Durability replication is the cost of surviving a zone failure, and for some services the savings never cover the extra replicas they need.

Trade-off Symptom Mitigation, or signal to skip
Skewed traffic per zone High latency or CPU in one zone while other zones are idle Scale each zone separately and keep the cross-zone fallback. Skip if traffic is uneven across zones.
Degraded pods keep local traffic Errors limited to one zone that load balancing would have hidden Use strict readiness probes and outlier detection.
Cross-zone balancing disabled at the edge Outage in a zone with few or no healthy targets Keep it enabled unless every zone always has healthy targets.
Cutting durability replication Data loss after a zone failure Don't cut it. Optimize reads only.
Low-volume service Extra replicas cost more than the transfer they save Skip it. Compare replica cost with the transfer bill.
Small line item overall Transfer is a minor share of the bill Leave it alone. Don't collapse the cluster to a single AZ to save it.

What are the alternatives, and how should you prioritize the work?

Each option trades locality against failover and effort differently, so match the tool to the flow rather than picking one for the whole cluster.

Option Locality Failover behavior Operational effort
Default cluster-wide routing None; with endpoints spread evenly across three zones, roughly two-thirds of requests leave the zone Best; any healthy endpoint serves None
Topology-aware hints Zone-local when endpoints are proportional to zone capacity Falls back to cluster-wide when hints are unsafe or missing Low: an annotation, plus keeping replicas balanced per zone
internalTrafficPolicy: Local Node-local, stricter than zone Traffic is dropped if the node has no local endpoint Low to set, but needs DaemonSet-style placement
Zone-aware clients / mesh locality Zone-preferred with weighted spillover Configurable failover to other zones Medium to high: mesh or client libraries to run and tune
App-level awareness (Kafka rack-aware follower fetching, read replicas per zone) Removes the largest data-plane flows Defined by the system's own replication Medium: broker and client config, plus testing
Single-AZ deployment Total None; the whole zone is one failure domain Low, but the risk is high

Prioritize by bytes, not by service count

Rank flows by cross-AZ bytes using the measurement from section 2. Fix the top three to five, re-measure for a full billing cycle, and expand only where the remaining flows justify the work. Cross-zone volume usually concentrates in a few paths, such as replication, log shipping, and chatty service pairs. Fixing those captures most of the savings while leaving most services on default routing. That avoids per-zone capacity tuning and new failover tests for workloads whose transfer never showed up on the bill.

How can it save engineering time?

The direct payoff of keeping traffic in zone is a smaller transfer bill, but the way you approach the work also decides how much engineering time it costs or saves.

  • Configuration instead of code. Topology hints, spread constraints, externalTrafficPolicy, and load balancer target mode are Service and Deployment settings. A platform team can apply them through shared manifests or templates without asking each application team to change client code, which is usually far slower to coordinate than a manifest change.
  • Fewer bill investigations. Without measurement, a jump in the cross-AZ line item starts a manual hunt through flow logs and pod lists. With the time-aware pod-IP join in place, attribution becomes a query you rerun, not a new project each time the bill moves.
  • Silent regressions get caught automatically. Hints can disappear after a scale-down or node replacement while the Service keeps working. The CI or periodic EndpointSlice check turns that into an alert with the Service name attached, instead of a cost surprise discovered weeks later and traced back by hand.
  • Less wasted effort on low-value flows. The break-even estimate and the bytes-based ranking tell you which services to leave alone. Skipping low-volume services avoids per-zone capacity tuning, extra failover tests, and the incident debugging that zone-skew problems cause, for workloads where the savings would not cover the work.
  • Time-boxed measurement. Running the flow-log pipeline for a fixed window, or on sampled traffic, gives you the ranking without committing engineers to maintaining another always-on data pipeline.

The time savings depend on doing the steps in order: measure, rank, fix the few largest flows, add the hint check, then stop until the bill gives you a reason to look again.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cross az data transfer cost kubernetes, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.