Cross-AZ Data Transfer Cost in Kubernetes: Measure, Then Fix

Why multi-AZ Kubernetes clusters send traffic across zones by default

Cross-AZ data transfer cost is what your cloud provider charges when bytes move between availability zones in the same region. In a multi-AZ Kubernetes cluster, much of that traffic is created by the platform's default behavior rather than by anything your application does on purpose.

Two defaults combine to produce it. First, the scheduler spreads replicas across zones for resilience, through default spreading behavior or explicit topology spread constraints. A Deployment with nine replicas in a three-zone cluster typically lands three Pods per zone. Second, Service load balancing ignores zones. kube-proxy in iptables mode picks an endpoint at random, IPVS mode commonly uses round-robin, and eBPF dataplanes behave similarly unless configured otherwise. None of them prefers an endpoint in the caller's zone by default.

The math follows directly. If endpoints are spread evenly across three zones and selection is uniform, a caller in any zone hits a local endpoint about one-third of the time. Roughly two-thirds of Service-to-Service requests cross a zone boundary, and their responses cross back.

On AWS, inter-AZ traffic within a region is billed per GB, and the charge applies in both directions: data leaving one AZ and data arriving in the other are each billed. Rates change, so check the current figures on the AWS pricing page rather than relying on a number you remember.

This matters most to platform and backend teams running high-volume, chatty services across zones. Examples include microservice meshes, caches, message brokers, and anything that fans out many small requests per user action.

Copy          +---------------------------------------------+
          | caller Pod (Zone A)                         |
          +---------------------------------------------+
                               |
                               v
          +---------------------------------------------+
          | Service load balancing on caller's node     |
          | kube-proxy iptables (random) / IPVS (rr) /  |
          | eBPF dataplane: endpoint picked w/o zone    |
          +---------------------------------------------+
               |                |                |
             ~1/3             ~1/3             ~1/3
               |                |                |
               v                v                v
      +--------------+ +--------------+ +--------------+
      | Zone A       | | Zone B       | | Zone C       |
      | backend Pods | | backend Pods | | backend Pods |
      | (3 of 9)     | | (3 of 9)     | | (3 of 9)     |
      +--------------+ +--------------+ +--------------+
        same zone:       cross-AZ:        cross-AZ:
        no inter-AZ      request A->B     request A->C
        charge           response B->A    response C->A
                         per-GB charge,   per-GB charge,
                         each direction   each direction

      Result: ~2/3 of Service-to-Service requests leave
      the caller's zone, and their responses cross back.

With replicas spread evenly across three zones and zone-unaware endpoint selection, about two-thirds of a caller's requests and their responses cross an AZ boundary, and AWS bills inter-AZ traffic per GB in each direction.

Who should use this approach

The measure-then-fix approach in this article is for teams that run Kubernetes across multiple availability zones and see a meaningful regional data transfer line on their cloud bill. In practice, that usually means:

• Platform and infrastructure teams who own the cluster, its networking (CNI, kube-proxy or eBPF dataplane, NAT, load balancers), and the shared agents that run on every node. They control most of the defaults that create accidental cross-zone traffic.
• Backend service owners whose services exchange large or frequent payloads with other services, caches, or brokers. They decide whether a Service gets zone-aware routing and whether their clients read from a local replica.
• FinOps or cost-owning engineers who need to explain where inter-AZ spend comes from and which team should act on it.

It is less useful if your cluster runs in a single zone, if your services are low-traffic, or if most of your cross-zone bytes come from replication you have deliberately configured for durability. In those cases, the measurement step will usually show that there is little to fix, which is still a useful result.

How the measure-then-fix approach works

The approach has three stages, and each later section covers one of them in detail:

1. Find the sources. Identify the paths that move bytes between zones, including Service-to-Service calls, broker and datastore traffic, load balancers, NAT gateways, and telemetry pipelines. Separate the crossings that protect durability from the ones that are only a side effect of placement.
2. Measure and attribute. Use the Cost and Usage Report to see how much regional transfer you pay for and which resources generate it. Then use VPC Flow Logs, mapped to zones and to Pods or workloads, to rank the specific source-destination flows that cross zones.
3. Fix the top flows, then validate. For each high-volume flow, pick the matching fix: zone-aware Service routing for traffic through kube-proxy or a mesh, zone-aware clients for brokers and datastores, per-zone NAT gateways or collectors for infrastructure paths. Compare cross-AZ bytes and per-zone latency and CPU before and after, and confirm the bill moved.

The order matters. Fixes such as topology aware routing change how load is distributed, and fixes such as per-AZ NAT gateways add recurring cost. Measuring first tells you which flows justify those trade-offs.

Where hidden cross-zone traffic comes from

Service-to-service calls are only part of the bill. Several other paths move bytes between zones, and the key question for each is whether the crossing protects durability or is just a side effect of placement.

Replication traffic is the price of surviving a zone outage, so reduce it by tuning replication factor or compression, not by removing it. The accidental sources scale with node count and request volume, so they tend to grow unnoticed. For the same reason, they are usually the cheapest to fix: add per-zone NAT gateways, run collectors as per-zone or node-local agents, and configure consumers to read from a replica in their own zone.

How to measure and attribute cross-AZ spend before changing anything

The inter-AZ charge doesn't appear as its own line in the bill. In the Cost and Usage Report, filter line_item_usage_type for values containing DataTransfer-Regional-Bytes, which carry a region prefix such as USE1-. Group by line_item_resource_id to find which instances or interfaces generate it. This usage type also covers other same-region transfer, such as traffic over VPC peering or to public IPs in the region, so treat it as an upper bound. It also counts both sides of a cross-zone flow, because each direction is billed.

The CUR tells you how much you spend, not which workloads cause it. For that, enable VPC Flow Logs with a custom format that includes srcaddr, dstaddr, bytes, subnet-id and az-id. The az-id field describes only the interface that logged the record, not the peer. Build a subnet-CIDR-to-AZ lookup and resolve the zone of both addresses, then keep only the records where the two zones differ.

Next, translate IPs into Pods and namespaces with an in-cluster network observability tool, or with snapshots of Pod IPs taken during the capture window. Pod IPs get reused, so the mapping must be time-aware. If your CNI uses an overlay, flow logs see node IPs instead of Pod IPs. In that case, attribution has to come from the in-cluster tool. Aggregate bytes by source and destination workload to get a ranked list of the top cross-zone flows.

Collection costs money too. Flow log ingestion and storage, and high-cardinality metrics, are billed continuously. Capture for a fixed window, such as one representative week, on the subnets that matter, then turn collection off.

Copy-- Rank resources by same-region data transfer charges in the Cost and Usage Report.
-- Run in Athena against your CUR table; replace the database and table names.
-- DataTransfer-Regional-Bytes includes cross-AZ traffic, but also VPC peering and
-- same-region public IP traffic, so treat these totals as an upper bound.
SELECT
  line_item_usage_type,
  line_item_resource_id,                      -- instance or network interface ID
  SUM(line_item_usage_amount)   AS gb_transferred,
  SUM(line_item_unblended_cost) AS cost_usd
FROM cur_db.cur_table
WHERE line_item_line_item_type = 'Usage'
  -- Matches region-prefixed values such as USE1-DataTransfer-Regional-Bytes
  AND line_item_usage_type LIKE '%DataTransfer-Regional-Bytes'
  -- Limit the scan to one billing period to keep the query cheap
  AND line_item_usage_start_date >= TIMESTAMP '2026-09-01 00:00:00'
  AND line_item_usage_start_date <  TIMESTAMP '2026-10-01 00:00:00'
GROUP BY line_item_usage_type, line_item_resource_id
ORDER BY cost_usd DESC
LIMIT 25;

An Athena query over Cost and Usage Report data that filters for the regional data transfer usage type and ranks the instances and interfaces that generate the most spend.

Illustrative estimate: what cross-zone Service traffic costs per month

Illustrative estimate (assumed numbers, not a benchmark). Every input below is an assumption. Replace each one with your own measurements.

• 2,000 requests/s between two Services, averaged over a 30-day month (2,592,000 s)
• 20 KB per call: a 4 KB request plus a 16 KB response
• Pods spread evenly across three zones, with no zone preference, so about 2/3 of calls cross a zone boundary
• $0.01 per GB, charged on both the sending and receiving side ($0.02 per GB crossed)

Formula: monthly cost = requests/s × seconds/month × GB per call × cross-zone fraction × 2 × rate

Monthly volume is 2,000 × 2,592,000 × 0.00002 GB = 103,680 GB. Two-thirds of that, 69,120 GB, crosses zones.

The cost scales linearly with request rate and payload size. Payload usually matters more, because response sizes vary far more between services than request rates do.

About $11,000 a year can justify a few days of engineering time. It does not automatically justify fixed-cost fixes. Per-AZ NAT gateways carry their own hourly charges. Per-zone agents or proxies consume CPU and memory on every node. Compare those recurring costs against the savings before committing to them.

At one-tenth the payload, the same savings drop to about $92 a month. At that level, a configuration change is worth making, but new infrastructure is not.

How topology aware routing works and how to verify it

Kubernetes offers two ways to keep Service traffic in-zone. Topology Aware Routing is enabled with the service.kubernetes.io/topology-mode: Auto annotation. The EndpointSlice controller then writes a hints.forZones entry on each endpoint, sized so that each zone gets a share of endpoints proportional to its allocatable CPU. kube-proxy reads those hints and routes only to endpoints hinted for its own zone. The newer trafficDistribution: PreferClose field on the Service spec is simpler. It tells the proxy to prefer same-zone endpoints when any exist, and it does no CPU-proportional balancing.

The annotation's main problem is that it fails silently. The controller withholds hints if a zone has no endpoints, if nodes lack the topology.kubernetes.io/zone label, or if proportional allocation would overload some zone beyond its tolerance. That last condition is common with few replicas or uneven node sizes. For example, three replicas cannot be split proportionally across zones with unequal CPU. When hints are absent, kube-proxy routes cluster-wide and nothing errors.

To confirm hints are present, list the slices with kubectl get endpointslices -l kubernetes.io/service-name=<name> -o yaml and check that every endpoint carries hints.forZones. Also check the Service's events for messages explaining why hints were disabled.

To keep in-zone routing working, use topologySpreadConstraints with topologyKey: topology.kubernetes.io/zone so replicas spread evenly across zones. Plan capacity per zone as well. With PreferClose, a zone running one replica still receives all of its local clients' traffic, so autoscaling has to keep the smallest zone large enough to carry its own share of the load.

Copy#!/usr/bin/env bash
set -euo pipefail

SVC=checkout        # placeholder Service name
NS=shop             # placeholder namespace

kubectl get nodes -L topology.kubernetes.io/zone

kubectl get service "$SVC" -n "$NS" -o jsonpath='{.metadata.annotations.service\.kubernetes\.io/topology-mode}{"\t"}{.spec.trafficDistribution}{"\n"}'

kubectl get endpointslices -n "$NS" \
  -l kubernetes.io/service-name="$SVC" \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\t"}{.zone}{"\t"}{.hints.forZones[*].name}{"\n"}{end}'

kubectl get endpointslices -n "$NS" \
  -l kubernetes.io/service-name="$SVC" \
  -o jsonpath='{range .items[*].endpoints[*]}{.zone}{"\n"}{end}' | sort | uniq -c

kubectl get events -n "$NS" \
  --field-selector involvedObject.kind=Service,involvedObject.name="$SVC"

A shell check that confirms zone labels, the Service's routing mode, per-endpoint zone hints, and endpoint counts per zone, then lists the Service events that explain why hints were withheld.

Alternatives and fixes outside Kubernetes Service routing

Service routing only covers traffic that passes through kube-proxy or a service mesh. Several large sources of cross-zone bytes never touch a Service, so they need fixes at the client, network, or agent level.

Zone-aware clients. Some datastores and brokers can serve reads from a nearby replica. Kafka's follower fetching lets a consumer that declares its rack read from a replica in its own zone instead of the partition leader. Similar read-preference settings exist in some databases. This saves the most on read-heavy workloads. Writes still replicate across zones, because that replication is where the durability comes from.

One NAT gateway per AZ. A single shared NAT gateway makes every other zone pay inter-AZ transfer on egress, on top of the NAT processing charge. Adding a gateway per zone, with each private subnet's route table pointing to its local gateway, removes that hop. The trade-off is a fixed hourly charge per extra gateway, so the change only pays off once egress volume is high enough to cover it.

Cross-zone load balancing. When this is enabled, a load balancer node can forward traffic to targets in any zone, and on some load balancer types that forwarding is billed as inter-AZ transfer. Disabling it keeps traffic local. In exchange, each zone needs enough healthy targets, or the load spreads unevenly and a zone with few targets can overload.

Telemetry. Node agents that ship logs and metrics to a collector in one zone produce constant cross-zone traffic. Two fixes help: running a collector per zone, and batching and compressing data before it leaves the node.

Single-AZ consolidation. Running everything in one zone removes the charge entirely, but it also removes zone-failure tolerance. That makes it a business decision about availability, and it should be made by the people who own that risk rather than applied quietly as a cost optimization.

How measuring first saves engineering time

The time savings come mostly from avoiding work that would not pay off.

• You fix a short list instead of every Service. A ranked list of cross-zone flows tells you which few workload pairs carry most of the bytes. Without it, teams tend to roll topology annotations or new infrastructure across many Services and then spend time debugging load imbalance on Services that were never expensive.
• You skip changes that cost more than they save. The formula in the illustrative estimate lets you price a flow in minutes. Flows that work out to a few dollars a month can be dropped from the list before anyone writes a change, reviews it, or monitors it.
• You match each flow to the cheapest fix. Knowing whether a flow goes through a Service, a broker client, a NAT gateway, or a telemetry pipeline tells you which team owns it and which lever applies. That avoids trying Service-level routing on traffic that never passes through kube-proxy.
• You catch silent failures quickly. The EndpointSlice check script turns "is topology aware routing actually on?" into a quick command run instead of a longer investigation of why the bill did not move.
• You avoid repeated debates. A before-and-after comparison of flow log bytes and the bill gives a concrete answer to whether a change worked, so the same question does not have to be reopened each billing cycle.

Collection also has a bounded cost in time: a fixed capture window on the relevant subnets, analyzed once, rather than a permanent observability project.

When to avoid zone-local routing, and how to validate the change

Zone-local routing assumes each zone receives roughly the traffic its replicas can absorb. When that assumption fails, it is better not to use it.

When to avoid it

• Uneven clients or replicas. If most callers sit in one zone, or replicas are spread unevenly, the local Pods in that zone take a disproportionate share of requests. The result is CPU saturation and higher tail latency in that zone, while capacity in other zones goes unused.
• Partial zone degradation. A Pod can pass its readiness probe while its zone has problems, such as a slow storage backend or packet loss to a dependency. Zone-local routing keeps sending traffic to those endpoints because they still report ready. Without zone-local routing, those requests would be spread across healthy zones.
• Durability traffic. Synchronous replication and quorum writes cross zones on purpose. That traffic is the price of surviving a zone loss, so do not treat it as waste to be cut.
• Low volume. If a flow costs a few dollars a month, the engineering time to make the change, review it and monitor it will cost more than the savings.

Validating the change

Record a baseline first. Compare cross-AZ bytes from flow logs, filtered to the affected source and destination workloads, before and after the change. Then confirm that the inter-AZ line on your bill moved by the same amount. Also watch per-zone p99 request latency and CPU skew across Pods. If one zone's Pods run noticeably hotter than the others after the change, the routing is concentrating load.

Work flow by flow, starting from the measurement. Cross-zone spend is usually concentrated in a few high-volume pairs, so fixing those few and leaving the rest alone saves most of the money with far less risk and effort.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cross az data transfer cost, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.