What is driving cross-AZ data transfer cost in a Kubernetes cluster?

On AWS, traffic between Availability Zones is billed per GB, and both the sending and receiving side are charged. A byte that crosses a zone boundary is therefore paid for twice.

Kubernetes produces this traffic by default. A ClusterIP Service distributes connections across all ready endpoints, and kube-proxy does not consider which zone the caller or the endpoint is in. If replicas are spread evenly across N zones, about (N−1)/N of pod-to-pod requests leave the caller's zone. In a three-zone cluster, that is two out of three.

        Zone A            Zone B            Zone C
     +-----------+     +-----------+     +-----------+
     | client    |     |           |     |           |
     | pod       |     |           |     |           |
     |   |       |     |           |     |           |
     | kube-proxy|     |           |     |           |
     |  /  |  \  |     |           |     |           |
     |  |  |   \-+-$$--+-> api-2   |     |           |
     |  |   \----+-----+-----$$----+-----+-> api-3   |
     |  v        |     |           |     |           |
     | api-1     |     |           |     |           |
     +-----------+     +-----------+     +-----------+
        local          billed hop        billed hop

Service-to-Service calls are not the only source:

  • Cross-zone load balancing. A load balancer node in one zone forwards traffic to targets in other zones.
  • Stateful replication. Databases, Kafka, and Elasticsearch copy every write to replicas in other zones, which is the price of zone fault tolerance.
  • Endpoints in other zones. Pods that read from a database primary, cache, or broker leader located in a different zone generate cross-zone traffic.

Illustrative estimate (assumed numbers, not a benchmark).

Formula: monthly cost = service GB × (N−1)/N × per-GB rate × 2

Assumed inputs: 100,000 GB/month, 3 zones, $0.01/GB per direction.

Result: 100,000 × 2/3 × 0.01 × 2 ≈ $1,333/month, from Service routing alone.

        Zone A                      Zone B                      Zone C
+--------------------+      +--------------------+      +--------------------+
|                    |      |     client pod     |      |                    |
|                    |      |         |          |      |                    |
|                    |      |         v          |      |                    |
|                    |      |     kube-proxy     |      |                    |
|                    |      |         |          |      |                    |
|        api-1 <-----|--$$--|---------+----------|--$$--|-----> api-3        |
|                    |      |         |          |      |                    |
|                    |      |         v          |      |                    |
|                    |      |       api-2        |      |                    |
+--------------------+      +--------------------+      +--------------------+
  cross-AZ (billed)           same zone (free)            cross-AZ (billed)
  out of B, into A                                        out of B, into C

  $$ = zone boundary crossing, billed per GB on both sides

A client pod in Zone B sends Service traffic through kube-proxy, which ignores zones. With one replica per zone, about two of every three requests cross a zone boundary, and each of those hops is billed on both sides.

How do you measure inter-AZ traffic before changing anything?

Start with the provider's billing data. Filter the cost breakdown to inter-AZ or regional data transfer usage types to get the total you are paying. That figure tells you whether the problem is worth solving, but it does not tell you which workloads cause it. To find those, join flow logs, which record source and destination IP, bytes, and subnet or zone, with pod and node metadata. Per-pod network byte counters from the kubelet add the Kubernetes view.

Aim to attribute bytes to specific Service-to-Service paths. Cross-zone spend is usually concentrated: a few chatty paths carrying large payloads, such as API-to-cache, ingestion-to-queue, or replication traffic, tend to account for most of it.

Source Answers Limitation
Billing breakdown Total inter-AZ spend No workload attribution
Flow logs Which IPs and zones exchange bytes Pod IPs churn, so you must enrich them with pod metadata
Per-pod metrics Bytes per pod over time No destination zone

Illustrative estimate. Assume 3 zones, one service pair moving 20 TB (20,000 GB) per month, and $0.01/GB charged in each direction:

20,000 × 2/3 × 0.01 × 2 ≈ $267/month

If routing keeps 90% of that traffic in-zone, only 10% of it still crosses zones:

20,000 × 0.10 × 0.01 × 2 = $40/month

These figures are illustrative estimates, not measurements. Use current pricing for your region.

Record this baseline before you change anything. Traffic shifts from releases, seasonality, or retired services can move the bill without any routing change. Without a baseline, you may credit those unrelated changes to routing and misjudge whether the engineering effort paid off.

How does topology-aware routing work?

In short, it works by labelling each endpoint with the zone or zones that should use it, and having each node route only to the endpoints labelled for its own zone. Requests from a pod in us-east-1a then go to backends in us-east-1a whenever the control plane judges that safe, and fall back to cluster-wide routing when it does not. No application code changes are involved. The decision happens entirely in the Service dataplane.

Topology-aware routing has two parts. The EndpointSlice controller in the control plane writes zone hints, and kube-proxy on each node reads them.

The controller allocates endpoints by zone capacity

For a Service with topology-aware routing enabled, the EndpointSlice controller assigns each endpoint to a zone using a hint. It sets the number of endpoints per zone in proportion to that zone's allocatable CPU across its nodes. If one zone holds half the cluster's allocatable CPU, it should receive about half the endpoints. The goal is load balance, not just locality. A zone with more nodes probably originates more traffic, so it needs more backends to absorb that traffic without overloading them.

In an EndpointSlice, look for hints.forZones on each entry under endpoints. It holds a list of zone names, for example name: us-east-1a. Each endpoint also has a zone field that shows where it actually runs. The two values usually match. They differ when the controller lends an endpoint to a zone that has too few local endpoints.

kube-proxy filters by the node's zone

kube-proxy reads the node's topology.kubernetes.io/zone label. It programs only the endpoints whose hints include that zone, so traffic from the node stays in its zone.

Hints are conditional

The controller writes hints only when it considers the allocation safe. It removes them in these cases:

  • There are too few endpoints for a balanced split.
  • Zone labels are missing.
  • The expected per-zone load drifts past an internal overload threshold.

If any endpoint in the Service lacks a hint, kube-proxy falls back to routing across all endpoints. Routing can therefore revert to cross-zone without any configuration change.

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: checkout-7x2kq
  labels:
    kubernetes.io/service-name: checkout
    endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
addressType: IPv4
ports:
  - name: http
    protocol: TCP
    port: 8080
endpoints:
  - addresses: ["10.0.1.15"]
    conditions:
      ready: true
    zone: us-east-1a          # where the Pod actually runs
    hints:
      forZones:
        - name: us-east-1a    # kube-proxy on us-east-1a nodes will use this endpoint
  - addresses: ["10.0.2.23"]
    conditions:
      ready: true
    zone: us-east-1b
    hints:
      forZones:
        - name: us-east-1b
  - addresses: ["10.0.1.31"]
    conditions:
      ready: true
    zone: us-east-1a          # runs in 1a...
    hints:
      forZones:
        - name: us-east-1c    # ...but lent to 1c, which has too few local endpoints

An EndpointSlice excerpt showing per-endpoint zone and hints.forZones fields, including one endpoint the controller has lent to a zone other than the one it runs in.

How do you enable topology-aware routing and confirm it is active?

Opting in per Service

Topology-aware routing is opt-in per Service. Older clusters use the service.kubernetes.io/topology-mode: Auto annotation. Newer releases add a trafficDistribution field in the Service spec, which accepts PreferClose. The two mechanisms assign hints differently, and their names, defaults and maturity have changed across releases. Check the documentation for your exact cluster version and for your managed offering before you choose one, because some providers lag upstream or expose only certain features.

Prerequisites that decide whether hints appear

  • Even replica placement. Use topology spread constraints on topology.kubernetes.io/zone. Skewed placement makes the controller withhold hints rather than overload one zone.
  • Enough replicas per zone. With too few endpoints, the controller cannot allocate them proportionally, and it gives up.
  • Accurate zone labels. Every node needs a correct zone label. A missing or wrong label on even one node can disable hints for Services with endpoints on that node.
  • Capacity that matches demand. Zone CPU capacity should roughly track where client traffic originates. If 60% of callers sit in one zone but only a third of the capacity does, local routing concentrates load on that zone's pods.

Verifying it is active

List the Service's EndpointSlices with the label selector kubernetes.io/service-name=<name> and inspect each endpoint for a hints.forZones entry. If any endpoint lacks a hint, the Service is not using zone-local routing. Then describe the Service and read its events. The EndpointSlice controller records events when it enables or withholds hints, and those events usually say why, for example insufficient endpoints or unbalanced allocation.

Alert on those disable events, or on EndpointSlices that lose their hints. Hint removal causes no errors. It only increases the cross-AZ transfer bill. An alert lets the team find the change when it happens instead of tracing it back from an invoice weeks later.

#!/usr/bin/env bash
set -euo pipefail

NS=shop
SVC=checkout-api

kubectl -n "$NS" patch service "$SVC" --type merge \
  -p '{"spec":{"trafficDistribution":"PreferClose"}}'
kubectl get nodes -L topology.kubernetes.io/zone

kubectl -n "$NS" get endpointslices \
  -l kubernetes.io/service-name="$SVC" \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\t"}{.zone}{"\t"}{.hints.forZones[*].name}{"\n"}{end}'

kubectl -n "$NS" get events \
  --field-selector involvedObject.kind=Service,involvedObject.name="$SVC" \
  --sort-by=.lastTimestamp

Opts a Service into topology-aware routing, then confirms that nodes carry zone labels, checks each endpoint in the EndpointSlices for zone hints, and reads the controller events that explain why hints were added or withheld.

Who should use topology-aware routing?

Topology-aware routing is aimed at platform and infrastructure teams that run multi-zone Kubernetes clusters on a cloud that bills per GB for traffic between zones. It fits best when most of the following are true:

  • Inter-AZ transfer is a visible line item. The billing breakdown shows meaningful regional data transfer spend, and flow logs attribute much of it to in-cluster Service-to-Service traffic rather than to replication or external load balancers.
  • A few high-volume paths dominate. Chatty internal APIs, caches, or ingestion pipelines carry large payloads between services that both run in every zone.
  • Backends are stateless and horizontally scaled. Each Service has several replicas in every zone, and any replica can serve any request.
  • Traffic goes through kube-proxy or a hint-aware dataplane. Callers reach the backend through a ClusterIP Service, not through client-side balancing over pod IPs or headless DNS.
  • Callers are spread roughly in line with capacity. Client pods are distributed across zones in about the same proportion as allocatable CPU.

Teams with single-zone clusters, low inter-AZ volume, small replica counts, or traffic that bypasses kube-proxy get little from it. The next section covers those cases in more detail.

When should you avoid topology-aware routing?

Topology-aware routing trades load balancing quality for lower transfer cost. That trade fails in several common situations.

Concentrated callers create hotspots. Without hints, kube-proxy spreads requests across every endpoint, which hides skew in where the callers run. With hints, a zone that hosts most of the callers sends all of that traffic to its local share of endpoints. Those endpoints can saturate while replicas in other zones sit idle.

Partial zone failures hit harder. When a zone loses some of its endpoints, the remaining local endpoints absorb that zone's traffic until the controller recalculates or drops the hints. A brownout, where pods are degraded but still pass readiness checks, never triggers redistribution at all.

Hints are fragile during change. The EndpointSlice controller removes hints when the endpoints in a zone fall too far below that zone's proportional share of capacity. HPA scale events, uneven node pools and cluster autoscaler placement can all cross that threshold. As a result, hints drop out and come back during rollouts, and cost becomes hard to predict.

Situation Why it hurts Lean toward
Fewer replicas than roughly 2–3 per zone One pod loss skews proportions Skip
Callers concentrated in one zone Local endpoints overload Skip or rebalance callers
Frequent autoscaling or rollouts Hints flap Measure first
Low inter-AZ volume Savings are small Skip

Balanced capacity costs money. Keeping enough replicas in every zone to satisfy the controller can add idle compute. For low-traffic services, that extra compute can cost more than the cross-AZ transfer you avoid.

What traffic does topology-aware routing not cover, and what are the alternatives?

Topology-aware routing only changes how kube-proxy, or a dataplane that reads EndpointSlice hints, chooses an endpoint for ClusterIP traffic. Any traffic that never consults those hints is unaffected:

  • Ingress and external load balancers. A cloud load balancer with cross-zone balancing enabled spreads requests across nodes in every zone before kube-proxy sees them. You have to configure zone locality on the load balancer or ingress controller separately.
  • Client-side load balancing. gRPC clients that resolve pod IPs and balance across them hold long-lived connections straight to pods. kube-proxy is not in that path, so hints have no effect.
  • Headless Services. DNS returns every pod IP regardless of zone. The client picks one, usually without knowing which zone it is in.
  • ExternalName. This is only a DNS CNAME. No endpoints exist, so there is nothing to hint.
  • Data-layer replication. Database replicas, Kafka followers and cache clusters copy data across zones by design. Locality is controlled by the system's own replica placement and read-preference settings.
Option Best for Trade-off
internalTrafficPolicy: Local Node-local agents (log shippers, metrics collectors) Traffic is dropped if no endpoint runs on the node
Mesh or client-library zone awareness gRPC, headless, long-lived connections More configuration to maintain; failover behaviour must be tested
Pod affinity Tightly coupled, chatty service pairs Constrains scheduling and can concentrate a zone failure
Accept the cost Low-volume paths None, if the measured traffic stays small

Use the measurements from earlier to rank paths by bytes transferred, and spend effort on the top few. For the rest, the engineering time usually costs more than the transfer charges it would remove.

How can topology-aware routing save engineering time?

The main time saving comes from using a built-in Kubernetes feature instead of building zone awareness yourself.

  • No application changes. Locality is handled in the Service dataplane. Teams do not have to add zone discovery, custom resolvers, or locality-aware client libraries to each service, and they do not have to coordinate those changes across languages and repositories.
  • No extra infrastructure to operate. For ClusterIP traffic that already goes through kube-proxy, you get zone-local routing without deploying and upgrading a service mesh or sidecars just for this purpose.
  • A one-line, reversible opt-in. Enabling it is a single field or annotation on the Service. Rolling back is removing it. That makes per-Service experiments cheap and keeps the blast radius of a mistake to one Service.
  • Automatic rebalancing and fallback. The EndpointSlice controller recomputes hints as nodes and endpoints change, and falls back to cluster-wide routing when locality would be unsafe. Without it, engineers would have to maintain per-zone Services or hand-tuned affinity rules and update them after every scaling or node pool change.
  • Debugging uses standard objects. EndpointSlices and Service events show what the controller decided and why, so troubleshooting relies on kubectl output the team already understands.

The time savings depend on keeping the scope small. Measuring first, enabling only on the dominant paths, and alerting on hint removal means the team spends effort where the transfer bill justifies it, instead of tuning every Service in the cluster.

Should your team adopt topology-aware routing? A decision checklist

Adopt it when a small number of Service-to-Service paths carry most of your inter-AZ bytes and the backends behind them can absorb uneven load. Work through the steps in order. Each one is cheap, and each can stop the rollout before it costs capacity.

  • Rank paths by bytes. Take the top three to five Service pairs from your flow data. Ignore the long tail.
  • Check zone balance. For each candidate, compare client traffic per zone against endpoint count per zone. If one zone sends most of the requests but holds a third of the pods, you will need extra replicas there. Price that compute before going further.
  • Enable per Service. Annotate only the candidate Services. If a single Service misbehaves, you can roll it back without affecting the rest of the cluster.
  • Verify hints. Confirm that the EndpointSlices carry zone hints. Without hints, kube-proxy routes the way it did before.
  • Alert on hint removal. The control plane drops hints when endpoints become too imbalanced across zones. This fails silently: traffic stays healthy and goes cross-zone again. Alert on missing hints or the related controller events.
  • Re-measure after a full billing cycle. Use billed transfer, not dashboard estimates, and compare it against the extra replicas you added.

Illustrative estimate (assumed inputs): one path moving 50 TB/month at an assumed $0.02/GB of combined inter-AZ charges costs about $1,000/month. If keeping it zonal requires two extra pods costing $150/month, the change is clearly worth making. A path moving 500 GB/month would save about $10, which does not pay for the alerting you would have to maintain. Concentrating on the dominant paths keeps the operational burden proportional to the savings.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cross az data transfer cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.