Where does Istio sidecar latency overhead come from on each hop?
In the sidecar model, every pod runs an Envoy proxy container alongside the application. iptables rules, installed by an init container or the Istio CNI plugin, redirect the pod's inbound and outbound TCP traffic through that proxy. Teams accept this because the proxy provides mutual TLS, uniform metrics and traces, and traffic policy (routing, timeouts, retries, authorization) without changes to application code.
The cost is that one logical call now passes through two proxies:
[app A] -> iptables -> [Envoy A, outbound] ==mTLS==> [Envoy B, inbound] -> loopback -> [app B]
Each traversal adds its own work:
- Redirection: packets make extra passes through the kernel network stack and loopback. Each proxy also terminates the connection and opens a new one, which adds copies between kernel and user space.
- mTLS: new connections require a TLS handshake. Pooled, reused connections spread that cost across many requests. Every byte is still encrypted on one side and decrypted on the other.
- HTTP parsing: both proxies parse request and response headers so they can route and observe the traffic. A plain TCP forwarder would not need to do this.
- Filter-chain work: telemetry, authorization policy evaluation, and retry and timeout bookkeeping run on every request. Each enabled filter adds CPU time directly on the request path.
Lifecycle ordering is a separate problem. If the app starts before Envoy is ready, its first outbound calls fail. If Envoy exits before the app finishes draining during shutdown, in-flight requests break. These failures appear as error spikes and as tail latency inflated by retries during deploys and scaling events. They are often mistaken for steady-state proxy overhead, so measure them separately.
Pod A (client)
+------------------------------------+
| app A |
| | plain HTTP to service B |
| v |
| iptables redirect | <- kernel passes, loopback copies
| v |
| Envoy A (outbound) | <- HTTP parse, routing, retries,
| | | timeouts, telemetry filters
+----|-------------------------------+
|
| mTLS, pooled connection <- TLS handshake on new connections;
| encrypt on A, decrypt on B
+----v-------------------------------+
| iptables redirect | <- kernel passes, loopback copies
| v |
| Envoy B (inbound) | <- HTTP parse, authz policy,
| | | telemetry filters
| v loopback |
| app B |
+------------------------------------+
Pod B (server)
Response returns the same way: app B -> Envoy B -> Envoy A -> app A.
Not per-hop cost: if Envoy starts after the app or exits before it
drains, requests fail or retry during deploys and scaling.
One logical call from app A to app B passes through iptables redirection and two Envoy proxies, and each step adds its own per-hop cost; sidecar startup and shutdown ordering is a separate source of failures.
Why does per-hop proxy overhead compound across a call graph?
That separation matters because steady-state overhead is not a fixed tax on the mesh. It scales with three things: how many requests cross a proxy, how many sequential hops a user request traverses, and which features (mTLS, telemetry, retries, policy) each proxy executes.
An illustrative estimate
Assume each service-to-service hop, covering both client and server sidecars, adds 0.5 ms at the median and 3 ms at p99. Assume hops are sequential and their tails are independent.
| Sequential depth | Added median latency | Chance at least one hop hits its p99 |
|---|---|---|
| 1 | 0.5 ms | 1.0% |
| 3 | 1.5 ms | ~3.0% |
| 6 | 3.0 ms | ~5.9% |
| 12 | 6.0 ms | ~11.4% |
Illustrative estimate, not measured.
At depth 6, about one request in seventeen hits at least one slow hop. The edge p99 is therefore driven by per-hop tail values, not medians. The same effect pushes p99.9 further, because it samples the worst proxy behaviour, such as GC pauses, connection setup, and CPU throttling. A 0.5 ms median that looks harmless on one hop can show up as several milliseconds at the edge tail.
Retries multiply load
Retries compound multiplicatively rather than adding. Suppose the mesh retries up to 3 attempts and the application client also makes 3 attempts. One logical call can then become 9 upstream requests. Across two layers that both behave this way, the worst case is 81.
During a partial outage, this amplification lands on the service that is already failing. Choose one layer to own retries, cap them with a retry budget, and keep per-try timeouts inside the caller's overall deadline.
How do you benchmark service mesh performance fairly?
Run the same workload twice on identical node types, changing only injection. Create two namespaces from the same manifests: label the test namespace istio-injection=enabled (or istio.io/rev=<revision> for revision-based installs) and leave the control namespace unlabeled. If you need a control group inside an injected namespace, set sidecar.istio.io/inject: "false" on the pod template. Pin both groups to the same node pool so CPU model and noisy neighbors do not decide the result.
A generic load outline:
- Warm up for a fixed period and discard those samples.
- Measure first-connection cost separately by forcing new connections, because the mTLS handshake is a one-time per-connection cost, not a per-request one.
- Run open-loop tests at several fixed request rates. A closed-loop client slows down when the server stalls, which hides queueing delay (coordinated omission).
- Run closed-loop tests only to find maximum throughput.
- Record p50, p99, and p99.9 per rate. Averages hide the tail, and the tail is where proxies add cost.
Published numbers rarely transfer to your environment. Proxy cost depends on payload size (buffering and copying), protocol (HTTP/1.1 vs HTTP/2 vs gRPC streams), connection reuse (handshakes amortized or not), and enabled features such as telemetry, authorization policies, and retries. The same factors vary between your own services, so a chatty gRPC service and a bulk-upload endpoint need separate measurements.
apiVersion: v1
kind: Namespace
metadata:
name: bench-test
labels:
istio-injection: enabled
---
apiVersion: v1
kind: Namespace
metadata:
name: bench-control
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo-nosidecar
namespace: bench-test
spec:
replicas: 2
selector:
matchLabels:
app: echo-nosidecar
template:
metadata:
labels:
app: echo-nosidecar
sidecar.istio.io/inject: "false" # skip injection for this pod only
spec:
# Pin every group to the same node pool so CPU model and neighbors match.
nodeSelector:
bench-pool: perf-a # placeholder label applied to the benchmark nodes
containers:
- name: echo
image: registry.example.com/echo:1.0 # same image in all groups
ports:
- containerPort: 8080
Namespace labels and a pod-level opt-out that build matching control and test groups, with all groups pinned to one node pool.
How does per-hop measurement work?
The A/B setup gives you a difference, not a per-hop number. To turn it into one, work through these steps:
- Subtract at matched percentiles and rates. For each fixed request rate, subtract the control group's p50 from the test group's p50, and do the same for p99 and p99.9. Do not subtract averages, and do not compare runs taken at different rates. Queueing changes the shape of the distribution as load rises.
- Divide by the number of proxy traversals. A single client-to-server call in the benchmark crosses one hop, meaning two sidecars. If your test service calls a second service, the measured difference covers two hops. Divide by the number of hops only for the median. Tails do not divide evenly, as the compounding section shows, so report tail overhead per call-graph depth rather than per hop.
- Cross-check with the proxies' own telemetry. Istio's standard request-duration metric is reported by both the source (client sidecar) and the destination (server sidecar) proxy. The destination-reported duration covers the server proxy plus the application. The source-reported duration also includes the network and the client proxy. The gap between the two approximates the cost of the client proxy, the wire, and inbound handling for that pair of workloads. Envoy access logs can provide a similar split, because they record total request duration alongside the time the upstream took to respond.
- Separate steady state from events. Tag samples taken during deploys, scale-ups, and connection churn. Report them apart from steady-state numbers, because they measure lifecycle ordering and handshake cost, not per-request proxy work.
- Repeat with features toggled. Run again with access logging off, with an authorization policy added, or with a different protocol. Each delta tells you what that feature costs on your traffic. That is the number you need when deciding which lever to pull.
The result is a small table per workload class: per-hop median overhead, tail overhead at your typical call depth, and proxy CPU per thousand requests. Those three numbers feed every decision later in this article.
How much Envoy sidecar CPU and memory does the mesh actually cost?
Latency is only half the bill. Every sidecar is a container with its own CPU and memory footprint, and the scheduler reserves capacity for it whether or not it is used.
| Resource | What drives it | How to observe |
|---|---|---|
| Envoy CPU | Request rate, TLS handshakes, telemetry and policy filters | kubectl top pod <pod> --containers (needs metrics-server) |
| Envoy memory | Connection count, plus clusters, routes and endpoints pushed to the proxy | Same command, over time |
| Reserved capacity | Sidecar requests, summed per node |
kubectl describe node <node> (Allocated resources) |
CPU tracks traffic. Memory tracks configuration. Without scoping, each proxy receives configuration for every service in the mesh, so memory grows with mesh size even on pods that call nothing. That is why an idle pod's sidecar can still be large.
Requests matter more than usage for cost. The scheduler bin-packs on requests, so an oversized sidecar request is added to every replica and can push pods onto extra nodes. Sidecar requests can be tuned per workload with the sidecar.istio.io/proxyCPU and sidecar.istio.io/proxyMemory annotations.
Illustrative estimate (assumed numbers, not a benchmark)
Formulas:
- Reserved = per-sidecar request × replicas
- Waste = (request − observed usage) × replicas
Assumed inputs:
- 400 pods
- 100m CPU and 128Mi memory requested per sidecar
- 20m CPU observed per sidecar
Results:
- Reserved CPU: 400 × 100m = 40 cores
- Reserved memory: 400 × 128Mi = 50Gi
- Used CPU: 400 × 20m = 8 cores
- Reserved but idle CPU: 32 cores
Price those 32 cores at your own node rate to see what right-sizing is worth.
#!/usr/bin/env bash
set -euo pipefail
NS="${1:-shop}" # namespace to inspect (placeholder)
NODE="${2:-}" # optional node name for reserved-capacity view
echo "== Observed usage per sidecar (point in time; sample repeatedly) =="
kubectl top pod -n "$NS" --containers | awk 'NR==1 || $2=="istio-proxy"'
echo
echo "== Requested resources per sidecar (what the scheduler reserves) =="
kubectl get pods -n "$NS" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[?(@.name=="istio-proxy")]}{.resources.requests.cpu}{"\t"}{.resources.requests.memory}{end}{"\n"}{end}' \
| column -t
echo
echo "== Sidecar count (the 'replicas' term in reserved = request x replicas) =="
kubectl get pods -n "$NS" -o jsonpath='{range .items[*]}{range .spec.containers[?(@.name=="istio-proxy")]}{.name}{"\n"}{end}{end}' \
| grep -c istio-proxy || true
if [[ -n "$NODE" ]]; then
echo
echo "== Requests vs allocatable on $NODE (all containers, incl. sidecars) =="
kubectl describe node "$NODE" | grep -A 8 'Allocated resources'
fi
A read-only kubectl script that lists observed istio-proxy CPU and memory next to each sidecar's resource requests, counts sidecars in a namespace, and optionally shows a node's allocated resources, giving the inputs for the reserved-versus-used estimate.
Which configuration levers reduce sidecar overhead?
Four levers recover most of that idle capacity. Each one trades away some convenience or visibility.
Scope configuration with the Sidecar resource
By default, istiod pushes every proxy the configuration for every service in the mesh. Envoy memory and config-push time grow with mesh size rather than with what a workload actually calls. A Sidecar resource whose egress.hosts list contains only ./* (the namespace itself), istio-system/*, and the specific namespaces a workload depends on shrinks that config. This lowers memory and makes pushes faster after each deployment. The cost is maintenance. When a team adds a new dependency and forgets to update the list, the calls fail. Consider outboundTrafficPolicy settings to make that failure obvious rather than silent.
Trim telemetry and access logs
Per-request metrics with high-cardinality labels and access logs written for every request consume proxy CPU on the hot path. Use the Telemetry API to drop unused metric labels or disable access logging for noisy, low-value workloads. Do not disable it everywhere. The workload you silenced is often the one you need during the next incident.
Size requests from observed usage
The sidecar.istio.io/proxyCPU and sidecar.istio.io/proxyMemory pod annotations override mesh defaults per workload. Set them from the proxy's peak usage, not its average. If you set CPU too low, the proxy gets throttled. That shows up as p99 spikes, which teams usually blame on the application first.
Remove unneeded filters
Every EnvoyFilter, Lua script, or Wasm extension runs on every request. Audit them, and delete the ones left over from experiments.
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
name: default # 'default' with no selector applies namespace-wide
namespace: checkout
spec:
egress:
- hosts:
- "./*" # services in this namespace
- "istio-system/*" # control plane and gateways
- "payments/*" # declared dependency; update when adding new ones
outboundTrafficPolicy:
mode: REGISTRY_ONLY # unlisted hosts fail visibly instead of passing through
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
namespace: checkout
spec:
selector:
matchLabels:
app: checkout-api
template:
metadata:
labels:
app: checkout-api
annotations:
sidecar.istio.io/proxyCPU: "200m" # too low causes throttling and p99 spikes
sidecar.istio.io/proxyMemory: "128Mi"
sidecar.istio.io/proxyCPULimit: "1000m" # headroom for bursts
sidecar.istio.io/proxyMemoryLimit: "256Mi"
spec:
containers:
- name: app
image: registry.example.com/checkout-api:1.4.2
A namespace-wide Sidecar resource that limits egress config to the local namespace, istio-system, and one dependency, plus pod annotations that set the proxy's CPU and memory requests and limits for a single workload.
How does Istio ambient mode change the cost model?
Ambient mode removes the proxy from the pod and splits its work in two. A per-node layer-4 proxy, ztunnel, handles mTLS, workload identity, and L4 authorization for every pod on that node. Layer-7 work such as HTTP routing, retries, header-based policy, and L7 telemetry moves to waypoint proxies. These are Envoy deployments you opt into per namespace or service account. Workloads without a waypoint never pay for HTTP parsing, which is a large part of the per-request cost a sidecar adds.
The savings come with changes to how failures and capacity work:
- Shared failure domain. One misbehaving ztunnel affects every pod on its node, not one pod.
- Noisy neighbors. A high-throughput pod can take proxy CPU away from its neighbors on the same node. Sidecars avoided this by charging each pod for its own proxy.
- Capacity planning moves to nodes. You size ztunnel against node-level traffic and scale waypoints like any other deployment, instead of adding proxy requests to every pod spec.
- Extra hop for L7. Traffic that needs L7 policy goes through a waypoint, which may run on a different node. Ambient is not automatically faster for those paths. Measure them with the method from the benchmarking section.
| Dimension | Sidecar | Ambient |
|---|---|---|
| L4 mTLS proxy | One per pod | One per node |
| L7 processing | Every pod, always | Only where a waypoint is enabled |
| Resource accounting | Per pod | Per node, plus waypoint deployments |
| Blast radius of a proxy fault | Single pod | All pods on a node, or a waypoint's scope |
| Extra network hop for L7 | No (the proxy is local) | Possible, through the waypoint |
| Upgrading the proxy | Restart the pods | Roll the node proxies and waypoints |
What are the alternatives to per-pod sidecars?
Ambient mode is one alternative. It is not the only one. The options below range from keeping Istio with less proxy work to removing the proxy from the path entirely:
- Selective injection. Keep sidecars, but disable injection on the specific workloads where measured overhead is too high. Use
sidecar.istio.io/inject: "false"on the pod template, or leave a namespace unlabeled. Those workloads lose mesh mTLS, telemetry, and policy, and peers that require strict mTLS will reject their traffic unless you add exceptions. - Ambient mode. Shared per-node L4 proxying, with L7 only where a waypoint is deployed, as described above.
- Proxyless gRPC. gRPC clients that consume xDS configuration directly can get some mesh routing and security features without a proxy in the data path. Istio supports this for a subset of features only, and it applies only to gRPC services, so check what your version supports before planning around it.
- Application libraries and no mesh. Implement mTLS, retries, timeouts, metrics, and tracing in each service's client and server libraries. Combine this with Kubernetes NetworkPolicy for L3/L4 segmentation. The proxy hop disappears, and the cost moves into code you maintain in every language you run.
- Plain Kubernetes networking for internal, low-risk traffic. Some internal traffic does not need encryption or L7 policy at all, for example batch jobs inside a single trusted namespace. Excluding it from the mesh is a legitimate choice if your security requirements allow it.
Use the per-hop numbers from your own benchmark to compare these options. Which alternative is cheaper depends on hop depth, request rate, and which mesh features each path actually uses.
When should you use sidecars, ambient mode, or no mesh at all?
The deciding ratio is proxy cost against the latency budget of the request, not proxy cost alone. A few milliseconds of mesh overhead is noise on a 300 ms endpoint that waits on a database. On a 5 ms internal lookup called twenty times per request, the same overhead can become most of the latency budget. High request rates shift the concern from latency to CPU, because proxy work grows with every request.
| Situation | Reasonable choice | Why |
|---|---|---|
| Latency-tolerant services, need L7 policy everywhere | Sidecars | Per-pod isolation; overhead is small relative to app time |
| Need mTLS and identity broadly, L7 only on some paths | Ambient mode | L4 cost is shared per node; waypoints only where needed |
| A few hot, fan-out-heavy paths | Selective injection or ambient opt-out | Remove proxies from the paths where overhead compounds |
| Small fleet, one language, simple traffic | No mesh | The control plane's operational cost outweighs the benefit |
Dropping the mesh does not remove its jobs. You move mTLS, metrics, tracing, retries, and timeouts into application libraries. This costs engineering time in every language you run. It also causes drift: services pin different library versions, retry budgets diverge, and certificate rotation depends on redeploys. Do the per-hop math before you accept that trade.
Decision checklist
- Measure p50 and p99 per-hop overhead and proxy CPU on your own workloads.
- Multiply by the hop count on your critical call paths and by your peak request rate.
- Compare the result with each path's latency SLO and your compute budget.
- Apply ambient mode, selective injection, or no mesh only where the numbers exceed those limits.
Who should measure per-hop sidecar cost?
- Platform teams running Istio in sidecar mode. They own the mesh defaults, the injected resource requests, and the capacity plan. Per-hop numbers tell them which defaults to change and what right-sizing is worth.
- Teams with deep or fan-out-heavy call graphs. If a user request crosses many sequential services, the compounding effect on p99 makes per-hop tail cost a first-order concern.
- Teams with tight latency SLOs on internal calls. A path with a single-digit-millisecond budget cannot absorb unmeasured proxy overhead.
- Teams evaluating a move to ambient mode or away from the mesh. A migration decision without before-and-after numbers from your own traffic is a guess.
- Anyone debugging unexplained p99 regressions. Measuring per-hop cost separates proxy overhead, CPU throttling of the sidecar, and lifecycle errors during deploys from genuine application slowness.
When should you avoid sidecars, or skip this measurement?
Avoid the sidecar model, or exclude specific workloads from it, in these cases:
- Latency-critical, fan-out-heavy paths where your own measurements show per-hop overhead consuming a large share of the latency budget.
- Very high request-rate services where proxy CPU per request, multiplied by peak traffic, is a significant share of the service's compute.
- Short-lived workloads such as batch Jobs. A long-running sidecar can keep a Job's pod from completing, and startup ordering can break the first outbound calls. Native sidecar support in recent Kubernetes and Istio versions addresses part of this. Confirm it is enabled in your cluster before relying on it.
- Small fleets with simple traffic, where operating a control plane costs more than the features it provides.
Skip or defer a detailed per-hop benchmark when:
- The overhead is clearly negligible. If every path is dominated by database or external API time and your SLOs have ample headroom, a quick A/B check is enough.
- You cannot build a fair control group. Benchmarks on shared, noisy nodes, or with different images or node types between groups, produce numbers that are worse than none because they look authoritative.
- You plan to publish or reuse the numbers as a general claim. Results apply to your payloads, protocols, and enabled features. They do not transfer to other environments.
How can measuring per-hop cost save engineering time?
- Faster incident triage. When you know the expected per-hop overhead at p50 and p99, a p99 regression can be checked against that baseline in minutes. This avoids days spent tuning application code for latency that the proxy, or a throttled proxy CPU limit, actually caused.
- Targeted fixes instead of mesh-wide changes. Feature-toggle runs show whether access logging, a policy, or a leftover EnvoyFilter accounts for the cost. You change one Telemetry or Sidecar resource instead of planning a migration.
- Scoped migrations. If only a few hot paths exceed their budget, you can opt those paths out or move them to ambient mode. The rest of the fleet keeps its current setup, which avoids rewriting and retesting every service.
- Fewer per-language reimplementations. Measurement often shows that the mesh is affordable on most paths. Keeping mTLS, retries, and telemetry in the proxy means teams do not maintain parallel library implementations in every language, or chase the version drift between them.
- Repeatable capacity decisions. The same scripts and A/B setup can be rerun after each Istio upgrade or configuration change. Sizing questions get answered from data, without starting a new investigation each time.
Illustrative estimate (assumed numbers, not measured): assume a team spends 3 engineer-days investigating each p99 regression without a baseline, sees 4 such regressions per quarter, and that a baseline cuts each investigation to 1 day. That saves 8 engineer-days per quarter. Compare that with the one-time cost of building the benchmark using your own figures.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to istio sidecar latency overhead, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.