What does a service mesh do for mTLS and retries, and how do Istio and Linkerd differ?
A service mesh moves network concerns out of application code and into a proxy layer the platform team controls. This article covers two of those concerns. The first is automatic mTLS: every workload gets a cryptographic identity, and traffic between workloads is encrypted and authenticated without anyone touching TLS libraries or certificates in the service. The second is safe retries: retry and timeout behavior is defined once and applied the same way everywhere, instead of being reimplemented differently in every client library.
That uniformity is the main reason teams adopt a mesh. Identity based on service accounts, rather than IP addresses, makes authorization policy meaningful. Consistent retries with timeouts reduce the chance of one team's aggressive client turning a slow dependency into an outage.
Istio emphasizes breadth and configurability. Linkerd emphasizes a smaller surface area and opinionated defaults.
| Aspect | Istio | Linkerd |
|---|---|---|
| Data-plane proxy | Envoy (C++) | linkerd2-proxy (Rust), purpose-built |
| Default mTLS | Permissive: accepts plaintext and mTLS, with strict mode opt-in | On by default between meshed pods |
| Retry model | Per-route policy with attempts, per-try timeout and retry conditions | Opt-in per route, with retry budgets to cap extra load |
| Authorization resources | AuthorizationPolicy, PeerAuthentication |
Server, AuthorizationPolicy, MeshTLSAuthentication |
| Sidecarless option | Ambient mode (ztunnel plus waypoint proxies) | Sidecar only |
| Debugging tools | istioctl analyze, istioctl proxy-config |
linkerd check, linkerd viz (tap, stat, top) |
Both projects are open source, but the way each distributes stable releases and the licensing terms for them differ and have changed over time. Check the current terms for each project before you assume both ship production-ready binaries the same way.
How does a service mesh work?
Both meshes split into two parts: a control plane that holds configuration and issues identities, and a data plane made of proxies that carry the actual traffic.
- Injection. When a pod is created in a namespace or with a workload marked for the mesh, a Kubernetes mutating admission webhook rewrites the pod spec to add the proxy container. Istio uses the
istio-injection=enablednamespace label or a revision label. Linkerd uses thelinkerd.io/inject: enabledannotation. Pods that already exist are not changed until they restart. - Traffic interception. An init container, or a CNI plugin if you install one, writes iptables rules in the pod's network namespace so inbound and outbound TCP traffic is redirected through the proxy. The application still connects to normal service DNS names and does not know the proxy is there.
- Configuration and discovery. The proxy connects to the control plane to learn about endpoints and policy. Istio's istiod pushes Envoy configuration over the xDS APIs. Linkerd's proxies query the destination and policy controllers for endpoints, service profiles and authorization rules.
- Identity. At startup, the proxy sends a certificate signing request to the control plane CA along with its ServiceAccount token, and receives a short-lived certificate that it renews automatically. The next section covers this in detail.
- Per-request handling. On an outbound call, the client-side proxy picks an endpoint, applies load balancing, timeouts and retries, and opens an mTLS connection to the server-side proxy. The server-side proxy verifies the client's certificate, applies authorization policy and forwards the request to the application over localhost. Both proxies emit metrics for every request, which is where the mesh's observability comes from.
Because retries happen in the client-side proxy and authorization happens in the server-side proxy, the same request passes through both policy layers without either application implementing them.
How does mTLS work in Istio and Linkerd?
Both meshes run a certificate authority in the control plane and hand each proxy a short-lived certificate tied to a Kubernetes ServiceAccount. In Linkerd, the identity component validates the proxy's ServiceAccount token and signs its certificate using an issuer certificate chained to a trust anchor. In Istio, istiod is the CA and issues SPIFFE-style identities (spiffe://<trust-domain>/ns/<ns>/sa/<sa>). Proxies renew their certificates before they expire, so a leaked key is only useful for a short time. Both meshes let you supply your own root or intermediate CA.
[ control plane CA: Linkerd identity / istiod ]
^ CSR + SA token ^ CSR + SA token
| signed cert | signed cert
Pod A [app -> proxy] ==mTLS==> [proxy -> app] Pod B
The defaults differ. Linkerd uses mTLS automatically between meshed TCP endpoints. Istio's PeerAuthentication starts in PERMISSIVE mode, which accepts both mTLS and plaintext. That means a misconfigured or unmeshed client can connect without encryption and nothing reports it. To close that path, apply a mesh-wide PeerAuthentication with mtls.mode: STRICT in the root namespace.
mTLS proves who the caller is and encrypts the connection, but it does not decide whether the caller is allowed in. For that, use Istio AuthorizationPolicy with principals, or Linkerd Server plus AuthorizationPolicy and MeshTLSAuthentication.
Gaps to plan for:
- Unmeshed workloads: traffic from workloads outside the mesh has no mesh identity.
- Egress: traffic leaving the cluster is not covered by mesh mTLS.
- CA rotation: expired or badly rotated trust anchors and issuers break every connection at once.
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: {name: default, namespace: istio-system}
spec:
mtls: {mode: STRICT}
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: {name: payments-allow-frontend, namespace: payments}
spec:
selector: {matchLabels: {app: payments}}
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/shop/sa/frontend"]
---
apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata: {name: payments-http, namespace: payments}
spec:
podSelector: {matchLabels: {app: payments}}
port: 8080
proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: frontend-identity, namespace: payments}
spec:
identities: ["frontend.shop.serviceaccount.identity.linkerd.cluster.local"]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: payments-allow-frontend, namespace: payments}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: payments-http}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: frontend-identity}
Enforcing strict mTLS in Istio and restricting the payments service to the frontend ServiceAccount identity, first with an Istio AuthorizationPolicy and then with Linkerd's Server, MeshTLSAuthentication and AuthorizationPolicy resources.
How do you configure safe retries in each mesh?
Istio
Retries live in the http route of a VirtualService. attempts sets the maximum number of retries, perTryTimeout bounds each attempt, and retryOn lists the triggering conditions, such as connect-failure, reset, 503 or gateway-error. The Gateway API HTTPRoute also defines retry fields. Check that your Istio release supports them before you rely on them.
Linkerd
Linkerd marks individual routes as retryable and caps retries with a retry budget. The budget is a ratio of retries to original requests over a time window, plus a minimum rate so low-traffic services can still retry. A fixed attempt count multiplies load exactly when a backend is failing. A budget does the opposite: during an incident, retries stay bounded to a fraction of real traffic, so they cannot turn into a retry storm.
What is safe to retry
Retry only idempotent requests. GET, HEAD, PUT and DELETE are idempotent by definition, and POST is safe only with an idempotency key. Retry connection failures, 502, 503 and 504. Do not retry 4xx responses.
Illustrative estimate. Assume the ingress makes 2 attempts, the sidecar makes 3, and the client library makes 3. One user request can become 2 × 3 × 3 = 18 backend calls. If the request is a POST that commits before it times out, you can get up to 17 duplicate writes.
Guidance
- Let one layer own retries. Usually that is the mesh for transport errors, or the client when only the client knows whether a request is idempotent.
- Make each outer timeout longer than the inner layer's attempts ×
perTryTimeout, so deadlines get shorter as the request moves inward.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: orders
namespace: shop
spec:
hosts:
- orders.shop.svc.cluster.local
http:
# Writes: no mesh retries. A POST that committed before timing out
# would be duplicated. Let the client retry with an idempotency key.
- match:
- method:
exact: POST
route:
- destination:
host: orders.shop.svc.cluster.local
timeout: 2s
retries:
attempts: 0
# Reads and other idempotent calls: retry transport failures and 503 only.
- route:
- destination:
host: orders.shop.svc.cluster.local
# Up to 1 original try + 2 retries at 1s each = 3s worst case,
# so the route timeout is set above that. Any caller's own timeout
# should be longer than 4s.
timeout: 4s
retries:
attempts: 2
perTryTimeout: 1s
retryOn: connect-failure,reset,503
An Istio VirtualService that turns off mesh retries for POST and allows a small number of bounded retries for other requests, with the route timeout sized to cover every attempt.
How much sidecar overhead does a service mesh add, and how do you measure it?
Every meshed pod runs an extra proxy container: Envoy for Istio, Linkerd's Rust-based micro-proxy for Linkerd. That proxy has its own CPU and memory requests, and each call passes through two proxies, one on the client side and one on the server side. Don't rely on vendor numbers. Run kubectl top pod --containers to see per-container usage, then compare p50 and p99 latency for the same load test with injection on and off. The p99 matters most, because proxy queuing and TLS handshakes show up in the tail.
Illustrative estimate (assumed inputs, not a benchmark): at 0.05 vCPU and 60 MiB per proxy across 400 pods, the mesh reserves 20 vCPU and about 23 GiB of memory before your applications do any work.
Envoy memory grows with the size of the service registry, because by default each sidecar receives configuration for every service in the mesh. Istio's Sidecar resource limits each workload to the hosts it actually calls. In large clusters, this is often the largest single reduction.
The proxy also changes the pod lifecycle:
- Startup: the init container rewrites iptables rules. If the application starts before the proxy is ready, its first connections fail.
- Jobs: the sidecar keeps running after the main container exits, so the Job never completes.
- Shutdown: if the proxy exits before the application drains, in-flight requests are dropped.
Kubernetes native sidecar containers and mesh-specific startup and shutdown settings address these problems. Each workaround is still something your team has to discover, test and maintain. Count that engineering time alongside the reserved compute when you calculate what the mesh costs.
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
name: checkout-egress
namespace: shop # placeholder namespace
spec:
# Apply only to pods with this label; omit workloadSelector to make
# this the namespace-wide default (one per namespace, usually named "default").
workloadSelector:
labels:
app: checkout
egress:
- hosts:
# Format is <namespace>/<dns name>.
- "./*" # services in the same namespace (shop)
- "istio-system/*" # mesh control plane and gateways
- "payments/payments-api.payments.svc.cluster.local" # one specific cross-namespace dependency
An Istio Sidecar resource that limits the checkout workload's Envoy configuration to the hosts it calls, with a kubectl command for comparing proxy memory before and after.
What are the alternatives to sidecar-based meshes?
Istio's ambient mode removes the per-pod sidecar. A per-node proxy, called ztunnel, intercepts traffic for every enrolled pod on that node and handles L4 work: mTLS, workload identity and L4 authorization. L7 features run in optional waypoint proxies, which are Envoy deployments you add per namespace or service account.
This changes the overhead math. Sidecar cost grows with pod count, because every replica reserves its own proxy CPU and memory. Ambient's L4 cost grows with node count, and waypoints grow with the number of services that need L7. The catch for this article's topic is that L7 retry policy requires a waypoint. ztunnel does not parse HTTP, so it cannot decide whether a request is safe to retry. If most of your services need retries, you deploy waypoints for most of them, and the savings shrink.
The failure domain also changes. A crashed or misconfigured ztunnel affects every meshed workload on its node, not one pod. Ambient tooling, debugging guides and community experience are also less mature than for sidecars, so plan for more time spent working out failures on your own.
| Approach | mTLS | Retries | Main trade-off |
|---|---|---|---|
| Sidecar mesh | Per pod | Per pod, L7 | Overhead scales with pods |
| Ambient (ztunnel only) | Per node | None at L7 | Node-wide failure domain |
| Ambient + waypoint | Per node | Waypoint, L7 | Extra hop and components |
| No mesh | App or ingress TLS | Client library | Consistency depends on each team |
Without a mesh, you can retry in client libraries with backoff and budgets, terminate TLS at ingress or in the application, and use Kubernetes NetworkPolicy for coarse access control. This is enough when you run few services, use one or two languages, and do not need identity-based authorization between services.
How hard is each mesh to operate and debug?
Istio's operational surface is much larger. A typical install uses VirtualService, DestinationRule, Gateway, ServiceEntry, Sidecar, PeerAuthentication, AuthorizationPolicy and sometimes EnvoyFilter. Linkerd uses fewer resources, mainly Server, HTTPRoute and authorization policies. Every resource type is one more thing a reviewer has to reason about and one more schema that can change during an upgrade.
Answering "why did this request fail?"
In Istio:
- Run
istioctl analyzeto catch misconfigurations such as references to missing hosts or conflicting rules. - Use
istioctl proxy-config routes,clustersandendpointsagainst the client pod to see what Envoy actually received. - Check the Envoy access log response flags, for example
NR(no route) orUF(upstream connection failure).
In Linkerd:
- Run
linkerd checkto confirm control plane and proxy health. - Use
linkerd viz statandlinkerd viz tapto watch live requests and their status codes between specific workloads.
Istio's configuration risks
- Overlapping VirtualServices. Two VirtualServices that bind the same host can conflict, and only one may take effect.
- Permissive mTLS left on. PERMISSIVE mode is often enabled during migration and never switched to STRICT, so plaintext traffic is still accepted.
- Policies that fail open. If no ALLOW AuthorizationPolicy applies to a workload, all traffic is allowed. A policy with a mistyped selector matches nothing, which also leaves the workload open.
Linkerd's limits
Linkerd offers fewer controls and has no equivalent of Envoy filters. Its header-based routing is narrower, and it ships no built-in ingress. You need a separate ingress controller.
Why less configuration saves time
A smaller configuration surface makes reviews shorter, narrows the set of suspects during an incident, and reduces the number of resources to re-validate at each upgrade. Istio's revision-based canary upgrades are safe, but they involve more steps than Linkerd's.
#!/usr/bin/env bash
set -euo pipefail
NS=shop
CLIENT_POD=$(kubectl get pod -n "$NS" -l app=web -o jsonpath='{.items[0].metadata.name}')
istioctl analyze -n "$NS"
istioctl proxy-config routes "$CLIENT_POD" -n "$NS"
istioctl proxy-config clusters "$CLIENT_POD" -n "$NS" | grep api
istioctl proxy-config endpoints "$CLIENT_POD" -n "$NS" | grep api
kubectl logs "$CLIENT_POD" -n "$NS" -c istio-proxy --tail=200 | grep -E ' (NR|UF) '
kubectl get peerauthentication -A -o yaml | grep -n 'mode:'
kubectl get authorizationpolicy -n "$NS"
linkerd check
linkerd check --proxy -n "$NS"
linkerd viz stat deploy -n "$NS"
linkerd viz tap deploy/web -n "$NS" --to deploy/api
A side-by-side triage script showing the commands used to answer "why did this request fail?" in Istio (analyze, proxy-config, access-log flags, mTLS and policy checks) versus Linkerd (check, viz stat, viz tap).
Who should use a service mesh?
A mesh pays off for teams that have the problem it solves and the capacity to run it:
- Platform teams serving many services. When dozens of services are owned by different teams, a mesh gives one place to enforce encryption, identity and retry behavior instead of auditing each codebase.
- Polyglot organizations. If services are written in several languages, keeping TLS setup, timeouts and retry logic consistent across client libraries is hard. A proxy applies the same behavior regardless of language.
- Teams with encryption-in-transit or zero-trust requirements. If you must show that every service-to-service call is encrypted and authorized by workload identity rather than network location, mesh mTLS plus authorization policy gives you that with a small amount of declarative configuration.
- Teams that need per-request observability without code changes. Both meshes report success rate, request volume and latency per workload from the proxies.
- Organizations with a clear owner. Someone has to own CA and trust-anchor rotation, control plane upgrades and proxy version drift. A mesh suits teams that can assign that work.
Within that group, Linkerd suits teams whose needs are mostly mTLS, identity-based authorization and bounded retries. Istio suits teams that also need advanced traffic management, Envoy extensibility or ambient mode.
When should you avoid a service mesh?
Avoid a mesh, or postpone it, when its costs outweigh the consistency it buys:
- You run only a few services. Library-level TLS, client retries with backoff and NetworkPolicy cover the need with fewer moving parts.
- No one can own it. An unmaintained mesh with stale proxies or a trust anchor nearing expiry is a larger risk than having no mesh.
- Workloads are mostly Jobs, CronJobs or batch. Sidecar lifecycle issues, such as a proxy that keeps a Job from completing, add work for little benefit unless you use native sidecar containers or ambient mode and have tested them.
- Latency budgets are very tight. Each call passes through two extra proxies. If your p99 target leaves no room for that, measure first and keep the hot path out of the mesh if needed.
- Traffic is mostly non-HTTP or unusual protocols. Protocols where the server sends the first bytes, or that the proxy cannot detect, need explicit per-port configuration in both meshes. If most of your traffic is like this, you get mTLS but few of the L7 features that justify the operational cost.
- Workloads cannot be meshed. Pods using
hostNetwork, or workloads running outside Kubernetes, do not get sidecar injection in the usual way, so the mesh leaves gaps you still have to cover another way. - Your retries depend on application knowledge. If only the application knows whether a call is idempotent, the mesh cannot safely own retries, and you lose one of its main benefits.
When should you choose Linkerd, Istio, or neither?
Pick the mesh that fits the problem you have now. Feature lists matter less than this.
| Need | Linkerd | Istio | Neither |
|---|---|---|---|
| mTLS on by default with almost no config | Strong fit | Possible, more setup | — |
| Budgeted retries that limit retry storms | Strong fit | Needs per-route tuning | — |
| Weighted routing, fault injection, header matching | Limited | Strong fit | — |
| Custom Envoy filters or extensions | No | Strong fit | — |
| Multi-cluster, ingress and egress in one control plane | Basic | Strong fit | — |
| Few services, mostly Jobs or batch | Poor fit | Poor fit | Strong fit |
Choose Linkerd when your main needs are encrypted, identity-bound traffic and retries that cannot multiply load during an outage. Less configuration gives you fewer ways to misconfigure the mesh, and your team spends less time on it.
Choose Istio when traffic policy is a product requirement, such as canary splits, egress allowlists or cross-cluster failover. Envoy's extensibility justifies the extra operational work only if you use it.
Skip the mesh in three cases:
- Small service count: a handful of services can use library-level TLS and client retries.
- Job and batch workloads: these struggle with sidecar lifecycle, because a sidecar can keep a Job from completing.
- No owner for the mesh: if no team can own CA rotation and regular upgrades, an unmaintained mesh becomes a security liability.
Roll out in stages:
- Mesh one namespace.
- Enforce strict mTLS there.
- Add authorization policies.
- Move retry ownership from application code into the mesh. Doing this last prevents client retries and mesh retries from stacking on each other.
Before committing to production, check the current licensing terms and how stable releases are distributed for whichever mesh you choose. Both have changed over time.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to istio vs linkerd, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.