Service to Service Authentication: mTLS, JWTs and Token Exchange

Two identities: which workload is calling and on whose behalf

Every internal request carries two separate identity claims, and service to service authentication has to verify both. The first is workload identity: is the caller really the orders service, or something on the network pretending to be it? The second is delegated user identity: is orders acting for user U, and which permissions does U grant for this call? A request can be right on one question and wrong on the other. The billing service may genuinely be calling, yet have no right to read another tenant's invoices.

mTLS, JWT propagation and token exchange are not rival choices. Each answers a different part of the problem:

This matters most to backend and platform engineers who run internal APIs that serve user-scoped data. Most internal auth bugs come from treating one identity as if it covered the other. One common case is trusting mTLS alone, so any authenticated service can request any user's data. Another is forwarding a user's JWT unchanged, so a compromised downstream service can replay a broadly scoped token anywhere. A zero trust design checks both identities at every hop.

How mTLS establishes workload identity, and where it stops

In mutual TLS, both sides present a certificate during the handshake. The server proves its identity to the client as usual, then requests a client certificate and validates its chain against an internal CA rather than a public one. If the chain verifies and the client proves possession of the matching private key, the server knows which workload is on the other end. That knowledge comes from the certificate, not from an IP address or subnet. A SPIFFE-style identity places a URI such as spiffe://prod.example/ns/billing/sa/invoice-api in the Subject Alternative Name, and authorization keys off that string.

For example, after the handshake, a service handling /refunds reads the peer certificate, extracts the URI SAN, and checks it against an allowlist containing only the checkout service's identity. Any other workload gets a 403, even one holding a valid certificate from the same CA.

The handshake is the easy part. The hard part is lifecycle. A long-lived certificate is a password that is difficult to revoke. Short-lived certificates, issued automatically and rotated well before expiry, limit how long a stolen key remains useful and remove the need for revocation lists.

mTLS has clear limits. It authenticates the connection, not each request on it, and it carries no user context. When a sidecar or proxy terminates TLS, the application sees only the identity headers the proxy forwards. Any traffic that reaches the app directly, such as through a misconfigured port or a pod sharing the network namespace, can forge those headers.

Copyimport ssl

# Only checkout may call /refunds. A valid cert from our CA is necessary, not sufficient.
ALLOWED = {
    "/refunds": {"spiffe://prod.example/ns/checkout/sa/checkout-api"},
}

def make_server_context() -> ssl.SSLContext:
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    ctx.load_cert_chain("/run/certs/svid.pem", "/run/certs/svid-key.pem")  # rotated by an agent
    ctx.load_verify_locations(cafile="/run/certs/internal-ca.pem")      # internal CA, not public roots
    ctx.verify_mode = ssl.CERT_REQUIRED  # handshake fails without a valid client cert
    return ctx

def peer_spiffe_id(conn: ssl.SSLSocket) -> str | None:
    cert = conn.getpeercert()  # dict form; populated only for a verified peer
    uris = [v for (k, v) in cert.get("subjectAltName", ()) if k == "URI"]
    # Expect exactly one SPIFFE URI; anything else is ambiguous, so reject it.
    if len(uris) != 1 or not uris[0].startswith("spiffe://"):
        return None
    return uris[0]

def authorize(conn: ssl.SSLSocket, path: str) -> int:
    identity = peer_spiffe_id(conn)
    if identity is None or identity not in ALLOWED.get(path, set()):
        return 403
    # This authenticates the connection, not the request: every request on
    # this socket inherits the identity, and no end-user context is present.
    return 200

# If a sidecar terminates TLS instead, the app never sees the cert, only
# headers the proxy forwards. Those are trustworthy only if nothing else can
# reach the app's port; bind it to localhost and block direct ingress.

A Python server built on the standard ssl module requires client certificates from an internal CA, extracts the SPIFFE URI from the peer certificate's SAN, and checks it against a per-path allowlist.

JWT propagation and validating tokens correctly at every hop

The simplest way to carry user context downstream is to forward the caller's access token unchanged in the Authorization header. It needs no extra network calls, adds no new infrastructure, and every service sees the same subject and scopes the edge API saw.

The cost is reach. That token was issued for the edge API, so its aud names the edge API. Each internal service that accepts it becomes another place where the token is valid. If any one of those services is compromised, it holds a bearer credential it can replay against every other service in the mesh until the token expires.

Every hop must validate the token fully:

• Signature. Verify against the issuer's published JWKS. Cache the keys, and when a token carries an unknown kid, refetch, with a rate limit so forged kid values cannot hammer the issuer. This is how key rotation works without a restart.
• Algorithm. Pin the expected algorithm in configuration. Never trust the token's alg header, which opens the door to none and to HMAC-with-public-key confusion attacks.
• Issuer. Require an exact match on iss.
• Audience. Require that aud contains this service.
• Time. Check exp and nbf, allowing only a small clock skew.
• Scopes. Confirm the scopes cover the requested operation.

Skipping the aud check is the common failure. A service then accepts any token from the shared issuer, including one minted for an unrelated low-privilege app. It then acts on that token with its own privileges, which is a classic confused deputy.

Propagation is reasonable only with a few equally trusted services and strict audience discipline, meaning tokens that explicitly list every service allowed to accept them.

Copyimport jwt  # PyJWT, installed as pyjwt[crypto]
from jwt import PyJWKClient

ISSUER = ""
THIS_SERVICE = "orders-api"   # this hop's own identifier, not the edge API's
ALGORITHMS = ["RS256"]         # pinned in config; the token's alg header is never trusted

# One client per process. It caches the JWKS for `lifespan` seconds and refetches
# when a token has an unknown kid, so issuer key rotation needs no restart.
# Put a rate limit in front of this path (gateway or middleware) so forged kid
# values cannot drive repeated JWKS fetches.
jwks = PyJWKClient(f"{ISSUER}.well-known/jwks.json", cache_jwk_set=True, lifespan=300)


def authorize(token: str, required_scope: str) -> dict:
    # Select the public key by kid. Fails if the kid is unknown after a refetch.
    signing_key = jwks.get_signing_key_from_jwt(token)

    claims = jwt.decode(
        token,
        signing_key.key,
        algorithms=ALGORITHMS,   # blocks alg=none and HS256-with-public-key confusion
        issuer=ISSUER,           # exact match on iss
        audience=THIS_SERVICE,   # aud must include *this* service. Omitting this check
                                 # accepts any token from the shared issuer: confused deputy
        leeway=30,               # small clock skew allowance for exp and nbf
        options={"require": ["exp", "iss", "aud"]},  # nbf is checked when present
    )

    # Signature and audience prove who may hold the token, not what it permits.
    # Check that the granted scopes cover this specific operation.
    if required_scope not in claims.get("scope", "").split():
        raise PermissionError(f"missing scope {required_scope}")
    return claims


# Usage: wrap in try/except jwt.InvalidTokenError (and jwt.PyJWKClientError for
# key lookup failures) and return 401. Return 403 on PermissionError.
# claims = authorize(bearer_token, "orders:read")

A Python (PyJWT) validator that each service runs on a propagated token: it fetches and caches the issuer's JWKS with refetch for key rotation, pins the algorithm, and checks iss, aud for this specific service, exp/nbf with small skew, and the required scope.

OAuth 2.0 Token Exchange (RFC 8693) for scoped, re-minted user context

Token exchange avoids that requirement. Instead of forwarding the user's token, each service trades it for a new one that is valid only for the next hop.

The exchange

The calling service sends a POST request to the authorization server's token endpoint with these RFC 8693 parameters:

• grant_type=urn:ietf:params:oauth:grant-type:token-exchange
• subject_token: the token it received
• subject_token_type=urn:ietf:params:oauth:token-type:access_token
• audience=inventory-service
• scope=inventory:read

It can also send actor_token and actor_token_type to prove its own identity. The server applies policy and returns an access_token along with issued_token_type. Its claims might be sub: user-42, aud: inventory-service, scope: inventory:read, and act: {sub: orders-service}. The new token cannot be replayed against billing, and it cannot carry more scope than the original.

Delegation versus impersonation

With impersonation, the issued token has no act claim, so the downstream service sees only the user. This is simpler, but the audit trail loses the information about which workload made the call. With delegation, the act claim names the current actor. Earlier actors appear as nested act claims, which records the whole call chain. The RFC treats only the outermost actor as relevant for access decisions. The nested entries are informational.

Costs

• Every hop adds a network call to the request path. Caching tokens per user, audience and scope reduces this.
• The exchange endpoint becomes a runtime dependency with its own availability requirements.
• Policy becomes hard to reason about across chains of four or five actors.
• Support varies between authorization servers and client libraries. Check that yours handles actor_token and act before you design around them.

Copy# orders-service exchanges the user's incoming token for one scoped to inventory-service.
# Only RFC 8693 parameters are used. Client authentication method depends on your AS.
curl -s -X POST  \
  -u 'orders-service:REPLACE_WITH_CLIENT_SECRET' \
  --data-urlencode 'grant_type=urn:ietf:params:oauth:grant-type:token-exchange' \
  --data-urlencode "subject_token=${INCOMING_USER_TOKEN}" \
  --data-urlencode 'subject_token_type=urn:ietf:params:oauth:token-type:access_token' \
  --data-urlencode "actor_token=${ORDERS_WORKLOAD_JWT}" \
  --data-urlencode 'actor_token_type=urn:ietf:params:oauth:token-type:jwt' \
  --data-urlencode 'audience=inventory-service' \
  --data-urlencode 'scope=inventory:read'

# Example response (illustrative):
# {
#   "access_token": "eyJhbGciOi...",
#   "issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
#   "token_type": "Bearer",
#   "expires_in": 300,
#   "scope": "inventory:read"
# }
#
# Decoded claims of the issued token under delegation (illustrative):
# {
#   "sub": "user-42",              // still the end user
#   "aud": "inventory-service",    // rejected anywhere else, e.g. billing
#   "scope": "inventory:read",     // narrowed, never wider than the subject token
#   "act": {
#     "sub": "orders-service",     // current actor: used for access decisions
#     "act": { "sub": "api-gateway" }  // prior actor: informational only
#   }
# }
# Under impersonation the AS omits "act" and inventory-service sees only user-42.

An RFC 8693 token exchange in which orders-service presents the user's token and its own actor token and receives a short-lived token limited to inventory-service and inventory:read, with nested act claims recording the call chain.

A layered reference design with a three-service walkthrough

Every internal hop carries two credentials: an mTLS client certificate proving which workload is connecting, and a short-lived token stating which user the call is for. Authorization checks both together, for example "orders may call POST /charge, and only for users with scope payments:charge".

Copyclient --[external JWT, aud=api]--> gateway
  gateway: validate external token, then exchange it
  --[mTLS: gateway | token aud=orders, scope=orders:write]--> orders
  orders: exchange again, narrowing scope
  --[mTLS: orders | token aud=payments, scope=payments:charge, act=orders]--> payments

The external token stops at the gateway. Internal tokens are minted by an internal authorization server and signed with keys that no external party trusts. A leaked internal token is therefore useless outside the mesh, and a forged external token cannot be replayed inward. Each exchange does three things:

• It issues a token for exactly one audience.
• It translates public scopes into internal ones.
• It drops claims the next service doesn't need.

Hostnames, internal routes and upstream service names stay out of the claims, so a downstream service learns nothing about the topology above it.

Background jobs and event consumers have no live user token. The default is service-only identity: the consumer authenticates over mTLS and acts under its own narrow permissions, and the event payload records which user triggered it, for auditing. If the job must act as the user, for example to call a third-party API on their behalf, store a delegated credential that the user explicitly granted, scoped to that single purpose. Treat that stored grant as a secret that can be revoked, and exchange it for a fresh token at run time. Holding onto the original access token for later use is the wrong approach.

Copy client
   |  external JWT, aud=api (external issuer keys)
   v
+------------------------+   exchange   +----------------------------+
| gateway                |------------->| internal authz server      |
| validate ext token     |              | internal-only signing keys |
| ext token stops here   |      +------>| one aud per token,         |
+------------------------+      |       | scopes translated,         |
   | mTLS id=gateway             |       | claims trimmed             |
   | aud=orders                  |       +----------------------------+
   | scope=orders:write          | exchange             ^
   v                             |                      |
+------------------------+      |                      |
| orders                 |------+                      |
| checks caller + scope  |                             |
+------------------------+                             |
   | mTLS id=orders                                    |
   | aud=payments, act=orders                          |
   | scope=payments:charge, no hosts/routes            |
   v                                                   |
+------------------------+   mTLS id=worker            |
| payments               |<--------------+             |
| orders may POST        |               |             |
| /charge only with      |   +-----------+----------+  |
| payments:charge        |   | job / event consumer |  |
+------------------------+   | own mTLS identity;   |--+
                             | user id in payload;  |
                             | optional stored grant|
                             | exchanged at run time|
                             +----------------------+

Each hop pairs an mTLS workload identity with an internally signed, single-audience token; the external token stops at the gateway, and background jobs use their own identity or exchange a stored, revocable grant.

Alternatives, trade-offs and when to avoid each approach

Self-contained JWTs are fast because each hop validates them locally. For the same reason, a stolen or revoked token stays valid until it expires. Introspection gives immediate revocation, but every protected call then depends on the authorization server being up. Most teams split the difference: short token lifetimes everywhere, and introspection only on high-risk operations.

Avoid propagation in large or mixed-trust estates. Every service that sees the token can replay it against any other service that accepts the same audience. Avoid exchange on low-sensitivity, high-volume paths, where the extra round trip costs more than the narrower scope protects. Avoid a self-managed CA until rotation is automated. Long-lived certificates issued by hand cause outages when they expire unnoticed, and they stay usable long after a compromise.

Calling an architecture zero trust does not make it zero trust. mTLS and validated JWTs only produce two verified identities. The gaps usually sit in the policy layer that must combine them, for example "service A may call this endpoint for user U". That is the layer most often left implicit.

Operations, cost and an incremental adoption path

that most incidents trace back to, so build it to be observable and cheap to run.

Running it

Accept a small clock-skew window on exp, nbf and iat, typically tens of seconds, and alert on node clock drift instead of widening the window. Exchanged tokens should outlive the exchange round trip by a large margin. Otherwise every call pays that latency. Cache exchanged tokens keyed by subject, actor, audience and scope, and evict them shortly before exp so a token never expires mid-request. Log the sub and the full act chain on every hop. Never log the token itself, because a logged bearer token can be replayed.

Cost

An internal CA is a production system that needs its own on-call. A failed rotation is especially dangerous because every certificate expires within the same window, so alert on remaining certificate lifetime, not only on rotation errors. A mesh also makes certificate issuance depend on its control plane. Short token lifetimes add issuance load.

Illustrative estimate (assumed numbers, not a benchmark): exchanges/s = R without a cache, and ≈ (U × A) / T_refresh with one. Assume R = 2,000 req/s, U = 5,000 active users, A = 1 audience and T_refresh = 240 s. That gives 2,000/s uncached and about 21/s cached.

The layered model gives you one policy shape, "workload X for subject Y", and one audit trail. During an incident, that shortens the work of answering who did what through which service.

Adoption path

1. Run mTLS in permissive mode and record who calls whom.
2. Enforce strict mTLS, then strict aud validation.
3. Add token exchange on the most sensitive call paths first.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to service to service authentication, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.