What problem does service to service authentication actually solve?
Every internal request carries two separate claims, and service to service authentication has to verify both.
The first is which workload is calling. When the order service calls the payment service, payment needs proof that the caller really is the order service and not a compromised pod or a script with a copied token. This is service identity, established with mTLS certificates or platform-issued workload credentials, and it is bound to the connection or the process.
The second is on whose behalf. The order service is acting for a specific user or tenant, and payment's authorization decision depends on that subject. Should this user be able to refund this order? That context comes from the end user's authentication and has nothing to do with which pod sent the bytes.
Designs that merge these two lose one of them. If you forward only the user's JWT, any workload holding that token looks identical to a legitimate caller. A stolen bearer token works from anywhere, and payment cannot tell its trusted caller apart from an attacker. If you rely only on mTLS, payment knows the order service is calling but has to trust it blindly about the user. The result is a confused deputy: any bug in the order service becomes a way to act as any user.
Neither JWT propagation nor token exchange replaces workload identity. Both are ways to carry the second claim. The first still needs its own mechanism underneath, which is what makes a zero trust posture possible.
How does JWT propagation work, and why is it the default?
With JWT propagation, the gateway validates the user's access token and forwards it unchanged. Each service reads it from the incoming Authorization header, validates it, and attaches the same header to every outbound call. The token that left the identity provider is the token that reaches the deepest service in the call graph.
Teams default to this because it is cheap. There is no extra network call per hop and no new component to run. Every service gets the full user context (subject, tenant, roles) without a lookup. Implementation is usually a middleware or HTTP client interceptor that copies one header, so the development cost is close to zero.
The trouble starts with the aud claim. A token minted for the public API has to be accepted by the orders service, the billing service and everything behind them. Teams respond in one of two ways. They issue tokens with a long list of audiences, or they configure services to skip audience validation. Either way, every internal service accepts the token.
The result is that any single service is enough to compromise the rest. If one service is breached, or writes request headers to logs that many people can read, an attacker can replay the token against any other service. That token carries the full scope and remaining lifetime of the edge token.
Propagation still fits small systems and single trust domains. It holds up when services authenticate each other through strong workload identity such as mTLS, and when each service enforces a narrow, explicit audience.
How does OAuth token exchange (RFC 8693) work internally?
In token exchange, the calling service sends the token it received to the authorization server's token endpoint and asks for a new one. The request uses the grant type urn:ietf:params:oauth:grant-type:token-exchange. The subject_token is the user's incoming token, and it identifies whose authority is being used. The optional actor_token is the calling service's own credential, and it identifies who is acting. The audience or resource parameter names the single downstream service, and scope requests only what that service needs. The authorization server applies policy and can refuse, narrow scope, or shorten lifetime.
How you use the actor determines the semantics. With impersonation, the new token simply represents the user, and downstream services cannot tell that an intermediary was involved. With delegation, the token keeps the user as sub and adds an act claim naming the calling service. Nested act claims record the earlier hops.
A typical downscoped result has these properties:
audis set toinventory-serviceonly.scopeisinventory:read.- It expires in a few minutes.
- It carries
sub,iss,exp,jti, andact: { sub: "order-service" }. - It does not carry the user's email, roles, or other audiences.
This limits blast radius in a zero trust model. If the token is stolen from inventory-service or from its logs, it is useless against any other service and expires quickly.
Long act chains are a practical problem. Policies that reason about the order of five nested actors are difficult to write and audit. RFC 8693 treats only the outermost act as the current actor and describes nested actors as informational. Most teams follow that approach and authorize on the user plus the immediate caller.
# order-service exchanges the user's incoming token for one scoped to inventory-service.
# Client authentication (-u) identifies order-service to the authorization server.
curl -s \
-u 'order-service:PLACEHOLDER_CLIENT_SECRET' \
--data-urlencode 'grant_type=urn:ietf:params:oauth:grant-type:token-exchange' \
--data-urlencode "subject_token=${USER_ACCESS_TOKEN}" \
--data-urlencode 'subject_token_type=urn:ietf:params:oauth:token-type:access_token' \
--data-urlencode "actor_token=${ORDER_SERVICE_TOKEN}" \
--data-urlencode 'actor_token_type=urn:ietf:params:oauth:token-type:jwt' \
--data-urlencode 'audience=inventory-service' \
--data-urlencode 'scope=inventory:read'
# subject_token: whose authority is used (the user)
# actor_token: who is acting (order-service); its presence lets the server issue a delegation token
# audience/scope: ask only for what the single downstream call needs
# Response (abridged):
# { "access_token": "eyJ...", "issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
# "token_type": "Bearer", "expires_in": 300, "scope": "inventory:read" }
# Decoded payload of the new access_token:
# {
# "iss": "",
# "sub": "user-8f3a2c", <- still the user (delegation, not impersonation)
# "aud": "inventory-service", <- rejected by every other service
# "scope": "inventory:read", <- narrowed from the user's original scopes
# "iat": 1762520400,
# "exp": 1762520700, <- 5-minute lifetime
# "jti": "b7e1d9a4-2f60-4c3e-9a1b-0d5c7e8f1234",
# "act": {
# "sub": "order-service", <- current actor: authorize on this
# "act": { "sub": "api-gateway" } <- earlier hop: informational only
# }
# }
# No email, roles, or extra audiences are copied from the original token.
An RFC 8693 token exchange request from order-service and the decoded downscoped token it returns: a single audience, a narrow scope, a five-minute lifetime, minimal claims and a nested act claim recording the delegation chain.
Where should you exchange tokens: edge, trust boundaries or every hop?
Exchanging at every hop gives the tightest audience binding, but it puts the token service on the critical path of every internal call and multiplies the latency and availability cost covered in section 6. Pure propagation avoids that cost, but a single token with a broad audience then works against every service it reaches. A hybrid balances the two. Exchange once at the gateway, exchange again wherever a request crosses a trust boundary, and propagate inside a domain a token whose audience is limited to that domain.
client --[user token, aud=api]--> gateway
(exchange: aud=orders-domain)
gateway ==mTLS: spiffe id=gateway==> orders
orders ==mTLS: id=orders==> inventory (propagate, aud=orders-domain)
orders -- boundary: exchange aud=payments, act=orders --
orders ==mTLS: id=orders==> payments (aud=payments)
The JWT says on whose behalf the call runs and where it may be used. mTLS says which workload is calling. Every hop checks both.
Draw a boundary where one or more of these hold:
- Data sensitivity. Payments, credentials or regulated personal data should never accept a token that was minted for a less sensitive domain.
- Team ownership. If another team runs the receiving service, that team should own its audience and its scopes.
- External exposure. Partner-facing services and services reachable through a different ingress need their own audience.
- Blast radius. Ask what a stolen token would unlock. If the answer covers more than one domain, add a boundary.
Avoid adding exchanges between services that one team deploys together and that share a database. They are not a real trust boundary, so the round trip adds latency without adding isolation.
client
| user token, aud=api
v
+---------+ exchange: aud=orders-domain +---------------+
| gateway |<----------------------------->| token service |
+---------+ +---------------+
| mTLS id=gateway ^
| aud=orders-domain |
+---|---------- orders domain -------------+ |
| v | |
| +--------+ mTLS id=orders +-----------+ | |
| | orders |---------------->| inventory | | |
| +--------+ propagate: +-----------+ | |
| | aud=orders-domain | |
+-----|------------------------------------+ |
| exchange: aud=payments, act=orders |
|<------------- trust boundary ------------>+
+-----|------------ payments domain -------+
| v mTLS id=orders, aud=payments |
| +----------+ |
| | payments | accepts only aud=payments |
| +----------+ |
+------------------------------------------+
The gateway exchanges the user token for an orders-domain audience that is propagated inside the domain, orders exchanges again at the payments trust boundary, and every hop also carries the caller's mTLS workload identity.
What validation must every service do regardless of approach?
Whether a token arrived by propagation or exchange, the receiving service must assume it could be forged, replayed, or meant for someone else. Each service should run these checks on every request:
- Verify the signature against a cached JWKS. Fetch the issuer's public keys, cache them, and refresh when you see an unknown
kid(key ID). Rate-limit those refreshes so an attacker cannot make you flood the identity provider by sending random key IDs. - Pin the algorithm. Keep an allowlist, such as
RS256orES256, and reject everything else, includingnone. If you trust the token's ownalgheader, an attacker can choose a weaker algorithm or turn your public key into an HMAC secret. - Check the issuer exactly. Compare
issagainst a configured value, not a substring or prefix. - Check the audience.
audmust name this service. This check gives exchanged tokens their value. Without it, a token minted for the billing service also works against the user service. - Enforce expiry with bounded skew. Validate
expandnbf, and allow a small fixed tolerance, typically tens of seconds rather than minutes. - Ignore unverified identity headers. A header like
X-User-Idthat a gateway supposedly set is just a string that any pod on the network can send. Under zero trust, identity comes from a verified token or not at all.
The example below runs these checks in order, and checking the signature first means you never act on claims you have not verified. Each check is simple, but subtle mistakes are common. Put the logic in one shared library or sidecar, test it once, and patch it in one place. That is cheaper than having every team reimplement it and get a different edge case wrong.
import jwt # PyJWT 2.x; the same checks apply in any JWT library
from jwt import PyJWKClient
ISSUER = "" # exact string, never a prefix match
AUDIENCE = "billing-service" # this service's own identifier
ALLOWED_ALGS = ["RS256", "ES256"] # pinned; 'none' and HS* are rejected
MAX_SKEW_SECONDS = 30 # small, fixed clock tolerance
# One long-lived client: caches JWKS keys and refetches when it sees an unknown kid.
# Put a rate limit in front of refetches so random kids cannot flood the IdP.
jwks = PyJWKClient("")
class Unauthorized(Exception):
pass
def authenticate(headers: dict) -> dict:
# Identity comes only from the bearer token. Headers such as X-User-Id are
# ignored because any caller on the network can set them.
auth = headers.get("Authorization", "")
if not auth.startswith("Bearer "):
raise Unauthorized("missing bearer token")
token = auth[len("Bearer "):]
try:
# Select the key by kid from the cached JWKS, not from anything the token embeds.
signing_key = jwks.get_signing_key_from_jwt(token)
# decode() verifies the signature first, then checks iss, aud, exp and nbf.
return jwt.decode(
token,
signing_key.key,
algorithms=ALLOWED_ALGS, # the token's own alg header is not trusted
issuer=ISSUER,
audience=AUDIENCE,
leeway=MAX_SKEW_SECONDS,
options={"require": ["exp", "iss", "aud", "sub"]},
)
except jwt.exceptions.PyJWTError as exc:
# Fail closed and do not use any claim from a token that failed validation.
raise Unauthorized(f"invalid token: {exc}") from exc
A single validation function that every service could share: it ignores identity headers, selects the key from a cached JWKS, pins the allowed algorithms, checks the issuer and audience exactly, enforces expiry with 30 seconds of skew, and rejects the request if any check fails.
What does token exchange cost in latency, availability and operations?
Every exchange adds a network round trip to the request path, so latency is usually the first cost teams notice.
Illustrative estimate (assumed numbers, not a benchmark): added latency ≈ serial hops × exchange time × cache-miss rate. With 5 serial downstream calls, 10 ms per exchange and no caching, that is 5 × 10 ms × 1.0 = 50 ms. At a 90% cache hit rate, it drops to 5 ms.
Cache exchanged tokens keyed by subject, audience and scope. If you key by audience alone, one user's token can be served to another user. Refresh each token shortly before it expires so it never lapses mid-call. When downstream calls are independent, run the exchanges in parallel so you pay for one round trip instead of five.
The token service now sits in front of every request. Run it redundantly and decide in advance what happens when it is unreachable. Fail closed for writes and sensitive data. A degraded mode, such as serving from cache or going read-only, should be an explicit choice made per endpoint, not the side effect of a broad catch block.
A service that can mint tokens for any audience is a high-value target. Restrict which callers may request which audiences, and keep its signing keys in a hardware-backed store.
The operational costs are less visible:
- Revocation: caching delays revocation by up to the cached lifetime, so keep exchanged tokens short-lived.
- Policy drift: exchange rules tend to become more permissive as "temporary" broad grants pile up. Manage them as reviewed code.
- Debugging: correlate the original and exchanged tokens through trace IDs and token identifiers. Log claims, never raw tokens.
Budget engineering time for all of this before you commit.
How do you handle async jobs and long-running workflows?
A user access token is sized for an interactive request: minutes, not hours. Put it in a message and it may sit behind a backlog, a retry with backoff, or a nightly batch window. By the time a worker picks it up, the token has expired, and the job fails for reasons unrelated to its logic. Extending user token lifetimes to cover this weakens every token you issue. Forwarding refresh tokens to workers is worse, because it hands long-lived user credentials to infrastructure that never needed them.
There are three explicit alternatives.
- Exchange at enqueue time. Before publishing, the producer exchanges the user token for a delegation token scoped to one purpose, such as
audience=report-workerandscope=reports:generate, with a lifetime matched to the expected queue delay. Theactclaim preserves who delegated to whom, so auditability is strong. The cost is choosing a lifetime: too short and jobs still fail, too long and a leaked message carries a usable credential. - Service-issued delegation tokens. A trusted internal issuer mints tokens bound to a job ID that are valid only for that job's operations. This gives tight least privilege and fits workflows that run for days, but you now operate a token issuer and must protect its signing keys.
- Subject as data. The message records the user ID, tenant and requested action. The worker authenticates with its own workload identity and enforces authorization itself. This is the simplest option operationally. However, the worker's identity must be broad enough to act for any user, and the audit trail relies on the message being honest. Producers must be authenticated, and messages should be signed or integrity-protected.
What are the alternatives, and how do you migrate from propagation to exchange?
Most systems don't need to pick one model everywhere. The options sit on a spectrum:
| Approach | Least privilege | Latency | Operational burden | Development cost | Best fit |
|---|---|---|---|---|---|
| Plain JWT propagation | Weak: any holder can replay anywhere | None added | Low | Low | Small, single-team systems |
| Propagation + strict per-service audience | Moderate: replay limited to listed audiences | None added | Low–moderate | Moderate (multi-audience tokens) | Small to mid-size |
| Exchange at trust boundaries | Strong at boundaries | One exchange per boundary crossing | Moderate | Moderate | Mid-size, multiple domains |
| Exchange on every hop | Strongest | One exchange per hop | High (token service is critical path) | High | Regulated or high-risk paths |
| Workload identity + subject as context | Depends on caller honesty | None added | Moderate (mTLS/identity infra) | Moderate | Mesh-heavy platforms |
Migrating incrementally
- Inventory acceptance. Log the issuer, audience and calling workload for every token each service accepts. You can't tighten what you haven't mapped, and the data usually shows a few tokens reaching far more services than intended.
- Enforce audience in log-only mode. Add the check, but record violations instead of rejecting requests. That reveals hidden callers without causing outages.
- Exchange at the riskiest boundaries first. Start with payment, admin and cross-tenant edges, where a replayed token does the most damage.
- Tighten acceptance. Once logs show no violations, switch to rejecting and remove legacy audiences.
Each stage is reversible and leaves current callers working. Teams change their services on their own schedules instead of in a coordinated cutover, and effort goes to the boundaries where the risk actually is.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to service to service authentication, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.