What is service to service authorization, and why does every request need two identities?
Service to service authorization is the decision a service makes when the request comes from another service instead of from a person. When services act for end users, that decision depends on two separate questions. The first is which workload is calling. The second is on whose behalf it is calling. For example, a payments service should issue a refund only when the request comes from the orders service and the order belongs to the user the request represents.
| Workload identity | User identity | |
|---|---|---|
| Answers | Which service is calling? | Which person is this for? |
| Established by | Platform (certificates, mTLS) | Identity provider (login, tokens) |
| Typical lifetime | Hours, rotated automatically | Minutes to a session |
| Authorizes | Which APIs a caller may reach | Which data and actions are allowed |
Problems start when one credential has to carry both answers. Suppose orders calls payments with only its own service credential and sends the user ID as a plain request field. Payments trusts orders broadly, so it refunds whatever order ID arrives. If orders forgets an ownership check, a user can refund someone else's order. Payments did exactly what a trusted caller asked, using the caller's authority for a user who never had it. This is the confused deputy problem. The opposite shortcut, forwarding only the user's token, fails differently. Payments then cannot tell whether orders or some compromised internal service presented that token.
This matters for any team running several services that act for end users across trust boundaries, such as separate teams, clusters, or networks.
Who should use mTLS identity plus token exchange?
The combination is worth adopting if most of the following describe your system:
- Services act on behalf of end users. If a downstream service needs to know which user a request is for, and not just which service sent it, you need a user identity that travels across hops in a verifiable form.
- Several teams own services in the same call graph. When the team that owns payments cannot review every caller's code, payments has to enforce its own rules instead of trusting that callers checked ownership.
- Calls cross trust boundaries. Separate clusters, networks, cloud accounts, or tenants make IP-based trust meaningless and make a leaked token more dangerous.
- You handle sensitive or regulated data. Audit requirements often ask who acted, for whom, and through which path. The
actchain and mTLS peer identity answer that directly. - You run multi-tenant workloads. Narrow audiences and scopes limit how far a single compromised service can reach across tenants.
The roles involved differ. Platform or infrastructure teams run the CA, the node agents that issue workload certificates, and the token exchange service. Service teams consume a shared validation library and write the fine-grained authorization rules for the data they own. Security teams define the exchange policies that state which workload may act for which users toward which audiences.
It is likely overkill for a single team running a few services behind one gateway in one trust domain. That setup is covered at the end of this article.
How does it work, end to end?
The model layers two mechanisms, each answering one of the two identity questions:
- Workload identity through mTLS. Every service gets a short-lived certificate whose URI SAN names the workload, for example
spiffe://prod.example.internal/ns/payments/sa/orders. A node agent attests the workload and rotates the certificate automatically. On every connection, both sides present certificates, so the receiver knows which workload is calling before reading the request. - User identity through token exchange. The user logs in and gets a token for the front door. At each hop, the calling service sends the token it received (the
subject_token) and its own credential (theactor_token) to a token service, following OAuth 2.0 Token Exchange (RFC 8693). The token service applies policy and returns a new token whosesubis still the user, whoseaudis only the next service, whose scope is narrowed, whose lifetime is short, and whoseactclaim records the chain of services that acted. - Validation at every receiver. Each service checks the token's signature, issuer, audience, and expiry, then confirms that
act.submatches the mTLS peer identity. That binding ties the token to the workload actually presenting it. - Layered authorization decisions. The gateway makes coarse decisions (authenticated, right scope). Each service makes fine-grained decisions about the data it owns (does this user own this order?), and can also restrict which upstream workloads may reach a given API.
The sections below cover each step in detail, followed by a full request walkthrough.
How does mTLS establish workload identity?
With mutual TLS, both sides of a connection present certificates, so the server learns who is calling before any request bytes arrive. Each service gets its own certificate, and its identity is encoded as a URI in the Subject Alternative Name rather than a hostname. In SPIFFE-style naming, that looks like spiffe://prod.example.internal/ns/payments/sa/orders: a trust domain plus a path describing the workload. Authorization policy keys off this URI ("only orders may call POST /charges") instead of IP addresses, which change constantly in containerized environments.
Certificates should be short-lived, typically hours rather than months. A stolen key then stops working quickly without a revocation mechanism, which is useful because certificate revocation checking is unreliable in practice. Short lifetimes only work with automated rotation. An agent on each node attests the workload, requests a fresh certificate before the old one expires, and hot-reloads it without a restart. Each service also needs the current trust bundle, the set of CA certificates it accepts. That bundle must be distributed and updated ahead of any CA key rollover.
mTLS proves which workload is calling. It says nothing about which user the call is for. orders authenticates perfectly whether it acts for Alice, for Bob, or for nobody.
The operational cost is real. You run a CA, protect its keys, and monitor rotation. When rotation fails, expired or misdistributed certificates show up as handshake errors, connection resets, or timeouts that look like application bugs. Alert on certificate expiry and handshake failure rates directly so you catch these before they are misdiagnosed.
#!/usr/bin/env bash
set -euo pipefail
CERT=/var/run/workload/svid.pem # leaf cert written by the node agent
BUNDLE=/var/run/workload/bundle.pem # current trust bundle (accepted CA certs)
EXPECTED_ID="spiffe://prod.example.internal/ns/payments/sa/orders"
MIN_REMAINING=900 # seconds; certs live ~1h and rotate well before expiry
if ! openssl x509 -in "$CERT" -noout -ext subjectAltName | grep -qF "URI:${EXPECTED_ID}"; then
echo "CRIT: cert does not carry identity ${EXPECTED_ID}" >&2
exit 2
fi
if ! openssl verify -CAfile "$BUNDLE" "$CERT" >/dev/null; then
echo "CRIT: cert does not verify against trust bundle" >&2
exit 2
fi
if ! openssl x509 -in "$CERT" -noout -checkend "$MIN_REMAINING" >/dev/null; then
echo "WARN: cert expires within ${MIN_REMAINING}s, rotation may be stuck" >&2
openssl x509 -in "$CERT" -noout -enddate >&2
exit 1
fi
echo "OK: $(openssl x509 -in "$CERT" -noout -enddate)"
A monitoring script that uses openssl to confirm a workload certificate carries the expected SPIFFE ID in its URI SAN, chains to the current trust bundle, and is not close to expiring, so a stalled rotation triggers an alert instead of being misdiagnosed as an application failure.
Why is forwarding the user's JWT untouched a risky shortcut?
mTLS tells a service which workload is calling. It does not say which user the call is for. The most common shortcut is for the gateway to pass the user's original JWT unchanged to every downstream service. This works at first, but it creates four problems.
- Broad audience and scopes. The token was issued for the front door. It probably carries every scope the client asked for, and an audience that every service accepts. A low-privilege service ends up holding a token that could call the most sensitive one.
- Replay. A bearer token works for anyone who holds it. If any service in the chain is compromised, or simply buggy, it can reuse the token against any other service until the token expires.
- Leakage. The more hops a token makes, the more likely it ends up in a log, a trace attribute, an error report, or a message on a queue.
- Wide issuer trust. Every service has to fetch the external identity provider's keys and follow its rotation. A single misconfigured validator weakens the whole system.
| Concern | Forwarded user JWT | Exchanged downstream token |
|---|---|---|
| Audience | Gateway-wide | One target service |
| Scope | Everything the client requested | Only what this hop needs |
| Replay reach | Any service | Only the intended audience |
| Who trusts the external issuer | Every service | One exchange point |
Forwarding can still be acceptable in a small system with one team, one trust domain, short token lifetimes, and strict audience checks. Avoid it when there are multiple teams, third-party integrations, regulated data, or any service you cannot fully audit.
Migrations are often messy. During a migration, services typically accept both forwarded and exchanged tokens. That dual path tends to stay open long after the migration, which means an attacker can still use the old, weaker format. Set a removal date for the old path and count which callers still use it, so you can tell when it is safe to turn off.
How does OAuth 2.0 token exchange (RFC 8693) work?
Token exchange replaces forwarding with a deliberate swap. When the orders service needs to call payments, it sends a POST to the token service with grant_type=urn:ietf:params:oauth:grant-type:token-exchange. The request carries two tokens. The subject_token is the user's incoming token and says on whose behalf the call is made. The actor_token is the calling service's own credential and says who is acting. The caller also sets an audience (here, payments) and a narrower scope, such as payments:charge rather than the user's full grant.
The token service checks that orders may act for this user toward payments. If the policy allows it, it issues a new token with:
audset to payments only- the reduced scope
- a short expiry
- an
actclaim recording the delegation
If payments then exchanges again to call a ledger service, the new token's top-level act.sub names payments, and the earlier actor (orders) is nested inside it. The sub claim remains the user throughout. Per the RFC, only the outermost actor is meant for authorization decisions. The nested entries are history, useful for audit.
The trade-offs:
- Latency: every hop, and every distinct audience, adds a network round trip.
- Revocation: caching exchanged tokens removes that round trip but widens the revocation window to the cache lifetime.
- Dependency and risk: the token service becomes a critical dependency. It is also a high-value target, because it can mint tokens for any user.
- Header size: deep call chains grow the
actnesting until headers approach proxy size limits.
curl -s \
--cert /etc/certs/payments.crt --key /etc/certs/payments.key \
--data-urlencode 'grant_type=urn:ietf:params:oauth:grant-type:token-exchange' \
--data-urlencode "subject_token=${INCOMING_TOKEN}" \
--data-urlencode 'subject_token_type=urn:ietf:params:oauth:token-type:access_token' \
--data-urlencode "actor_token=${PAYMENTS_SVC_TOKEN}" \
--data-urlencode 'actor_token_type=urn:ietf:params:oauth:token-type:jwt' \
--data-urlencode 'audience=' \
--data-urlencode 'scope=ledger:write'
An RFC 8693 token exchange request from the payments service to obtain a ledger-scoped token, and the decoded result showing the user as sub and a nested act claim recording payments acting after orders.
What must a receiving service validate on every token?
Treat every inbound token as hostile until each check passes, and run the checks in a fixed order in middleware so no handler can skip them:
- Signature against a trusted key set. Fetch keys only from the issuer you configured, cache them, and refresh on unknown
kid. Pin the allowed algorithms yourself. Never trust the token'salgheader, ornoneand key-confusion attacks become possible. - Issuer.
issmust exactly match the token service. A valid signature from the wrong issuer is still a forgery. - Audience.
audmust equal this service's own identifier. This check is what makes the exchanged token useless if it is replayed against a sibling service. - Expiry. Reject if
exphas passed ornbfis in the future, with at most a minute or so of clock-skew tolerance. Generous skew silently extends every token's lifetime. - Actor binding. Compare
act.subwith the SPIFFE ID or certificate subject of the mTLS peer. A mismatch means a stolen token is being used by the wrong workload.
Only after all five pass should the middleware attach the user and actor to the request context.
| Self-contained JWT | Introspection | |
|---|---|---|
| Revocation | Waits for exp, so keep lifetimes short |
Immediate |
| Per-request cost | Local CPU only | Network call to the authorization server |
| Failure mode | Keeps working if the issuer is down | Fails closed (or open, if misconfigured) |
Package these checks in one shared, versioned library rather than letting each team write its own. Subtle validation bugs, such as a missing audience check or a lax algorithm list, then get fixed once and rolled out everywhere.
import jwt # PyJWT
ISSUER = ""
AUDIENCE = "spiffe://example.org/ns/payments/sa/ledger" # this service's own ID
ALLOWED_ALGS = ["RS256"] # pinned list; the token's alg header is not trusted
JWKS = jwt.PyJWKClient(ISSUER + "/.well-known/jwks.json") # caches fetched keys
class Unauthorized(Exception):
pass
def validate(token: str, peer_spiffe_id: str) -> dict:
"""Run every check in a fixed order. Call from middleware, not handlers."""
try:
# 1. Select the key by kid from the configured issuer's key set only.
key = JWKS.get_signing_key_from_jwt(token).key
# 2-4. Signature, issuer, audience, exp/nbf with 60s of skew.
claims = jwt.decode(
token,
key,
algorithms=ALLOWED_ALGS,
issuer=ISSUER,
audience=AUDIENCE,
leeway=60,
options={"require": ["exp", "iss", "aud", "sub"]},
)
except jwt.PyJWTError as e:
raise Unauthorized(f"token rejected: {type(e).__name__}")
# 5. Actor binding: the workload named in the token must be the mTLS peer.
actor = (claims.get("act") or {}).get("sub")
if actor != peer_spiffe_id:
raise Unauthorized("actor does not match mTLS peer identity")
# Only now is it safe to attach identity to the request context.
return {"user": claims["sub"], "actor": actor}
A Python validation function using PyJWT that checks the signature against a pinned algorithm list and the issuer's key set, then the issuer, audience, expiry with bounded skew, and finally binds act.sub to the caller's mTLS SPIFFE ID.
Walkthrough: user → gateway → orders → payments
A user calls POST /orders with an access token issued to the web client, audience api-gateway. The gateway validates signature, expiry and audience, then makes only a coarse decision: is the caller authenticated, and does the token carry an orders:write scope? It cannot know whether the cart belongs to this user, and it shouldn't try.
The gateway exchanges the edge token for a new one with audience orders, a short lifetime, the user as sub, and itself recorded in the act claim. It calls orders over mTLS. Orders checks two things independently. The peer certificate must identify the gateway workload, and the token must be addressed to orders. Then orders makes the fine-grained check, does this user own this cart?, because only orders holds that data.
To charge the card, orders re-exchanges for audience payments, adding itself to the actor chain. Payments now sees both the user and the path the request took, and can refuse charges that didn't arrive via orders.
Suppose the payments token expires before the call lands. Payments returns 401. Orders may re-exchange once, but only if its own inbound token is still valid. Otherwise it fails upward rather than retrying with stale credentials. A chain should never outlive the user's original authorization.
Splitting decisions this way invites policy drift: the gateway's scope rules and each service's ownership logic evolve separately. A centralized policy engine gives one auditable source of truth. In exchange, every decision depends on its availability and latency, and resource data must be fed to it. Many teams distribute centrally authored policies and evaluate them locally in a sidecar or library.
+--------+ edge token: aud=api-gateway, scope=orders:write
| User |
+--------+
| ^
| | 401 fails upward; chain never outlives user's token
v |
+------------------------------------------+ +-------------+
| GATEWAY check sig, exp, aud=api-gateway |<-->| TOKEN |
| coarse: authn + orders:write | | EXCHANGE |
+------------------------------------------+ | (STS) |
| mTLS: peer cert = gateway workload | |
| token aud=orders sub=user act=gateway | |
v | |
+------------------------------------------+ | |
| ORDERS check peer cert + aud=orders |<-->| |
| fine: does user own this cart? | | |
| on 401: re-exchange once, | | |
| only if inbound token valid | +-------------+
+------------------------------------------+
| mTLS: peer cert = orders workload
| token aud=payments sub=user act=orders>gateway
v
+------------------------------------------+
| PAYMENTS check peer cert + aud=payments |
| fine: refuse if not via orders |
+------------------------------------------+
^ policies authored centrally, evaluated locally in sidecar/library
+------------------------------------------+
| POLICY ENGINE (source of truth, audited) |
+------------------------------------------+
One POST /orders request: the gateway makes coarse checks on the edge token, each downstream hop verifies the mTLS peer and an audience-bound exchanged token with a growing actor chain, the data owner makes fine-grained checks, and expiry fails upward instead of retrying with stale credentials.
How can this model save engineering time?
The model has real setup and operating costs, covered above. The time savings come from moving repeated, error-prone work out of individual services and into shared infrastructure:
- Service teams stop building their own auth plumbing. Certificate issuance and rotation are handled by the node agent, and token validation lives in one shared library. A new service imports the library, declares its own audience identifier, and writes only the authorization rules for its own data. It does not need custom code for API keys, shared secrets, or IP allowlists.
- No manual secret rotation. Short-lived certificates rotate automatically, so there are no long-lived service passwords or static API keys to rotate by hand, track in spreadsheets, or chase down after an engineer leaves.
- Validation bugs get fixed once. When the checks live in one versioned library, a flaw such as a missing audience check is patched and rolled out everywhere. Without that library, every team has to find and fix the same flaw in its own code.
- Faster incident investigation. Audit logs carry the user, the full
actchain, and the mTLS peer identity. When a suspicious action shows up, responders can see which service acted, for whom, and through which path, without correlating IP addresses against deployment history. - Smaller blast radius means smaller incidents. Because exchanged tokens are bound to one audience and a narrow scope, a leaked token or compromised service affects fewer systems. Containment and credential cleanup cover a smaller set of services.
- Simpler security reviews. Reviewers can read exchange policies ("orders may act for users toward payments with
payments:charge") and service-level rules in one place. They do not have to trace ad hoc trust assumptions through each service's code. - Fewer misdiagnosed outages. Monitoring certificate expiry and handshake failures directly, as in the health check script above, turns a stalled rotation into a clear alert. Without that monitoring, the same failure shows up as connection resets that teams investigate as application bugs.
As an illustrative estimate, with all inputs assumed: suppose an organization has 30 services, and without shared infrastructure each team spends an assumed 2 engineer-days per quarter on credential rotation, auth-related bug fixes, and integrating with new callers. That is 60 engineer-days per quarter across the organization. If the shared platform takes that per-service effort down to an assumed half day, the total falls to 15 engineer-days, and the platform team's operating cost must then be subtracted from the saving. Replace these assumptions with your own numbers. The savings grow with the number of services and teams, which is why the model fits large systems better than small ones.
How do you handle async jobs and audit identity without leaking tokens?
A queued message outlives the request that created it. By the time a worker picks it up, the user's access token may have expired, and storing that token, or worse a refresh token, in the queue gives anyone who can read the broker a credential to replay. Instead, put minimal user context in the message: the subject identifier, the tenant, the action requested, and a reference to the originating request. The worker then authenticates as itself, using its own mTLS identity, and acts under an explicit delegation. That delegation can be a narrowly scoped token obtained through token exchange, or a policy rule such as "the billing worker may settle orders on behalf of any subject in the same tenant." Either way, the grant is visible and reviewable rather than implied by a stolen bearer token.
Decide deliberately whether to re-check authorization at processing time. Checking only at enqueue time is simpler, but a user removed from an account an hour ago should probably not have their export job finish. For destructive or sensitive actions, re-evaluate. For idempotent, low-risk work, the enqueue-time decision is usually enough.
For audit, log the subject, the full actor chain, and the peer workload identity from the mTLS handshake. Never log raw tokens. Log the token's identifier and issuer if you need correlation.
This full model pays off when you have many teams, regulated data, or real multi-tenancy. A small system with a few services behind one gateway can often rely on mTLS plus a forwarded, validated user ID. Plan to move to the full model as the system grows.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to service to service authorization, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.