Two questions every internal request must answer
When one service calls another, the receiver has to answer two separate questions before it can authorize anything.
Which workload is calling? This is service identity. The cleanest answer comes from the transport layer. With mTLS, each workload presents a short-lived certificate that names the service, and the receiver verifies it against a trust bundle. A stolen certificate soon expires, and identity is proven by possession of a private key rather than by a string that can be copied.
On whose behalf is it acting? This is the delegation context: the end user, the tenant, the permissions that apply to this specific request. A token carries it, because the same service handles many different users.
If you collapse both answers into one bearer token, each gets weaker. Anyone who holds the token can claim to be the calling service, and the user context has no reliable binding to the workload that presents it.
The common shortcut is to forward the original external access token to every downstream service. That causes several problems:
- Broad audience. Its audience is usually broad, so a leaked token can be replayed against any internal service that accepts it.
- Wide scopes. It was scoped for the client's needs, not for the narrow operation each hop performs.
- Coupling. Every service becomes coupled to the external identity provider's token format and claims.
- Wrong lifetime. Its lifetime was chosen for the client session, not for a single internal call.
mTLS has its own cost. You need certificate issuance, automated rotation, and trust bundle distribution, and all of it has to stay running. Without that infrastructure, teams fall back to network location or shared secrets, and the question of which workload is calling goes unanswered.
How OAuth token exchange (RFC 8693) works
In token exchange, a service calls the authorization server's token endpoint with grant_type=urn:ietf:params:oauth:grant-type:token-exchange. It sends the inbound user token as subject_token, with subject_token_type set to a value such as urn:ietf:params:oauth:token-type:access_token. It can also send its own credential as actor_token with an actor_token_type. The request names the target through audience or resource and asks for a narrower scope. The server applies policy and returns a new token scoped to that one downstream service.
The actor_token determines the semantics. Impersonation issues a token that simply names the user as sub, so the receiver cannot tell an intermediary was involved. Delegation keeps the user as sub and adds an act claim identifying the calling service. Each further hop nests the previous act inside a new one, which produces a chain such as {"sub": "user-42", "aud": "billing", "scope": "invoices:read", "act": {"sub": "orders-svc", "act": {"sub": "gateway"}}}. The specification treats only the outermost actor as relevant for access decisions. Earlier entries are informational.
The costs:
- Latency. Each new hop or audience adds a network round trip.
- Availability. The exchange endpoint sits on the critical path, so an outage there stalls internal traffic.
- Caching. Cache keys need to combine subject, audience and scope, or you risk serving a token with the wrong privileges.
- Token size. Nested
actclaims grow with call depth. They are usually logged but rarely enforced, so you pay header bytes for audit data.
# orders-svc exchanges the inbound user token for one scoped to billing only.
# Client credentials authenticate orders-svc itself to the token endpoint.
curl -s -X POST \
-u 'orders-svc:PLACEHOLDER_CLIENT_SECRET' \
--data-urlencode 'grant_type=urn:ietf:params:oauth:grant-type:token-exchange' \
--data-urlencode "subject_token=${INBOUND_USER_TOKEN}" \
--data-urlencode 'subject_token_type=urn:ietf:params:oauth:token-type:access_token' \
--data-urlencode "actor_token=${ORDERS_SVC_TOKEN}" \
--data-urlencode 'actor_token_type=urn:ietf:params:oauth:token-type:access_token' \
--data-urlencode 'audience=billing' \
--data-urlencode 'scope=invoices:read'
# Response (abridged). Cache it under (subject, audience, scope), not subject alone:
# {
# "access_token": "eyJhbGciOiJSUzI1NiIs...",
# "issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
# "token_type": "Bearer",
# "expires_in": 300,
# "scope": "invoices:read"
# }
#
# Decoded payload of access_token (delegation, two hops deep):
# {
# "iss": "",
# "sub": "user-42", # the user stays the subject
# "aud": "billing", # valid only at billing
# "scope": "invoices:read", # narrower than the inbound token
# "exp": 1767225900,
# "act": { # current actor: the one used for access decisions
# "sub": "orders-svc",
# "act": { "sub": "gateway" } # prior hop: informational, usually only logged
# }
# }
An RFC 8693 token exchange request from orders-svc using curl, followed by the response and a decoded delegated token whose nested act claim records the gateway → orders-svc call chain.
How the internal JWT pattern works
The alternative concentrates the expensive work at the edge. A gateway or dedicated edge auth service validates the external credential once, whether that is an OAuth access token, a session cookie or an API key. It then mints a short-lived internal JWT signed with a key that only it holds. The claim set is normalized: sub, tenant, roles, aud and exp, with expiry measured in minutes or less. Downstream services never see the external token format. They verify the signature locally against an internal JWKS endpoint, cache the keys, and avoid a network call on every request.
Edge minting (one token for the whole chain)
Client -> Gateway --mTLS--> A --mTLS--> B --mTLS--> C
mint T T T T
Per-hop exchange (new token per hop)
Client -> Gateway --mTLS--> A --mTLS--> B --mTLS--> C
T0 STS:T0->T1 STS:T1->T2
mTLS proves which workload is on the other end of each connection. The JWT carries the user and tenant the call is made for. You need both.
The trade-offs follow from the design:
- Trust sits in one key. Every service accepts anything the minting key signs. If that key leaks, an attacker can forge any user in any tenant, and every service will accept the forged token. Protect it as you would a CA key: keep it in an HSM or KMS, rotate it, and publish overlapping keys during rotation.
- Audience drift. When one token travels through a deep call chain, its
audhas to name every service it might reach. Itsrolesgrow toward the union of what each hop needs. The token stops expressing least privilege and starts acting as a network-wide pass.
EDGE MINTING: validate once, one internal JWT for the whole chain
+--------+ ext cred +-----------+ mTLS +-----+ mTLS +-----+ mTLS +-----+
| Client |--------->| Gateway |----->| A |----->| B |----->| C |
+--------+ | validate, | T +-----+ T +-----+ T +-----+
| mint T | | | |
+-----------+ v v v
| verify T locally, internal JWKS
v (keys cached, no per-request call)
[ internal signing key ] one key: leak = forge anyone
T = {sub, tenant, roles=union of A+B+C, aud=A,B,C, exp=minutes}
PER-HOP EXCHANGE: each hop swaps its token for a narrower one
+--------+ ext cred +-----------+ mTLS +-----+ mTLS +-----+ mTLS +-----+
| Client |--------->| Gateway |----->| A |----->| B |----->| C |
+--------+ +-----------+ T0 +-----+ T1 +-----+ T2 +-----+
| ^ | ^
T0 v | T1 T1 v | T2
+------------------------------+
| Token exchange service (STS) |
+------------------------------+
T1: aud=B, roles for B only T2: aud=C, roles for C only
Edge minting validates the external credential once and passes one internal JWT, verified against an internal JWKS, through every mTLS hop. Per-hop exchange obtains a new, narrowly scoped token at each hop from a token exchange service.
Implementing validation in every receiving service
Whichever pattern you choose, every receiving service runs the same checks. Put them in one shared library so no team reimplements them differently.
- Pin the algorithms. Configure the verifier with an explicit allow-list (for example, only
ES256) and reject anything else. If the verifier trusts the token'salgheader, an attacker can choosenoneor switch to a symmetric algorithm and sign with your public key. - Check
issagainst the one issuer you expect. A token from your customer-facing authorization server should never pass as an internal token. - Check
audagainst this service's own identifier. Without this check, a token minted for a low-risk service can be replayed against a sensitive one. - Check
expandnbfwith a small clock skew of tens of seconds, not minutes. A generous skew quietly extends every token's lifetime. - Bind to the mTLS peer. Confirm that the calling workload's certificate identity is allowed to present this token. A token that leaks outside the mesh is then useless on its own.
The verifier example that follows shows these checks along with JWKS caching.
Key rotation and revocation
To rotate signing keys, publish the new key in the JWKS before you sign with it, and keep the old key published until every token it signed has expired. When a token arrives with an unknown kid, refetch the JWKS once, rate-limited so forged kid values can't flood the issuer. If the key is still unknown after the refetch, reject the token.
For revocation, set an explicit latency budget, such as "a revoked credential stops working within 60 seconds." Then pick the mechanism that meets it:
- Very short expiry costs nothing per request, but the token lifetime is the revocation latency.
- A pushed deny list works within the propagation delay.
- Introspection is close to immediate, but it adds a network call on every request.
import { createRemoteJWKSet, jwtVerify, type JWTPayload } from 'jose';
import type { TLSSocket } from 'node:tls';
// One JWKS client per process. jose caches keys, refetches on an unknown kid,
// and the cooldown rate-limits those refetches so forged kids can't flood the issuer.
const jwks = createRemoteJWKSet(
new URL(''),
{ cacheMaxAge: 10 * 60_000, cooldownDuration: 30_000, timeoutDuration: 2_000 },
);
const EXPECTED_ISSUER = '';
const THIS_SERVICE = 'spiffe://example.internal/ns/payments/sa/ledger';
export async function verifyInternalToken(bearer: string, socket: TLSSocket): Promise<JWTPayload> {
const { payload } = await jwtVerify(bearer, jwks, {
algorithms: ['ES256'], // allow-list; the token's alg header is never trusted
issuer: EXPECTED_ISSUER, // reject tokens from any other issuer
audience: THIS_SERVICE, // token must be minted for this service
clockTolerance: 30, // seconds of skew for exp/nbf, not minutes
requiredClaims: ['exp', 'iss', 'aud', 'sub'],
});
// Bind to the mTLS peer: the caller's certificate identity must match the token subject.
const cert = socket.getPeerCertificate();
const peerId = (cert.subjectaltname ?? '')
.split(', ')
.find((san) => san.startsWith('URI:spiffe://'))
?.slice('URI:'.length);
if (!socket.authorized || !peerId || peerId !== payload.sub) {
throw new Error('token not presented by the workload it was issued to');
}
return payload;
}
A shared TypeScript verifier using the jose library that pins ES256, checks issuer, audience, exp/nbf with 30 seconds of skew, caches the JWKS with a rate-limited refetch on unknown kid, and requires the token subject to match the caller's mTLS SPIFFE identity.
Non-user flows: background jobs, queues and scheduled work
A nightly reconciliation job, a cron-triggered cleanup, or a consumer draining a queue has no user behind it and should not pretend to. It authenticates as itself through workload identity, such as an mTLS certificate or a platform-issued service token, and receivers authorize it on that identity alone. Minting a synthetic "system user" JWT to make it look like user traffic muddies audit logs and tends to collect broad permissions that nobody reviews.
The harder case is a user action that starts work which finishes later. By the time a consumer picks up the message, the user's access token has probably expired. Copying it into the payload also puts a bearer credential into a broker that retains, replays and exposes messages to anyone with read access. Carry one of these instead:
- A purpose-scoped token minted for the job, with a narrow audience, a single permitted action and a lifetime sized to the expected queue delay.
- A reference, such as the user ID, tenant and the authorization decision already made. The consumer acts on it under its own workload identity, and receivers trust that service for that specific delegated scope.
Long-running requests, streaming connections and multi-hour exports raise the same tension: they outlive a short token. The tempting fix is to raise the lifetime globally. That stretches revocation latency for every token, not only the slow path, and nothing breaks to tell you it happened. A safer approach is to re-exchange at checkpoints, validate when a stream is set up, and recheck at intervals while it stays open.
Token exchange vs internal JWTs: when to use and when to avoid each
The baseline, forwarding the user's external token to every downstream service, is the option to move away from. Every service then accepts a credential whose audience is the public API, so a single compromised service can replay that token against any other service.
| Dimension | Forward external token | Internal JWT | Token exchange |
|---|---|---|---|
| Latency | None added | Local signing at the edge | Network call to the token service per exchange, unless cached |
| Blast radius | Whole system | Whole trust domain | One audience and scope |
| Key risk | External token leaks internally | One signing key trusted everywhere | Token service availability and policy errors |
| Revocation | External token lifetime | Short internal TTL | Per exchange, at issuance |
| Audit | User only | User plus edge | Full delegation chain (act claim) |
| Operational burden | Low | Low to moderate | High |
When to use each
Choose token exchange when several teams or trust domains share the system, when least privilege must hold per hop, or when auditors need to see who acted for whom. Avoid it when the token service would sit on a latency-critical path that has no caching budget, or when one team owns every service anyway.
Internal JWTs are enough for a single trust domain where authorization happens mostly at the edge. Avoid them when services hold very different privileges, because one leaked signing key forges identity everywhere.
A hybrid setup uses internal JWTs inside a domain and token exchange at domain boundaries. The cost is two issuers, two sets of validation logic, and two failure modes to debug, such as an expired internal token compared with a denied exchange.
Impact on development cost and engineering time
Whichever pattern you pick, the token issuer becomes a critical-path dependency. If the exchange endpoint or the minting service is slow or down, every downstream call fails. That service needs the same redundancy, monitoring and on-call ownership as your API gateway. mTLS adds its own operational work: issuing certificates, rotating them before expiry, and distributing trust bundles. Teams often underestimate this until the first outage caused by an expired certificate.
The largest recurring cost is usually per-service validation code. When each team writes its own JWT checks, issuer handling, audience rules and clock-skew tolerance drift apart, so the same token is accepted by one service and rejected by another. Three practices cut this cost:
- A shared verification library. A new service gets correct validation by adding one dependency.
- A normalized claim contract. Every service reads
sub,actand scopes from the same fields, so policies stay portable between services. - Separate workload and user identity. When a request fails, a 403 can be traced to one cause: the calling service was not permitted, or the end user was not.
Illustrative estimate (assumed numbers, not a benchmark)
Exchange calls/sec = requests/sec × hops per request × (1 − cache hit rate)
Assume 500 req/s, 4 hops and a 90% cache hit rate: 500 × 4 × 0.1 = 200 exchange calls/sec to the issuer.
If each cache miss costs 5 ms, the average added latency is 4 × 0.1 × 5 = 2 ms per request. A request that misses the cache at every hop pays 20 ms.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to service to service authentication jwt, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.