Service to Service Authentication: mTLS, SPIFFE and JWT Tokens

What service to service authentication is and why network location is not identity

Service to service authentication answers one question: which workload is making this call? Authorization answers a different one: is this caller allowed to perform this action on this resource, right now? Many systems merge the two. They treat "the request came from the billing service" as permission to do anything billing might ever need. That shortcut is where most internal breaches spread.

The most common shortcut uses network location as a stand-in for identity, through IP allowlists, security groups or "anything inside the cluster is trusted." This breaks in three ordinary ways:

• Pods reschedule. An IP that belonged to the payments service a minute ago can be reassigned to a batch job. Your allowlist now trusts the wrong workload.
• Networks flatten. VPC peering, shared clusters and service meshes connect more things over time. "Internal" ends up covering hundreds of workloads you never reviewed.
• Attackers move laterally. One compromised host inherits every trust relationship tied to its network position. No credential is needed.

This guide's thesis is that proving workload identity and carrying request-level authorization context are separate jobs. mTLS with SPIFFE handles the first. Scoped tokens handle the second, including which user or tenant a request is acting for. Most production systems need a layer for each.

This applies if you run many internal services across teams, host multi-tenant or regulated workloads, or are working toward zero-trust networking. It is overkill for a monolith plus a database, or for a single team running a handful of services on a truly isolated network. In those setups, the operational cost exceeds the risk it removes.

How mTLS and SPIFFE/SPIRE prove workload identity

In mutual TLS, both client and server present X.509 certificates during the handshake. Each side verifies the peer's chain against a trust bundle, which is the set of CA certificates it accepts. After the handshake, the server reads the caller's identity from the certificate's URI SAN, such as spiffe://prod.example.org/ns/payments/sa/api. It does not read identity from an X-Service-Name header, because any process that can reach the port can forge a header.

SPIFFE is a specification, not software. It defines:

• The SPIFFE ID format: spiffe://trust-domain/path.
• Two identity documents (SVIDs): X.509-SVIDs for mTLS and JWT-SVIDs for cases where TLS terminates early.
• Trust bundles: the CA material for each trust domain.
• The Workload API: a local endpoint that workloads call to fetch their SVIDs and bundles.

SPIRE is the reference implementation. It issues short-lived SVIDs and rotates them automatically, so a leaked certificate expires quickly.

The hard part is attestation, meaning proof of what a workload is. SPIRE agents first attest their node using platform evidence such as a cloud instance identity document or a Kubernetes projected service account token. They then attest workloads by inspecting kernel and orchestrator metadata. Because the evidence comes from the platform, no pre-shared credential has to be delivered to the workload. This removes the "secret zero" bootstrapping problem.

A registration entry maps selectors to an identity:

spire-server entry create -parentID spiffe://prod.example.org/agent/k8s -spiffeID spiffe://prod.example.org/ns/payments/sa/api -selector k8s:ns:payments -selector k8s:sa:api

To confirm what a peer will see, inspect the SVID with openssl x509 -in svid.pem -noout -text and check the Subject Alternative Name field.

Copy#!/usr/bin/env bash
set -euo pipefail

# 1. Register the workload on the SPIRE server. A workload in namespace
#    'payments' running as service account 'api' on a node attested by
#    the parent agent receives this SPIFFE ID. No secret is distributed.
spire-server entry create \
  -parentID spiffe://prod.example.org/agent/k8s \
  -spiffeID spiffe://prod.example.org/ns/payments/sa/api \
  -selector k8s:ns:payments \
  -selector k8s:sa:api

# Confirm that the entry exists.
spire-server entry show -spiffeID spiffe://prod.example.org/ns/payments/sa/api

# 2. Run this inside the workload. Fetch its X.509-SVID from the Workload
#    API over the local agent socket. The agent attests the caller from
#    kernel and orchestrator metadata, so the request carries no credential.
mkdir -p /tmp/svid
spire-agent api fetch x509 \
  -socketPath /tmp/spire-agent/public/api.sock \
  -write /tmp/svid
# This writes svid.0.pem, svid.0.key and bundle.0.pem.

# 3. Inspect the identity that a peer reads during mTLS. The URI SAN must
#    match the registered SPIFFE ID. The subject CN is not used.
openssl x509 -in /tmp/svid/svid.0.pem -noout -ext subjectAltName
# Expected output contains:
#   URI:spiffe://prod.example.org/ns/payments/sa/api

# The SVID is short-lived, so check its validity window.
openssl x509 -in /tmp/svid/svid.0.pem -noout -dates

# On OpenSSL versions without -ext, read the SAN from the full text dump.
# openssl x509 -in /tmp/svid/svid.0.pem -noout -text | grep -A1 'Subject Alternative Name'

Registers a SPIRE workload entry using Kubernetes selectors, fetches the resulting X.509-SVID through the Workload API, and uses openssl to confirm the SPIFFE ID in the URI SAN and the short validity window.

How JWT service tokens carry authorization and user context

mTLS tells a service which workload is calling. It says nothing about what that call may do or on whose behalf. A JWT carries that information. The claims that matter are iss (who minted it), sub (the workload or user), aud (the single service meant to accept it), exp, and scope. A decoded payload might read: {"iss":"","sub":"spiffe://example.org/ns/payments/sa/api","aud":"ledger","scope":"ledger:write","exp":1767225600}.

The audience claim stops replay. If someone copies this token from the ledger's logs, billing rejects it because aud does not match. Short lifetimes, measured in minutes, mean a leaked token expires before it is useful. Sign tokens with an asymmetric algorithm such as RS256 or ES256, and have verifiers fetch public keys from the issuer's JWKS endpoint. With a shared HMAC secret, every verifier can also mint tokens.

Validation must check the signature, iss, aud and exp. It must also pin the accepted algorithm, so a token that declares none or swaps RS256 for HS256 is refused.

There are three ways to carry user context across hops. Service-only credentials create the confused deputy problem. The downstream service sees only the caller's broad privileges, so a user can request another tenant's data and the service fetches it on its own authority. Token exchange (RFC 8693) avoids this by issuing a downscoped token per hop.

Copy# Verify a service JWT before acting on it (PyJWT with the crypto extra installed).
import jwt
from jwt import PyJWKClient

ISSUER = ""
AUDIENCE = "ledger"  # this service's own name, never a shared or wildcard value
JWKS_URL = ""  # placeholder endpoint

# Fetch the issuer's public keys. Verifiers never hold anything that can mint tokens.
jwks_client = PyJWKClient(JWKS_URL)

def verify_service_token(token: str, required_scope: str) -> dict:
    # Select the public key using the token's 'kid' header.
    signing_key = jwks_client.get_signing_key_from_jwt(token)

    claims = jwt.decode(
        token,
        signing_key.key,
        # Pin the algorithm. Tokens that declare 'none' or switch to HS256 are rejected,
        # whatever their header says.
        algorithms=["ES256"],
        issuer=ISSUER,      # reject tokens minted by any other issuer
        audience=AUDIENCE,  # reject tokens meant for billing or any other service
        leeway=30,          # allow small clock skew, nothing more
        # Fail if these claims are absent instead of silently skipping their checks.
        options={"require": ["exp", "iss", "aud", "sub"]},
    )

    # Signature, iss, aud and exp are now verified. Authorization is a separate step.
    if required_scope not in claims.get("scope", "").split():
        raise PermissionError(f"missing scope {required_scope}")
    return claims

# Usage in a request handler:
# try:
#     claims = verify_service_token(bearer_token, "ledger:write")
# except (jwt.InvalidTokenError, PermissionError):
#     return 401 if the token is invalid, 403 if the scope is missing

A PyJWT validator for the ledger service. It fetches public keys from the issuer's JWKS endpoint, pins ES256, requires and checks iss, aud and exp, and then confirms the token carries the scope the operation needs.

A layered reference design and where to enforce policy

Combine all three layers. mTLS with X.509 SVIDs authenticates the calling workload. A short-lived JWT whose aud names the receiving service carries the user and scopes. A central policy repository holds each service's allowlist. Each layer covers a gap in the others: the certificate cannot carry user context, and a bearer token can be replayed by anyone who obtains it.

Copy[SPIRE agent] --SVID via Workload API--> [orders] [payments] <--SVID-- [SPIRE agent]
                                            |  mTLS + JWT(aud=payments)  ^
                                            +----------------------------+
                                                           |
                                              peer SPIFFE ID + claims
                                                           v
                                                  [policy decision point]

Where you evaluate policy changes how consistent it is and what each check costs:

A practical split puts identity checks in the proxy and object-level checks in middleware, and both query the same rule source.

Write rules against identities. Default to deny. An illustrative rule: allow POST /refunds only when the peer SPIFFE ID is spiffe://prod.example/ns/billing/sa/orders, the token aud is payments, and scope includes refunds:write. Deny everything else.

Hostnames and IP addresses do not work as rule subjects. They change when workloads reschedule, and anything that reaches the address inherits the permission. A SPIFFE ID changes only when the SPIRE registration changes, and that change goes through review.

Copy# Default-deny authorization check, run after mTLS and JWT verification.
# peer_id comes from the verified X.509 SVID's URI SAN; claims come from a
# JWT whose signature, expiry and issuer have already been checked.

ALLOW_RULES = [
    {
        "method": "POST",
        "path": "/refunds",
        "peer": "spiffe://prod.example/ns/billing/sa/orders",
        "aud": "payments",          # this service's own name
        "scope": "refunds:write",
    },
]

def _as_set(value, split=False):
    # 'aud' may be a string or a list (RFC 7519); 'scope' is space-delimited.
    if value is None:
        return set()
    if isinstance(value, str):
        return set(value.split()) if split else {value}
    return set(value)

def authorize(method, path, peer_id, claims):
    audiences = _as_set(claims.get("aud"))
    scopes = _as_set(claims.get("scope"), split=True)
    for rule in ALLOW_RULES:
        if (method == rule["method"]
                and path == rule["path"]
                and peer_id == rule["peer"]          # exact match, no hostnames
                and rule["aud"] in audiences
                and rule["scope"] in scopes):
            return True
    return False  # nothing matched: deny

# authorize("POST", "/refunds", "spiffe://prod.example/ns/billing/sa/orders",
#           {"aud": "payments", "scope": "refunds:write"})  -> True
# Same request from any other SPIFFE ID, audience or scope      -> False

An illustrative default-deny check that allows POST /refunds only when the caller's SPIFFE ID, the token audience and the token scope all match. No hostnames or IP addresses appear in the rule.

Trade-offs and when to avoid this approach

Every piece of this design moves failure modes rather than removing them. SPIRE, or any internal PKI, becomes critical infrastructure. If the server or its agents are unavailable when certificates reach expiry, workloads cannot renew, and connections fail across the fleet at roughly the same time. Plan for that availability the way you would for DNS, and alert on time-to-expiry, not only on issuance errors.

Short lifetimes help containment, but they make rotation a constant operation. A failed renewal, a hot-reload bug or clock skew on a node is now enough to cause an outage.

Sidecar-terminated mTLS authenticates the proxy, not the application process. If the app port is reachable directly, any caller can skip the check, so bind the application to localhost or block that port with network policy. TLS-terminating load balancers and gateways have the same effect: the backend sees the gateway's identity, not the original caller's.

Bearer JWTs can be replayed by anyone holding them until they expire. Revocation is weak for both certificates and tokens, so in practice short lifetimes serve as the revocation mechanism. Forwarding the user's token deep into the call graph spreads a broad credential to every hop and ties each service to the identity provider's token format.

Trust-domain names and federation topology end up embedded in policies and IDs, so they are expensive to change later.

A simpler approach is often better in these cases:

• You run a handful of services under one team.
• Everything sits inside a single cloud account where IAM roles already identify workloads.
• Nobody is available to operate a PKI around the clock.

Alternatives: API keys, shared secrets, cloud IAM and mesh-only identity

Each alternative gives up part of the layered design. The question is whether you need that part.

Shared HMAC secrets fail because of the "any holder" problem. A secret known to five services cannot tell those five apart. Cloud IAM is good enough as long as you stay inside one provider. It stops being enough once workloads span clusters, clouds or on-premises hosts. Mesh-only mTLS is the closest alternative. Choose it when every authorization decision depends on which service is calling and none depends on which user the request is for.

Migration path, development cost and engineering time

Migrate one caller-to-callee edge at a time, and make every step reversible with a configuration change instead of a redeploy.

1. Inventory callers. For each API key or shared secret, record who holds it and which endpoints it reaches. Unknown holders are the main risk, because they break silently at enforcement time.
2. Run mTLS in permissive mode. The server accepts both mTLS and legacy credentials. Nothing breaks, so this step costs little to undo.
3. Log peer identities. Record the peer SPIFFE ID next to the legacy credential on every request. A caller that appears only with a key has not migrated yet.
4. Enforce per service. Switch that service to strict mode, then revoke its old key. Keep strict versus permissive as a per-service flag so rollback is a single setting.

Costs. The upfront work is deploying SPIRE, defining attestation, and integrating clients. The ongoing work is PKI operations: CA rotation, trust bundle distribution, and treating SPIRE availability as production-critical. Token exchange adds latency. Cache tokens until they are close to expiry, so the exchange happens once per token lifetime instead of once per request.

Savings. SVIDs are short-lived and rotate automatically, so nobody rotates or distributes secrets by hand. Policy is written against identities the same way in every service. Audits can answer "who called this" from the logs.

Illustrative estimate (assumed numbers, not a benchmark):

• Rotation hours per year = services × secrets per service × rotations per year × hours per rotation.
• With 40 × 3 × 4 × 1.5, that is 720 hours per year.
• Against that, assume 4 hours per week of PKI operations, which is 208 hours per year.
• Replace these inputs with your own numbers.

Copy# Hypothetical per-service auth config (illustrative schema, not a specific tool).
# One flag per service, so rollback is a config change, not a redeploy.
services:
  orders-api:
    # Step 2: accept mTLS and legacy API keys side by side.
    auth_mode: permissive
    trust_domain: example.internal
    legacy_api_keys_allowed: true
    # Step 3: log the peer identity on every request.
    log_peer_identity: true

  billing-api:
    # Step 4: enforce. Only callers with a valid X.509-SVID are accepted.
    auth_mode: strict
    trust_domain: example.internal
    legacy_api_keys_allowed: false
    allowed_peers:
      - spiffe://example.internal/ns/prod/sa/orders-api
      - spiffe://example.internal/ns/prod/sa/checkout-api
    # Rollback: set auth_mode back to permissive and
    # legacy_api_keys_allowed back to true.

# Example access log lines written while orders-api runs in permissive mode.
# A caller with peer_spiffe_id=none and a key_id has not migrated yet.
#
# ts=2026-10-24T09:14:02Z svc=orders-api mode=permissive
#   peer_spiffe_id=spiffe://example.internal/ns/prod/sa/checkout-api
#   legacy_key_id=none path=/v1/orders status=200
#
# ts=2026-10-24T09:14:05Z svc=orders-api mode=permissive
#   peer_spiffe_id=none
#   legacy_key_id=key-reporting-07 path=/v1/orders status=200

A per-service config that flips one service between permissive and strict mode, with sample log lines that record each caller's SPIFFE ID next to any legacy key so you can find callers that have not migrated yet.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to service to service authentication, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.