What is SPIFFE, and what does SPIRE add?
SPIFFE (Secure Production Identity Framework for Everyone) is a set of specifications for workload identity. It defines formats and interfaces, not software.
- SPIFFE ID: a URI such as
spiffe://prod.example.org/payments/api. The host part names the trust domain and the path identifies the workload. - Trust domain: the administrative boundary whose signing keys vouch for its IDs.
- X.509-SVID: an X.509 certificate that carries the SPIFFE ID in its URI SAN. It is the credential used in mTLS. The spec also defines a JWT-SVID.
- Trust bundle: the set of CA certificates a workload uses to verify SVIDs from a trust domain.
- Workload API: a local endpoint, usually a Unix domain socket, where a workload fetches its SVID, private key and bundles, and receives updates as they change.
SPIRE is the reference implementation. The SPIRE Server signs SVIDs and stores registration entries, which map selectors (for example, a Kubernetes namespace and service account) to SPIFFE IDs. A SPIRE Agent runs on each node. The agent proves the node's identity to the server through node attestation. It then identifies calling processes through workload attestation, using kernel and orchestrator metadata.
| SPIFFE concept | SPIRE component |
|---|---|
| Trust domain | SPIRE Server configuration and its signing CA |
| SPIFFE ID | Registration entries on the server |
| X.509-SVID | Signed by the server, cached and delivered by the agent |
| Trust bundle | Maintained by the server, distributed through agents |
| Workload API | Exposed by the agent on each node |
Teams adopt this model because a workload never holds a bootstrap secret. Identity comes from attested facts about where the workload is running and what it is. That removes the hand-copied keys and long-lived certificates that make mTLS fragile.
How does SPIFFE mTLS work between microservices?
Identity is established in two stages: first for the node, then for the workload.
+---------------------+
| SPIRE Server |
| (CA, registration |
| entries) |
+----------+----------+
node attestation | node attestation
+--------------+--------------+
| |
+--------v---------+ +--------v---------+
| Node A | | Node B |
| SPIRE Agent | | SPIRE Agent |
| | Workload | | | Workload |
| | API sock | | | API sock |
| [orders svc] |<========>| [billing svc] |
+------------------+ mTLS +------------------+
Node attestation. When an agent starts, it proves to the server which machine it is running on. Platform evidence does this, such as a Kubernetes projected service account token or a cloud instance identity document. The server then sends the agent only the registration entries scoped to that node.
Workload attestation. A workload connects to the agent's Unix domain socket. It presents no credentials. The agent asks the kernel for the caller's process ID and then inspects that process. It gathers selectors such as unix:uid:1001 or k8s:ns:payments plus k8s:sa:billing. If those selectors match a registration entry, the agent returns an X.509-SVID and the trust bundle. The workload never needs a secret to ask for a secret.
The handshake. Each side presents its SVID and validates the peer's chain against the trust bundle. It then reads the SPIFFE ID from the URI SAN.
Selectors are the weak point. Suppose an entry matches only k8s:ns:payments. Then every pod in that namespace receives the billing identity, including a debug pod someone launched there. Combine namespace and service account, at minimum, so that an identity maps to exactly the workloads that should hold it.
+--------------------------------+
| SPIRE Server |
| CA + registration entries |
+---------------+----------------+
(1) node attestation | (1) node attestation
+--------------------------+--------------------------+
| |
+--------v-------------------+ +-----------------v----------+
| Node A | | Node B |
| +----------------------+ | | +----------------------+ |
| | SPIRE Agent | | | | SPIRE Agent | |
| +----------+-----------+ | | +----------+-----------+ |
| | | | | |
| Workload API socket | | Workload API socket |
| (2) attest by selectors | | (2) attest by selectors |
| (3) X.509-SVID + bundle | | (3) X.509-SVID + bundle |
| | | | | |
| +----------v-----------+ | (4) mTLS | +----------v-----------+ |
| | orders svc |<===================>| billing svc | |
| | unix:uid:1001 | | | | k8s:ns:payments | |
| +----------------------+ | | | k8s:sa:billing | |
+----------------------------+ | +----------------------+ |
+----------------------------+
Risk: an entry matching only k8s:ns:payments gives the billing SPIFFE ID
to every pod in that namespace. Pin namespace AND service account.
The SPIRE server attests each node's agent (1). Each agent attests local workloads over the Workload API socket by kernel-derived selectors (2) and issues X.509-SVIDs with the trust bundle (3). The workloads then authenticate each other over mTLS (4).
Why do short-lived SVIDs make certificate rotation routine?
When every SVID expires within an hour or so, rotation stops being a quarterly change ticket and becomes something the system does continuously. The agent renews each workload's SVID well before expiry, commonly around the halfway point of its lifetime, so a new key pair and certificate replace the old one many times a day. A failed renewal path shows up within hours rather than surfacing on the one day a year a long-lived certificate expires.
Short lifetimes also limit how long a stolen key is useful. SPIFFE does not depend on CRL- or OCSP-style revocation, so a leaked SVID stays valid until it expires. The TTL is your residual exposure window after a compromise.
Shorter is not always better. Very short TTLs increase signing load on the server, add renewal traffic, and make validation more sensitive to clock skew between hosts. They also shrink how long an agent can keep serving valid SVIDs after it loses contact with the server.
Illustrative estimate. Assume 2,000 workloads and renewal at half of TTL. The worst case is an agent that loses server contact just before a renewal, which leaves half the TTL before its cached SVIDs expire.
| SVID TTL | Worst-case outage tolerance | Renewals per second | Leaked-key window |
|---|---|---|---|
| 5 minutes | ~2.5 minutes | ~13 | ≤5 minutes |
| 1 hour | ~30 minutes | ~1.1 | ≤1 hour |
| 24 hours | ~12 hours | ~0.05 | ≤24 hours |
Choose the shortest TTL whose outage tolerance still exceeds your realistic time to restore the server.
How should services consume rotated SVIDs?
A service that reads its certificate from disk once at startup keeps presenting that certificate after it expires. New handshakes then fail, and the process looks healthy until peers begin rejecting it. Each service needs some way to pick up new SVIDs without a redeploy.
Stream from the Workload API. The process keeps a connection open to the SPIRE agent and receives each new SVID and trust bundle as it is issued. In Go, the crypto/tls config exposes callbacks such as GetCertificate, which run on every handshake and return whatever SVID is current at that moment. The go-spiffe library wraps this pattern in an X.509 source that you pass into its TLS config helpers. The code example below uses that approach.
Sidecar writes files. A helper process such as spiffe-helper fetches SVIDs, writes the certificate, key and bundle to a shared volume, then signals the application to reload. This suits software you cannot modify, but it only works if the application actually re-reads those files when signalled.
| Approach | Code changes | Restart needed | Main failure mode |
|---|---|---|---|
| Startup load | None | Yes, per rotation | Expired certificate, handshakes fail |
| Workload API stream | SPIFFE library, TLS callback | No | Agent unreachable, so the last SVID eventually expires |
| Sidecar plus reload | Signal handler, file reload | No, if reload works | Missed signal or partial file write |
Both approaches change only new handshakes. An established connection keeps the identity that was verified when it opened, so an hours-old HTTP/2 or gRPC connection ignores both rotation and revocation. To make new certificates take effect within a known window, cap connection lifetime, for example with a server-side maximum connection age shorter than the SVID TTL.
// Go: gRPC server that serves the current SVID on every handshake and caps connection age
package main
import (
"context"
"log"
"net"
"time"
"github.com/spiffe/go-spiffe/v2/spiffeid"
"github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
"github.com/spiffe/go-spiffe/v2/workloadapi"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/keepalive"
)
func main() {
ctx := context.Background()
// Keeps a stream open to the SPIRE agent; new SVIDs and bundles replace old ones in memory.
source, err := workloadapi.NewX509Source(ctx, workloadapi.WithClientOptions(
workloadapi.WithAddr("unix:///run/spire/sockets/agent.sock")))
if err != nil {
log.Fatalf("workload API: %v", err)
}
defer source.Close()
// The source backs the TLS certificate and root callbacks, so each handshake sees the latest SVID.
clientID := spiffeid.RequireFromString("spiffe://example.org/ns/default/sa/frontend")
tlsConf := tlsconfig.MTLSServerConfig(source, source, tlsconfig.AuthorizeID(clientID))
// Rotation only affects new handshakes: force reconnects well inside the SVID TTL (assumed 1h).
srv := grpc.NewServer(
grpc.Creds(credentials.NewTLS(tlsConf)),
grpc.KeepaliveParams(keepalive.ServerParameters{
MaxConnectionAge: 20 * time.Minute,
MaxConnectionAgeGrace: 30 * time.Second, // let in-flight RPCs finish
}),
)
lis, err := net.Listen("tcp", ":8443")
if err != nil {
log.Fatal(err)
}
log.Fatal(srv.Serve(lis))
}
A Go gRPC server that uses a go-spiffe X509Source so every new handshake presents the current SVID, and sets a maximum connection age so long-lived connections are recycled and pick up rotated or revoked identities.
Authorizing on SPIFFE IDs instead of hostnames
Chain verification and authorization are two separate decisions. Validating the peer's certificate against the trust bundle proves only that the SPIRE server for that trust domain issued it. Any workload in the domain can pass that check. To decide whether this particular caller may connect, read its SPIFFE ID from the URI SAN and compare it against an explicit allowlist or policy.
Hostname matching does not fit this model. X.509-SVIDs often carry no DNS SANs at all. Pod IPs and hostnames change on every reschedule. A hostname also says where a process runs, not what it is. The SPIFFE ID stays stable across all of that because it comes from attested workload properties.
In Go, the go-spiffe library expresses this as an authorizer passed to its mTLS config helpers. The example below uses tlsconfig.AuthorizeID to accept exactly one caller, such as spiffe://example.org/ns/payments/sa/api.
Mistakes that show up in real deployments:
- Accepting any ID in the trust domain.
AuthorizeMemberOforAuthorizeAnyon a sensitive service turns "issued by our CA" into "allowed to call us." Reserve them for genuinely open endpoints. - Prefix or substring matching. If you match by prefix,
spiffe://example.org/ns/paymentsalso matches.../ns/payments-admin. Parse the ID and compare whole path segments or exact values. - Skipping verification to drop hostname checks. Setting
InsecureSkipVerifywithout a replacement verifier disables chain validation entirely. - Rejecting malformed SVIDs too loosely. An X.509-SVID must carry exactly one URI SAN. Reject certificates that carry several.
// Go (go-spiffe v2): mTLS server that accepts exactly one caller SPIFFE ID.
package main
import (
"context"
"log"
"net/http"
"github.com/spiffe/go-spiffe/v2/spiffeid"
"github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
"github.com/spiffe/go-spiffe/v2/workloadapi"
)
func main() {
ctx := context.Background()
// The X509Source fetches this workload's SVID and the trust bundle from the
// SPIRE agent, and picks up rotated certificates without a restart.
source, err := workloadapi.NewX509Source(ctx, workloadapi.WithClientOptions(
workloadapi.WithAddr("unix:///run/spire/sockets/agent.sock")))
if err != nil {
log.Fatalf("unable to create X509Source: %v", err)
}
defer source.Close()
// Authorization is an exact match on one ID. No prefix matching, and not
// AuthorizeMemberOf(trust domain).
allowed := spiffeid.RequireFromString("spiffe://example.org/ns/payments/sa/api")
// Chain verification against the bundle runs first, then the authorizer.
// No hostname or DNS SAN check is involved.
tlsConfig := tlsconfig.MTLSServerConfig(source, source, tlsconfig.AuthorizeID(allowed))
server := &http.Server{Addr: ":8443", TLSConfig: tlsConfig}
// Empty cert and key paths: certificates come from tlsConfig, not from files.
log.Fatal(server.ListenAndServeTLS("", ""))
}
A Go server built with go-spiffe v2 that verifies the client's chain against the SPIRE trust bundle and then accepts only the exact SPIFFE ID spiffe://example.org/ns/payments/sa/api, with no hostname matching.
How do you rotate the trust root and federate trust domains?
Workload SVIDs rotate every few hours, but the CA that signs them is longer-lived and must also be replaced. The difficulty is that every peer validates against the trust bundle, so the bundle must change before the signing key does.
Overlap, not cutover
A trust bundle is a set of CA certificates, not a single root. A safe rotation has three phases:
- Publish. Add the new CA to the bundle while the old CA continues to sign. Wait until every agent and workload has received the updated bundle through the Workload API.
- Activate. Switch signing to the new CA. Peers already trust it, so new SVIDs validate immediately. SVIDs issued by the old CA also remain valid because the old CA is still in the bundle.
- Retire. Remove the old CA only after the last SVID it signed has expired. That means waiting at least the maximum SVID TTL, plus a margin.
SPIRE Server follows this sequence when it rotates its own signing keys. It prepares the next CA ahead of time and distributes it in the bundle before using it.
Skipping the overlap breaks validation across the fleet. If you activate a new CA that peers have not yet received, every freshly rotated workload presents a chain that nobody trusts. If you drop the old CA too early, every workload still holding an older SVID fails at once. Bundle propagation is never instantaneous, so either mistake causes handshake failures across many services rather than one.
Federation
Federation lets workloads in separate trust domains authenticate each other, such as spiffe://payments.internal and spiffe://ledger.internal. Each domain publishes its bundle at a bundle endpoint, and the other domain fetches it periodically. Workloads receive the federated bundles alongside their own bundle and validate a foreign SVID against the bundle for that SVID's trust domain. Authorization policy still decides which foreign SPIFFE IDs are allowed.
How do you test and monitor SVID rotation in production?
Rotation that has never been exercised is a hypothesis. In staging, shrink SVID lifetimes until renewal happens every few minutes instead of every few hours. The accompanying example lowers the SPIRE server's default X.509 SVID TTL and CA TTL for the staging trust domain. You can also set a short TTL on individual registration entries to target one service. Then run normal traffic and load tests for longer than several TTLs.
Short TTLs expose the failures that a one-hour SVID hides:
- Startup-only loading. A service reads its certificate once at startup and never reloads it.
- Stale pooled connections. A connection pool keeps reusing sessions after the peer has rotated, so failures appear only when a pooled connection reconnects.
- Unwatched files. A sidecar writes new files to disk, but the application never watches them.
Also restart SPIRE agents mid-test. Workloads should keep serving on their current SVID and pick up a new one after the agent recovers.
In production, alert on three signals:
- Minimum SVID time-to-expiry per workload. Export this from the workload or its sidecar. Alert when it drops below roughly half the TTL, because that means renewal has stalled while there is still time to act.
- Renewal failures. Track these in agent and server logs and telemetry, including attestation errors and Workload API connection failures.
- TLS handshake errors by reason. Watch for expired certificates and unknown authorities, broken down per caller and callee.
A staging failure costs one engineer an afternoon with a clear reproduction. The same bug in production appears as an outage across many services at once. It often starts at an inconvenient hour, and the first clue is a generic handshake error rather than a pointer to the code that cached the certificate.
# spire-server.conf (HCL) for the STAGING trust domain only.
# Goal: force frequent rotation so renewal bugs surface in hours, not weeks.
server {
bind_address = "0.0.0.0"
bind_port = "8081"
trust_domain = "staging.example.org"
data_dir = "/opt/spire/data/server"
log_level = "INFO"
# Short X.509 SVID lifetime. Agents renew at roughly half the TTL,
# so with 5m, workloads should receive a new SVID about every 2-3 minutes.
# Any service that caches its cert at startup will fail within minutes.
default_x509_svid_ttl = "5m"
# Keep JWT SVIDs short too, if your services use them.
default_jwt_svid_ttl = "5m"
# Short CA lifetime exercises CA rotation and trust bundle updates.
# Keep it well above the SVID TTL. SPIRE warns when ca_ttl is less than
# about 6x the SVID TTL, because SVIDs cannot outlive their signing CA.
ca_ttl = "1h"
}
plugins {
# DataStore, KeyManager, NodeAttestor, etc. unchanged from the normal
# staging config. Only the TTLs above differ.
}
# Production keeps its own config with longer TTLs. Never ship this file
# to the production trust domain.
A staging-only SPIRE server configuration that lowers the default X.509 SVID TTL to five minutes and the CA TTL to one hour, so certificate and CA rotation happen often enough to expose services that never reload their credentials.
When should you avoid running SPIRE, and what are the alternatives?
That failure mode is the real price of running SPIRE yourself. The SPIRE server becomes tier-zero infrastructure. It needs a highly available datastore, because agents can only renew SVIDs while the server is reachable. Upgrades have to be coordinated across server and agents. The upstream CA key has to be protected, backed up, and rotated by someone who understands what breaks when it changes. There is also development work: any library or sidecar that loads certificates only at startup must be changed to reload them, or it will fail as soon as its first SVID expires.
Self-run SPIRE pays off when you need one identity system across heterogeneous platforms, such as VMs, Kubernetes, and bare metal. It also fits multi-cluster or multi-cloud setups where no single provider issues identity everywhere, and teams that need to choose and audit their own attestation logic. Avoid it if you run a small deployment on one platform, or if no team has the capacity to be paged for critical infrastructure.
| Option | Rotation | Identity scope | Operational burden |
|---|---|---|---|
| Self-run SPIRE | Automatic, short-lived SVIDs | Any platform you can attest | High: datastore, upgrades, CA keys |
| Mesh-managed identity | Automatic, handled by the mesh | Workloads inside the mesh | Moderate: you run the mesh instead |
| Cloud-managed workload identity | Automatic, handled by the provider | One provider's platform | Low, but tied to one provider |
| Internal CA, long-lived certs | Manual or scripted, infrequent | Wherever you distribute certs | Low day to day, risky at expiry |
A long-lived internal CA looks cheap until a certificate expires and nobody remembers how to reissue it.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to spiffe mtls microservices, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.