Correlation IDs in Microservices: traceparent for HTTP and Queues

What is a correlation ID in microservices, and how is it different from a trace ID?

A correlation ID is a single identifier attached to everything caused by one originating request: the API call, the downstream HTTP calls it triggers, the messages it publishes, and the consumer work those messages start minutes later. Every service logs it, so one search returns the whole story.

A trace ID serves the same grouping purpose but comes with span IDs. Each unit of work gets its own span, and each span records its parent. Together they form a causal tree, so you can see that the payment call happened inside checkout and took most of its time.

A business key, such as an order ID, is a different thing. One order can span many requests (create, pay, refund), and one request can touch many orders. Log it alongside the trace ID, never instead of it.

Correlation pays off once you run more than a handful of services, or as soon as any work crosses an async queue. At that point grepping by timestamp stops working. Adopt it before your first cross-service incident, not after.

The limit: a correlation ID tells you which logs belong together. It does not tell you order or duration between steps. For that you still need real span instrumentation.

How does the W3C traceparent header work?

A traceparent header is four lowercase hex fields joined by dashes:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

• version (00): format version. ff is invalid.
• trace-id (32 hex chars): identifies the whole request across every service. All zeros is invalid.
• parent-id (16 hex chars): the span that made this call. Each hop replaces it with its own span ID. All zeros is invalid.
• trace-flags (2 hex chars): bit 0x01 means "sampled".

The trace-id stays the same for the entire request and is already propagated by tracing libraries, so it can serve as your correlation ID. You don't need a second header.

The companion tracestate header carries vendor-specific key-value pairs, such as sampling hints. Pass it along unchanged. Never use it for identity.

The sampled bit matters during incidents. An unsampled request still carries a valid trace-id, and your logs will record it, but no spans are exported. If you search for a trace and find nothing, check the logs before concluding that the request never happened.

The ingress rule

At each entry point, apply this rule:

• If the inbound traceparent is well formed, accept it.
• If the header is missing or malformed, generate a fresh one.
• At the network edge, decide deliberately whether to trust IDs from outside.

Trusting external IDs keeps the link to client-side traces. The cost is that clients can choose IDs, collide them with other requests, or force sampling. Regenerating at the edge prevents that, but you lose the client link. A common compromise is to regenerate at the edge and record the client's original trace-id as a span attribute or log field.

Copyimport re
import secrets

# version-traceid-parentid-flags, lowercase hex only; later versions may append fields
_TP = re.compile(r"^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})(-.*)?$")

def parse_traceparent(header):
    m = _TP.match(header.strip()) if header else None
    if not m:
        return None
    version, trace_id, parent_id, flags, rest = m.groups()
    if version == "ff" or (version == "00" and rest):  # v00 has exactly four fields
        return None
    if trace_id == "0" * 32 or parent_id == "0" * 16:  # all-zero IDs are invalid
        return None
    return trace_id, parent_id, flags

def new_traceparent(sampled):
    # 16 random bytes for trace-id, 8 for parent-id, encoded as lowercase hex
    flags = "01" if sampled else "00"
    return f"00-{secrets.token_hex(16)}-{secrets.token_hex(8)}-{flags}"

def ingress(header, trust_external, sample_new=False):
    """Return (traceparent to continue from, client trace-id to log or None)."""
    parsed = parse_traceparent(header)
    if parsed and trust_external:
        trace_id, parent_id, flags = parsed
        # Keep the client's trace-id; re-emit as version 00. The tracing library
        # still creates a local span, which replaces parent-id on outbound calls.
        return f"00-{trace_id}-{parent_id}-{flags}", None
    # Missing, malformed, or untrusted: start a fresh trace and choose sampling
    # ourselves so clients cannot force it on.
    client_trace_id = parsed[0] if parsed else None
    return new_traceparent(sample_new), client_trace_id

# Edge service: regenerate, but keep the client's ID as a log field / span attribute
tp, client_id = ingress("00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", trust_external=False)
print(tp, "client_trace_id=", client_id)

A Python sketch of the ingress rule: validate traceparent per the version-00 rules, accept it when trusted, and otherwise generate a fresh trace while keeping the client's original trace-id for logging.

How do you propagate the correlation ID across HTTP hops?

Propagation has two halves, and both have to run on every request. On the way in, server middleware reads traceparent (and tracestate), parses it, starts a server span whose parent is the incoming span-id, and stores the result in request-scoped context. On the way out, every HTTP client reads that context and writes a fresh traceparent with the same trace-id and the current span-id as the parent.

Where the context lives depends on the runtime. Java uses a ThreadLocal, .NET uses AsyncLocal, Python uses contextvars, Node uses AsyncLocalStorage, and Go passes an explicit context.Context argument. A concrete case is an Express middleware that extracts the header and runs the rest of the handler inside AsyncLocalStorage.run(). A wrapped fetch then reads from that store and sets the header on each outbound call. Handlers do not need to know about either step. That only holds if every outbound call goes through the wrapped client.

The failures happen where code leaves the instrumented path:

• Hand-built clients: raw sockets or a second HTTP library nobody wrapped.
• Retry wrappers: they rebuild the request and drop headers they did not create.
• Thread pool handoffs: a ThreadLocal does not follow work submitted to an executor.
• Async callbacks: the callback is registered under one context and fires under another.
• Background goroutines or tasks: they are started with context.Background() instead of the request context.
• Scheduled jobs: there is no inbound request, so they need a deliberate new root.
• Third-party SDKs: they make their own calls with their own client.

Each of these fails silently. The downstream service sees no traceparent and starts a new trace, and both halves look valid on their own. Nothing raises an error. You only notice when you look for a trace during an incident and it stops at the hop you forgot.

Copyconst { AsyncLocalStorage } = require('node:async_hooks');
const { randomBytes } = require('node:crypto');
const express = require('express');

const traceStore = new AsyncLocalStorage();
// W3C traceparent: version-traceid(32 hex)-parentid(16 hex)-flags(2 hex)
const TP = /^00-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/;

// Inbound half: parse the header, start a server span, scope it to this request.
function traceMiddleware(req, res, next) {
  const m = TP.exec(req.get('traceparent') || '');
  const ctx = {
    traceId: m ? m[1] : randomBytes(16).toString('hex'), // no valid header: new root
    parentId: m ? m[2] : null,                           // caller's span-id
    spanId: randomBytes(8).toString('hex'),              // this server span
    flags: m ? m[3] : '01',
    tracestate: m ? req.get('tracestate') : undefined,
  };
  traceStore.run(ctx, next); // handlers and their awaits run inside ctx
}

// Outbound half: only calls made through this wrapper carry the trace.
async function tracedFetch(url, init = {}) {
  const ctx = traceStore.getStore();
  const headers = new Headers(init.headers);
  if (ctx) {
    // Same trace-id, current span-id as the parent of the next hop.
    headers.set('traceparent', `00-${ctx.traceId}-${ctx.spanId}-${ctx.flags}`);
    if (ctx.tracestate) headers.set('tracestate', ctx.tracestate);
  }
  return fetch(url, { ...init, headers });
}

const app = express();
app.use(traceMiddleware);
app.get('/orders/:id', async (req, res) => {
  // Handler knows nothing about tracing; a plain fetch() here would split the trace.
  const r = await tracedFetch(`);
  res.json(await r.json());
});
app.listen(3000);

An Express middleware extracts traceparent into AsyncLocalStorage and a wrapped fetch injects it on outbound calls, so any request that bypasses the wrapper silently starts a new trace.

How do you carry trace context through message queues?

Queues break propagation in the same silent way, because there is no HTTP layer to carry headers for you. At publish time, inject the current context into the broker's message headers or attributes. In the consumer, extract it before any processing starts. Keep it out of the payload. The payload belongs to the domain contract, and schema validators, consumers in other languages, and archived data all have to tolerate a field that is really transport metadata.

Batch consumers need care. One poll can return messages from dozens of traces. If you set context once per batch, every log line inherits the first message's trace. Set the context per message, and clear it afterwards.

CopyHTTP ──traceparent──▶ Publisher ──headers──▶ Broker ──▶ Consumer
                                                          │
                     continue: same trace-id, child span ◀┤
                     link: new trace-id, link → producer ◀┘

Linking is often the more honest choice. A trace that stays open for six hours, or that branches into a thousand consumers, distorts latency views and buries the original request.

Brokers also differ in header support and size limits, and in whether headers survive bridges, connectors and replays. Some legacy paths have no headers at all, which forces you to fall back to an envelope field. That fallback couples every consumer to the envelope format.

Copyfrom opentelemetry import trace
from opentelemetry.context import Context
from opentelemetry.propagate import extract, inject
from opentelemetry.trace import Link, SpanKind

tracer = trace.get_tracer("orders-service")

def publish(broker, topic: str, payload: bytes) -> None:
    with tracer.start_as_current_span(f"{topic} publish", kind=SpanKind.PRODUCER):
        headers: dict[str, str] = {}
        inject(headers)  # writes traceparent (and tracestate) for the current span
        # Context rides in transport headers; the payload stays the domain contract.
        broker.send(topic, payload, headers=headers)  # placeholder broker client

def consume_batch(messages, handle, link_only: bool = False) -> None:
    # One poll can mix many traces, so extract and scope context per message.
    for msg in messages:
        producer_ctx = extract(msg.headers)  # assumes headers decoded to dict[str, str]
        if link_only:
            # New trace-id, with a link back to the producer's span.
            producer = trace.get_current_span(producer_ctx).get_span_context()
            links = [Link(producer)] if producer.is_valid else []
            parent = Context()  # empty context means no parent, so a new root span
        else:
            # Continue: same trace-id, consumer span is a child of the producer span.
            links, parent = [], producer_ctx
        # The with-block makes this span current only for this message and restores
        # the previous context on exit, so the next message's logs do not inherit it.
        with tracer.start_as_current_span(
            "orders process", context=parent, kind=SpanKind.CONSUMER, links=links
        ):
            handle(msg)

Using the OpenTelemetry Python API, a producer injects traceparent into message headers. A batch consumer extracts the context for each message and either continues the producer's trace or starts a new trace that links back to it.

How do you get the trace ID onto every log line?

Once context is propagated, logging should not need any per-call effort. Configure the logger, not the call sites: a log processor or formatter reads trace_id and span_id from the active context (the thread-local or async-local storage your tracing library already maintains) and writes them as top-level structured fields on every record. Use the same field names in every service so one query works everywhere.

This replaces passing a requestId argument through every function signature. That pattern breaks in predictable places: helper libraries that never accepted the parameter, background callbacks, and the one code path someone forgot. With context-driven enrichment, any log emitted while a span is active gets the IDs automatically. Make sure the enrichment covers the paths that matter most during an incident:

• Global error handlers and exception middleware
• Panic recovery and uncaught-exception hooks, which often run after request-scoped state is gone
• Framework, HTTP client, ORM and driver logs, routed through the same configured logger

The payoff comes during triage. Filtering on a single trace_id returns the gateway entry, each downstream call, the queue publish, the consumer's processing and the final error, ordered by timestamp across services. You no longer have to line up timestamps by hand.

The cost is real. Every line gets longer, so ingest and storage grow. trace_id is also extremely high-cardinality. Log backends index it well for search, but you should never use it as a metrics label. If you do, every request creates a new time series, and your metrics store will degrade or reject writes. Use exemplars or span links to connect metrics to traces instead.

Copyimport json, logging, sys
from opentelemetry import trace

class TraceContextFilter(logging.Filter):
    """Read IDs from the active span (contextvars-backed) for every record."""
    def filter(self, record):
        ctx = trace.get_current_span().get_span_context()
        record.trace_id = format(ctx.trace_id, "032x") if ctx.is_valid else None
        record.span_id = format(ctx.span_id, "016x") if ctx.is_valid else None
        return True  # enrich only, never drop

class JsonFormatter(logging.Formatter):
    def format(self, record):
        doc = {
            "ts": self.formatTime(record),
            "level": record.levelname,
            "logger": record.name,
            "msg": record.getMessage(),
            # Same top-level field names in every service so one query works everywhere
            "trace_id": getattr(record, "trace_id", None),
            "span_id": getattr(record, "span_id", None),
        }
        if record.exc_info:
            doc["exception"] = self.formatException(record.exc_info)
        return json.dumps(doc)

handler = logging.StreamHandler(sys.stdout)
# Filter on the handler, not a logger: records from library loggers
# (HTTP client, ORM, driver) propagate to root and pass through it too.
handler.addFilter(TraceContextFilter())
handler.setFormatter(JsonFormatter())
logging.basicConfig(level=logging.INFO, handlers=[handler])

def log_uncaught(exc_type, exc, tb):
    # Last-resort hook: the span has usually ended by now, so trace_id may be null.
    # Log errors in request middleware too, while the span is still active.
    logging.getLogger("uncaught").critical("unhandled exception", exc_info=(exc_type, exc, tb))

sys.excepthook = log_uncaught

A Python logging setup that uses a handler-level filter to copy the active OpenTelemetry trace_id and span_id onto every JSON log line, including library logs and uncaught exceptions, so call sites never pass a request ID.

How do you test that trace context propagation is not broken?

Propagation breaks quietly. A new HTTP client gets constructed by hand, a consumer is refactored to batch messages, or a library upgrade drops a header, and nothing fails until an incident shows a trace that stops partway. To catch this early, treat propagation as a contract and test it.

A contract test across one HTTP hop and one queue hop

Pick a fixed, valid traceparent with a recognizable trace-id and send it to the entry service. That service should call a second service over HTTP, and the second service should publish to a queue that a consumer reads. Route logs from all three into an in-memory sink or a test log collector, then assert three things:

• The same trace-id appears on log lines from every service.
• The parent-id differs at each hop, which proves each service created its own span instead of forwarding the header unchanged.
• If the consumer deliberately starts a new trace, it records a span link or attribute that references the original trace-id.

Run this test in CI against real broker and HTTP clients. Mocks will hide the bugs you are trying to find.

Make bypassing hard

Add a lint rule that flags direct construction of HTTP clients or broker producers outside your instrumented wrapper package. Add a matching code-review checklist item for new outbound integrations.

Cost

The shared middleware, client wrappers and this test are a one-time investment, plus maintenance. Illustrative estimate: if building them takes two engineer-weeks, and each major incident spends four engineer-hours reconstructing request paths by hand from timestamps, the work pays for itself after about 20 incidents. It also shortens each of those incidents.

When should you avoid traceparent, and what are the alternatives?

traceparent has costs. Its format is strict: a version, a 32-hex-character trace ID, a 16-hex-character parent ID and a flags field. A homegrown header accepts any string, so a truncated or garbage value passes through unnoticed. With traceparent, every receiver must validate the value, drop it if it is malformed, and start a fresh trace. That is extra code to write and extra cases to test.

Full adoption may not fit in three cases:

• Legacy brokers or protocols with no header support. The envelope is the only place the ID can go.
• Systems you cannot modify, such as vendor services. Regenerate the ID at the boundary and log the mapping.
• Very small systems. When three services write to one log store, grouping lines by a single ID may be all you need.

To migrate, accept and emit both headers for a fixed transition window. At the edge, map the legacy ID to traceparent, and record the old value as a log field so earlier searches still work. Set a firm date to remove the legacy header and publish it. Without a deadline, teams tend to support both headers indefinitely.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to correlation id microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.