Correlation ID Microservices Guide: Propagate Context via HTTP and Queues

What correlation IDs, request IDs and trace IDs actually are

A correlation ID ties together all work caused by one originating event, such as a checkout click or a webhook. It stays the same across every service, queue and retry. A request ID identifies a single hop, meaning one HTTP call or one message delivery. A retried call gets a new request ID but keeps the correlation ID. W3C Trace Context covers both ideas. Its trace ID plays the correlation role, and its span ID identifies each unit of work, with a parent pointer that links the spans into a tree.

An ID only helps if it survives every hop. If one service drops it, or one consumer ignores message headers, the logs split into fragments that no query can join. That gap usually appears where you most need answers, in async retries and dead-letter handling.

You need this if you run more than one service or any asynchronous processing. Platform engineers who own the shared HTTP, messaging and logging libraries should implement it once in those libraries, so each service team doesn't build its own version.

Copyclient
  |  X-Correlation-ID? (missing -> generate)
  v
[edge]      READ/validate -> STORE in context -> LOG -> INJECT header
  |  HTTP
  v
[orders]    READ header -> STORE -> LOG -> INJECT into message headers
  |  publish
  v
(queue)     ID travels in message headers, not the body
  |  consume (possibly retried)
  v
[billing]   READ message header -> STORE -> LOG

The rest of this article follows this diagram, one boundary at a time.

Copy client
   |  POST /checkout  (no X-Correlation-ID -> edge generates C1)
   v
+--------------------------------------------------------------+
| edge      READ/validate -> STORE in ctx -> LOG -> INJECT hdr |
|           log: corr=C1 req=R1                                |
+--------------------------------------------------------------+
   |  HTTP  X-Correlation-ID: C1   (new request ID R2)
   v
+--------------------------------------------------------------+
| orders    READ hdr -> STORE in ctx -> LOG -> INJECT msg hdr  |
|           log: corr=C1 req=R2                                |
+--------------------------------------------------------------+
   |  publish  headers: {correlation-id: C1}  body: order data
   v
+--------------------------------------------------------------+
| (queue)   ID travels in message headers, not in the body     |
+--------------------------------------------------------------+
   |  delivery #1 (req R3) fails -> redelivery #2 (req R4)
   v
+--------------------------------------------------------------+
| billing   READ msg hdr -> STORE in ctx -> LOG                |
|           log: corr=C1 req=R3                                |
|           log: corr=C1 req=R4                                |
+--------------------------------------------------------------+

 log query corr=C1  ->  R1, R2, R3, R4 joined into one timeline

One checkout event flowing from the edge through the orders service and a queue to the billing consumer: the correlation ID C1 is read, stored, logged and injected at every boundary while each hop and redelivery gets its own request ID.

How correlation ID propagation works

The mechanism is four operations that repeat at every boundary. No central service is involved, and nothing is stored outside your existing logs.

1. Originate. The first component to see an event, usually the edge service or the producer of a scheduled job, checks for an incoming ID. It accepts a valid ID from a trusted caller or generates a new random one. This is the only point where a new correlation ID is created for the event.
2. Store in execution context. The ID goes into request-scoped storage, such as a thread-local, async-local storage or an explicit context object. Code deeper in the call stack can read it without it being passed as a function argument.
3. Log. The logging library reads the ID from that context and writes it as a structured field on every log line the request produces. Engineers never add it by hand.
4. Inject and extract. Whenever work leaves the process, as an HTTP call or a published message, a shared client wrapper copies the ID from context into a header. The receiving service extracts it from that header and repeats steps 2 to 4.

Because every service writes the same value into the same log field, your log backend can return all lines for one event with a single filter on that field. It does not need to understand the call graph. Tracing systems that use W3C Trace Context follow the same pattern, but they also create a new span ID at each hop and record the parent span ID, which lets them reconstruct the call tree as well as the flat list.

The sections that follow cover each step in detail: validation at origin, context and logging middleware, injection into queues, and keeping context intact when work moves between threads.

Who should use correlation IDs

• Teams running two or more services that call each other. As soon as one user action produces log lines in more than one process, a shared ID is the cheapest way to join them.
• Any system with asynchronous processing, even a single service with a background worker. Queues, retries and dead-letter handling separate cause and effect in time, so log timestamps alone stop being enough to connect them.
• On-call engineers and support staff. On-call can reconstruct one request across services during an incident. Support can take an ID a customer quotes from an error page or response header and hand it to engineering.
• Platform teams that own shared libraries. The work belongs in the common HTTP client, message producer, consumer and logging setup. Individual service teams then get propagation by using those libraries, not by writing it themselves.
• Teams planning to adopt distributed tracing later. Starting with the W3C traceparent format, as described in the alternatives section, gives you log correlation now and avoids a migration when tracing arrives.

If you already run OpenTelemetry with automatic instrumentation across your services, you already have this mechanism. In that case, the job is making sure the trace ID reaches your log lines, not building a second ID scheme.

Ingress rules: accepting, validating and echoing the ID at the edge

The edge service is the only place that decides whether an inbound ID becomes the ID. Every service behind it should trust whatever the edge hands down. That makes the rule at the edge simple. Accept an incoming ID only from callers you control, such as another internal service, your own gateway or a partner on an authenticated channel. For anything arriving from the public internet, generate a fresh ID.

Even trusted values need validation, because a bug upstream can produce garbage just as easily as an attacker can. For traceparent, that means checking the following:

• The header has exactly four dash-separated fields.
• The trace ID is 32 lowercase hex characters and is not all zeros.
• The parent ID is 16 hex characters and is not all zeros.
• The total length is 55 characters for version 00.

Reject anything else and generate a new value. Don't try to repair it.

When you discard a client-supplied value, keep it in a separate log field such as client_request_id. Truncate it and escape it first. Support staff can still search for what the customer saw, and that value never becomes a join key across services.

Echo the final ID in a response header, for example X-Request-Id. A customer can then paste it into a ticket, and on-call can find the full request path with one query.

Validation exists because of the trade-off. If you trust public IDs blindly, you invite log injection through embedded newlines, oversized headers that bloat every log line, and deliberately reused IDs that merge unrelated requests into one confusing trace. Losing the ability to pick your own ID from curl is a small cost by comparison.

Copyimport { randomBytes } from "node:crypto";
import type { Request, Response, NextFunction } from "express";

// Version 00 layout: 00-<32 hex trace id>-<16 hex parent id>-<2 hex flags>, 55 chars.
const TRACEPARENT_V00 = /^00-([0-9a-f]{32})-([0-9a-f]{16})-[0-9a-f]{2}$/;
const ALL_ZEROS = /^0+$/;

// Placeholder trust check. Prefer an mTLS identity or a gateway-signed marker.
// Note: Node may report IPv4 peers as "::ffff:10.20.x.x".
function isTrustedCaller(req: Request): boolean {
  const addr = req.socket.remoteAddress ?? "";
  return addr.startsWith("10.20.") || addr.startsWith("::ffff:10.20.");
}

function isValidTraceparent(value: string): boolean {
  const m = TRACEPARENT_V00.exec(value);
  return m !== null && !ALL_ZEROS.test(m[1]) && !ALL_ZEROS.test(m[2]);
}

function newTraceparent(): string {
  // Fresh random IDs; flags "01" marks the trace as sampled.
  return `00-${randomBytes(16).toString("hex")}-${randomBytes(8).toString("hex")}-01`;
}

export function correlationId(req: Request, res: Response, next: NextFunction) {
  const incoming = req.get("traceparent");
  const accepted = incoming !== undefined && isTrustedCaller(req) && isValidTraceparent(incoming);
  const traceparent = accepted ? incoming : newTraceparent();

  // Keep a rejected value for support searches, truncated and escaped so it
  // cannot inject newlines into logs. It is never used as a join key.
  if (incoming !== undefined && !accepted) {
    res.locals.clientRequestId = JSON.stringify(incoming.slice(0, 128));
  }

  res.locals.traceparent = traceparent; // handed to downstream calls
  res.locals.traceId = traceparent.split("-")[1];
  res.setHeader("X-Request-Id", res.locals.traceId); // quotable in tickets
  next();
}

Express middleware that accepts a traceparent header only from trusted callers when it passes strict version-00 format checks, generates a fresh one otherwise, stores any rejected client value in a separate escaped field, and echoes the trace ID in X-Request-Id.

How to implement HTTP propagation and log correlation as middleware

The pattern has three parts, and each sits in exactly one place so handlers never touch the ID.

Inbound. A middleware at the top of the request pipeline reads the validated ID, or generates one, and stores it in request-scoped context. That context is a thread-local in thread-per-request servers, async-local storage in event-loop runtimes, or an explicit context object where the language passes one through calls. The middleware also clears the context when the request ends, so a pooled thread cannot leak one request's ID into the next.

Outbound. Every service-to-service call goes through one wrapped HTTP client or interceptor. The interceptor reads the current ID from context and sets the header on the outgoing request. Handlers that build their own client bypass propagation silently, so make the wrapped client the only one your codebase can easily construct.

Logging. The same middleware writes the ID into an MDC-style logging context. The formatter then emits it as a dedicated structured field, such as correlation_id, on every line. Do not interpolate it into message text. Embedded IDs break whenever someone rewords a log message, and they force substring matching.

The difference shows up during an incident. Without the structured field, you grep for the ID across the gateway, orders and billing logs separately and hope every line includes it. With the field, one filter such as correlation_id = "7f3a…" returns the ordered lines from all three services.

Some log backends charge more to index and store high-cardinality fields. If yours does, tune indexing or retention instead of dropping the field.

Copyimport contextvars, json, logging, uuid
import requests

correlation_id = contextvars.ContextVar("correlation_id", default="-")

class CorrelationFilter(logging.Filter):
    # MDC-style: copy the current ID onto every log record automatically.
    def filter(self, record):
        record.correlation_id = correlation_id.get()
        return True

class JsonFormatter(logging.Formatter):
    # Emit the ID as a dedicated field, never interpolated into msg.
    def format(self, record):
        return json.dumps({"level": record.levelname, "service": "orders",
                           "correlation_id": record.correlation_id,
                           "msg": record.getMessage()})

class PropagatingSession(requests.Session):
    # The one client services use. Handlers never set the header themselves.
    def request(self, method, url, **kwargs):
        headers = dict(kwargs.pop("headers", None) or {})
        headers.setdefault("X-Correlation-ID", correlation_id.get())
        return super().request(method, url, headers=headers, **kwargs)

def run_with_correlation(handler, incoming_id=None):
    # Inbound middleware. incoming_id must already be validated.
    token = correlation_id.set(incoming_id or str(uuid.uuid4()))
    try:
        return handler()
    finally:
        correlation_id.reset(token)  # no leakage into the next request

out = logging.StreamHandler()
out.addFilter(CorrelationFilter())
out.setFormatter(JsonFormatter())
log = logging.getLogger("orders")
log.addHandler(out)
log.setLevel(logging.INFO)

A minimal Python sketch where a contextvar holds the request's correlation ID: a logging filter and JSON formatter attach the ID to every line as a structured correlation_id field, and a wrapped requests Session injects it into every outbound call.

Propagating context through message queues, retries and batches

A queue breaks the call chain that HTTP middleware relies on, so the ID has to travel with the message itself. Put it in message headers or attributes, not the payload body. Headers can be read and written without deserializing the business data, and the schema stays free of transport concerns. Set the header in one shared producer wrapper so no team has to remember it. In the consumer, restore the ID into context before any handler code runs, so every log line the handler emits carries it.

Some brokers or legacy message formats have no header support. The fallback is an envelope field wrapped around the payload. This works, but it couples your schema to an infrastructure concern, and every consumer must parse the envelope. Headers also disappear in places you might not expect. Some bridges, connectors and dead-letter paths do not copy them. Check each hop explicitly rather than assuming the ID survives.

A retried message usually keeps its correlation ID and gets a new attempt or span ID. That lets you group all attempts together while still telling them apart.

Batch consumers must set context per message, not per batch. Otherwise every message in the batch is logged under the first message's ID. In tracing systems, span links are the native answer: one consumer span links to the many producer spans it processed.

Do not reuse one ID everywhere. Long-lived workflows, sagas and scheduled jobs that act on many origins are better served by a new ID with an explicit link to the parent. Reusing an ID would collapse unrelated work into one confusing timeline.

Copyimport contextvars, logging, uuid

correlation_id = contextvars.ContextVar("correlation_id", default=None)
HEADER = "x-correlation-id"

class CorrelationFilter(logging.Filter):
    # Attach the current ID to every log record the handler emits.
    def filter(self, record):
        record.correlation_id = correlation_id.get() or "-"
        return True

def publish(send, topic, body, headers=None, attempt=1):
    # Shared producer wrapper: the ID travels in headers, never in the body.
    headers = dict(headers or {})
    headers.setdefault(HEADER, correlation_id.get() or str(uuid.uuid4()))
    headers["x-attempt"] = str(attempt)  # a retry reuses the ID, bumps the attempt
    send(topic, body, headers)

def extract_id(msg):
    headers = msg.get("headers") or {}
    if HEADER in headers:
        return headers[HEADER]
    # Legacy fallback: envelope field. Couples the schema to infrastructure.
    body = msg.get("body")
    if isinstance(body, dict):
        return body.get("meta", {}).get("correlation_id")
    return None

def consume_batch(messages, handler):
    for msg in messages:
        # Set context per message, not per batch, before handler code runs.
        token = correlation_id.set(extract_id(msg) or str(uuid.uuid4()))
        try:
            handler(msg)
        finally:
            correlation_id.reset(token)  # never leak one message's ID into the next

A broker-agnostic Python producer wrapper that writes the correlation ID and attempt number into message headers, and a batch consumer that restores the ID into a contextvar for each message before its handler runs, with an envelope-field fallback for messages without headers.

Keeping context across thread pools, executors and callbacks

Middleware sets the ID on the thread that handles the request, and most logging libraries store it in thread-local state such as a mapped diagnostic context. That works until the work moves to another thread. A task submitted to a thread pool runs on a worker that was created long before the request arrived and holds whatever context the previous task left behind, or none at all. The same applies to scheduled jobs, completion callbacks fired by an I/O library, and reactive pipelines, where one logical request can hop across several threads.

Capture at submit, restore at run, clear after

The fix is to wrap the executor so that every submitted task carries its own context:

1. At submit time, on the caller's thread, take a snapshot of the current context: the correlation ID and any other propagated fields.
2. At execution time, on the worker thread, install that snapshot before running the task.
3. In a finally block, restore the worker's previous context or clear it, so the next task does not inherit a stale ID.

The accompanying code example shows this wrapper and a unit test. The test sets an ID, submits a task that records the ID it sees, and asserts that the worker observed the same value. It then submits a second task with no ID set and asserts that the worker sees nothing.

Expect this to be the main ongoing maintenance cost of correlation IDs. When context is lost, nothing fails. Log lines simply show up with no ID, and you usually find out during an incident. Thread-local storage also breaks without warning in async runtimes, where many tasks share one thread, and in hand-rolled pools that skip the wrapper. Prefer the runtime's own task-scoped context mechanism where one exists, and route every pool through a single factory that applies the wrapper.

Alternatives: custom headers versus W3C traceparent and OpenTelemetry

Everything so far assumed a home-grown header, but there are three realistic choices, and the one you pick determines how much you rewrite when distributed tracing arrives.

The middle option costs little more than a custom header. It also avoids a dead end. When tracing lands, the trace ID becomes the identifier engineers search for. A separate correlation ID then forces every query to join two schemes. Log the trace ID as your correlation field from day one. If clients still send X-Correlation-ID, record it as a secondary attribute at ingress.

W3C Baggage, carried in the baggage header, holds key-value context such as a tenant or feature-flag cohort. It is copied onto every downstream hop and every message, so it needs to stay small. Each byte is repeated on every call, and large headers can exceed proxy limits and be rejected. At trust boundaries, strip baggage and any other non-standard context fields. That applies to outbound calls to third parties and to inbound requests from untrusted clients. Internal service names, user IDs or email addresses in baggage go to whoever receives the request, and inbound baggage is attacker-controlled input.

When to avoid correlation IDs, or a particular use of them

• A single service with no asynchronous work. An ordinary per-request ID in the access log and application logs already gives you everything a correlation ID would. A propagation library adds code without adding information.
• When you already have distributed tracing. If OpenTelemetry or another W3C Trace Context implementation is in place, do not introduce a separate custom correlation ID. Put the existing trace ID into your logs instead. Two parallel schemes mean every query has to join them, and they drift apart wherever one is propagated and the other is not.
• Long-lived workflows, sagas, scheduled jobs and fan-in batches. Avoid carrying one ID through work that spans many originating events or runs for days. Start a new ID and record a link to the parent IDs, as described in the queue section.
• Calls to third parties. Avoid sending your internal ID, or any baggage, to external systems unless you have agreed on it with that party. Return only what you deliberately echo to clients, such as the response header.
• As an authorization, deduplication or idempotency key. A correlation ID is a diagnostic label. Retries deliberately share it, and at the public edge it may originate from untrusted input, so it should not decide access or whether an operation is applied. Use a dedicated idempotency key for that.
• Carrying data. Do not encode user IDs, emails, tenant names or other meaningful values into the ID itself. Keep it random and opaque, so that it is safe to appear in logs, tickets and response headers.

Verification, development cost and engineering time saved

Enforce propagation as a contract

Treat propagation as a tested contract. A contract test sends a request with a fixed, known ID to service A. Service A calls service B over HTTP and publishes a message that service C consumes. The test then asserts three things:

• The ID appears in log lines from all three services.
• B's inbound request carried the ID in its header.
• The queued message carried the ID in its headers.

Run this test in CI against containerised dependencies. A refactor that drops a header should fail the build, not surface during an incident.

Pair the test with a lint or code-review rule that forbids constructing raw HTTP clients or producers outside the shared wrappers. The wrappers only cover code that uses them. Cron scripts, ad hoc direct clients and third-party SDK callbacks bypass them without any error, so the contract test and reviewers must look for them specifically.

Cost versus time saved

The up-front cost includes:

• A shared library per runtime.
• Context handling for each executor and callback model.
• Ongoing log storage for an extra field on every line.

Illustrative estimate (assumed numbers, not a benchmark): Hours saved per year = cross-service incidents per month × 12 × hours saved per incident.

Assume 6 incidents or escalations a month and 1.5 engineer-hours saved on each, mostly from not matching timestamps by hand: 6 × 12 × 1.5 = 108 hours/year.

Set that against an assumed build cost of 80 hours per runtime × 2 runtimes = 160 hours, plus maintenance and storage. With these inputs, the work pays back in roughly 18 months. The payback comes sooner as incident volume grows.

This is not worth the effort for a single service with no async work. In that case, an ordinary request ID in the access log already gives you the correlation.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to correlation id microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.