What is the difference between a correlation ID and a trace ID?
A trace ID identifies one causal execution tree. When a request enters an instrumented system, the first service starts a trace. Every downstream operation it causes, such as HTTP calls, database queries and message publishes, records a span that carries the same trace ID. Under W3C Trace Context, a trace ID is 16 bytes, written as 32 hex characters.
A span ID identifies a single operation inside that tree. It is 8 bytes. Each span also records its parent's span ID, which is how a tracing backend rebuilds the call graph rather than just a flat list of events.
A correlation ID (often called a request ID) is defined by your application and has no standard meaning. It is usually generated at the edge, at a load balancer, gateway or first service, and passed in a custom header such as X-Request-ID. Its meaning depends on your convention. It might cover one HTTP request, one user action, or one business process that spans days. That flexibility is useful, but nothing enforces it, so every service has to forward the header deliberately.
| Trace ID | Span ID | Correlation ID | |
|---|---|---|---|
| Scope | One causal tree of operations | One operation | Whatever your convention says |
| Defined by | W3C Trace Context / OpenTelemetry | W3C Trace Context / OpenTelemetry | Your team |
| Propagation | traceparent header, handled by instrumentation |
Changes at every hop; sent as the parent ID | Custom header, forwarded by your code |
| Lifetime | Ends when the causal chain ends | Duration of one operation | Can outlive any single trace |
| Groups | All spans and logs from one execution | Logs from one unit of work | Arbitrary related activity |
Who should use trace IDs and correlation IDs?
Any team running more than one service that handles a single user request should put a shared identifier in its logs. Once a request crosses a process boundary, you can no longer read one log file to understand what happened. The value grows with the number of hops, the number of teams owning those hops, and how often you debug production.
- Teams with synchronous microservices (HTTP or gRPC chains) should use the W3C trace ID as their primary request identifier and add it to every log line. This is the default for most teams.
- Teams with queues, background jobs, retries or long-running workflows should also define a correlation ID with a written meaning, because one business action can produce several separate traces.
- Platform and SRE teams that own gateways, proxies and shared logging libraries are the ones who make it work in practice. They decide which headers survive the edge and which fields every service emits.
- Support and customer-facing teams benefit from an ID returned in a response header, so a bug report can be tied to specific logs.
A single monolith with one log stream gets less from this. A per-request ID is still useful there for separating concurrent requests, but full distributed trace propagation adds little until there is a second service.
How does it work end to end?
At a high level, the flow for one request looks like this:
- The edge assigns an identity. The first instrumented component, usually a gateway or the first service, either continues a valid inbound
traceparentor creates a new trace ID. If you use a correlation ID, it is created or accepted here too. - Each service extracts, works, and injects. On receipt, instrumentation extracts the trace context from headers and starts a span with a new span ID whose parent is the caller's span. Before any outbound call or message publish, it injects the current context into the outgoing headers.
- Logs pick up the IDs from the active context. A logging integration or filter reads the current span and writes
trace_idandspan_idas structured fields. A correlation ID, if present, is written as its own field. - Backends store spans and logs separately. Spans go to a tracing backend, logs go to a log store. They are joined later by the shared
trace_idvalue, not by any direct link between the systems. - Debugging is a pivot on one value. From an error log you take the
trace_idand open the trace. From a slow span you take thetrace_idand search the logs.
The sections below cover each step in detail: the header format and propagation, adding IDs to logs, when a correlation ID is needed, how to handle untrusted input and sampling, and how to migrate.
How does W3C Trace Context propagation work with OpenTelemetry?
A traceparent header packs four dash-separated fields into one line:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
- Version (
00): the format version, so parsers can reject or adapt to future layouts. - Trace ID (32 hex chars): shared by every span in the request. This is the value you join logs on.
- Parent span ID (16 hex chars): the span that made this call. The receiver records it as its parent, which is how the call tree is rebuilt.
- Trace flags (
01): a bit field. The lowest bit means "sampled," so downstream services can follow the upstream sampling decision.
tracestate is a companion header that holds vendor-specific key-value pairs. It lets tracing systems carry their own data without changing the fixed traceparent format.
OpenTelemetry handles propagation through propagators. On the sending side, instrumentation calls inject, which writes the current context into a carrier such as HTTP headers or message headers. On the receiving side, it calls extract, which reads the carrier and restores the context before a new span starts. Because instrumented HTTP clients, servers and messaging libraries already make these calls, services do not need to copy headers by hand. Hand-rolled propagation tends to break when someone adds a new client or a new thread pool.
gateway ──inject──▶ service A ──inject──▶ service B ──inject──▶ [queue]
(HTTP hdr) extract (HTTP hdr) extract (msg hdr) │
▼
consumer: extract
For queues, the context travels inside the message itself, so it survives after the HTTP connection that created the message has closed.
from opentelemetry import trace
from opentelemetry.propagate import inject, extract
from opentelemetry.trace import SpanKind
tracer = trace.get_tracer("orders-service")
def publish_order(queue, order_json: str) -> None:
# Producer side (service B): start a span, then copy its context into message headers.
with tracer.start_as_current_span("orders publish", kind=SpanKind.PRODUCER):
headers: dict[str, str] = {}
inject(headers) # adds e.g. {'traceparent': '00-<trace-id>-<span-id>-01'}
queue.send(body=order_json, headers=headers) # placeholder queue client
def handle_message(message) -> None:
# Consumer side: rebuild the upstream context from the message, not from any HTTP request.
parent_ctx = extract(message.headers)
with tracer.start_as_current_span(
"orders process", context=parent_ctx, kind=SpanKind.CONSUMER
) as span:
# Same trace ID as the gateway request: this is the value to put in log lines.
trace_id = format(span.get_span_context().trace_id, "032x")
print(f"trace_id={trace_id} processing order")
# ... business logic ...
Shows a producer calling inject to write traceparent into message headers and a consumer calling extract to continue the same trace, which instrumented messaging libraries do for you.
How do you add trace IDs to logs for log correlation?
Every log line should carry trace_id and span_id as top-level structured fields, not embedded in the message text. Most OpenTelemetry SDKs offer a logging integration that does this automatically. Where one doesn't exist or doesn't fit your logger, a small hook works just as well:
- On each log call, read the active span from the current context.
- If the span context is valid, write its trace ID as 32 lowercase hex characters and its span ID as 16.
- If no span is active, skip the fields. Don't emit zeros, because they match each other in searches and produce false links.
A resulting line looks like this: {"ts":"2026-11-11T09:14:02Z","level":"error","service":"payments","msg":"card authorization failed","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7"}
Using the W3C hex format matters because it lets you search with the exact string your tracing UI shows. This gives you a pivot in both directions. From a slow span, you copy the trace ID and pull every log line it produced. From an error log, you jump to the trace to see which upstream call caused it. Searching that one ID across all services returns the whole request story in a single query, instead of a chain of timestamp-window guesses repeated for each hop.
The cost is that the change has to be made per language and per logging library. Each Go, Java, Python and Node service needs its own configuration, and legacy services are often skipped. Uneven coverage is worse than none, because a search that returns four services looks complete when six were involved. Track which services emit the fields, and treat a missing service as a known gap rather than a quiet one.
import json, logging, time
from opentelemetry import trace
class TraceContextFilter(logging.Filter):
# Copy the active span's IDs onto each record as separate fields.
def filter(self, record):
ctx = trace.get_current_span().get_span_context()
if ctx.is_valid: # no active span: skip the fields, never emit zeros
record.trace_id = format(ctx.trace_id, "032x") # W3C: 32 hex chars
record.span_id = format(ctx.span_id, "016x") # W3C: 16 hex chars
return True # annotate only; never drop the record
class JsonFormatter(logging.Formatter):
converter = time.gmtime # UTC timestamps
def format(self, record):
line = {
"ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%SZ"),
"level": record.levelname.lower(),
"service": "payments",
"msg": record.getMessage(),
}
# Top-level fields, not text inside msg, so search can match them exactly.
for key in ("trace_id", "span_id"):
if hasattr(record, key):
line[key] = getattr(record, key)
return json.dumps(line)
handler = logging.StreamHandler()
handler.addFilter(TraceContextFilter()) # on the handler, so child loggers are covered too
handler.setFormatter(JsonFormatter())
logging.getLogger().addHandler(handler)
tracer = trace.get_tracer("payments")
with tracer.start_as_current_span("authorize_card"):
logging.getLogger("payments").error("card authorization failed")
A Python logging filter that reads the active OpenTelemetry span context and adds trace_id and span_id as top-level JSON fields in W3C hex format, and leaves them out when no span is active.
When is a trace ID alone enough, and when should you avoid a separate correlation ID?
For most teams, the trace ID is the only request identifier they need. If your system is a chain of synchronous HTTP or gRPC calls where one inbound request produces one trace, the trace ID already does everything a correlation ID would: it is generated at the edge, propagated on every hop, and links logs to spans without extra work.
Adding a second ID in that setup costs more than it returns. Every log line now carries two identifiers that mean almost the same thing, and engineers have to remember which one a given dashboard, alert, or runbook expects. The two values drift. One service regenerates the request ID on a retry, another copies it into a field with a slightly different name, and a third only logs one of them. During an incident, someone pastes the request ID into the trace ID search box, gets zero results, and wastes ten minutes concluding the request never arrived. One identifier with one name and one meaning is easier to search, easier to document, and harder to get wrong.
The counter-risk is real, though. Trace-only debugging works only if instrumentation is everywhere. A legacy service without an OpenTelemetry SDK starts a fresh trace or none at all. A proxy or API gateway that forwards only an allowlist of headers drops traceparent. Either one splits the request into disconnected fragments. A plain custom header is sometimes more likely to survive those components, because operators already allowlist it or because the legacy code copies all X- headers forward. Before you drop a request ID, check that every hop on the path actually propagates trace context.
When does a correlation ID earn its place across async hops, retries and workflows?
A trace ID describes one causal tree of spans. A correlation ID describes a business action, and some actions produce more than one trace:
- Fan-out: a request returns, and a background job picks up the work minutes later.
- Retries and replays: a dead-letter replay or a manual re-run often starts a new trace.
- Long-running sagas: steps run over hours, with timers and compensations.
- Customer-facing IDs: support staff need a stable reference that survives sampling and tracing outages.
For messaging, you choose between two approaches. If the consumer continues the producer's trace, every step stays in one tree, but a saga becomes a huge trace that stays open for hours and is hard to render and sample. If the consumer starts a new trace with a span link to the producer, each trace stays small, but the links scatter the story across many traces. That scattering is why teams drift back to a shared correlation ID.
The usual compromise is to use links and also record the correlation ID as a span attribute and a log field on every hop. Then a single query finds every trace and log line for the action.
Baggage propagates the ID automatically, but it is copied into every outbound call, including calls to third parties. Every entry adds header size. Never put user data in it.
| Option | Groups multiple traces | Propagates automatically | Main risk |
|---|---|---|---|
| Trace only | No | Yes | Huge, long-lived traces |
| Span links | Pairwise | Per message | Scattered traces |
| Correlation ID attribute | Yes | No, set it explicitly | Inconsistent adoption |
| Baggage | Yes | Yes | Header size, data leaks |
import logging
from opentelemetry import trace, propagate
from opentelemetry.context import Context
tracer = trace.get_tracer("orders.consumer")
log = logging.getLogger("orders.consumer")
def handle_message(headers: dict, body: bytes) -> None:
# Recover the producer's span context from the W3C traceparent header.
producer_ctx = propagate.extract(headers)
producer_span_ctx = trace.get_current_span(producer_ctx).get_span_context()
links = [trace.Link(producer_span_ctx)] if producer_span_ctx.is_valid else []
# The correlation ID is carried as an explicit message header, not Baggage.
# This consumer's outbound calls, including third-party calls, will not
# forward it unless code adds it on purpose.
correlation_id = headers.get("x-correlation-id", "unknown")
# An empty Context starts a new root trace, so a saga step does not keep
# the producer's trace open for hours. The link records the causal edge.
with tracer.start_as_current_span(
"process order.created",
context=Context(),
kind=trace.SpanKind.CONSUMER,
links=links,
# Searchable span attribute: one query finds every trace for the action.
attributes={"app.correlation_id": correlation_id},
):
# The same ID goes into a log field so log searches match trace searches.
log.info("processing order event", extra={"correlation_id": correlation_id})
# ... business logic ...
A Python OpenTelemetry message consumer that starts a new trace with a span link to the producer, then records the correlation ID as a span attribute and a log field instead of propagating it through Baggage.
Should you trust inbound trace and request ID headers, and how does sampling affect them?
Treat every inbound header as untrusted input. A client-supplied traceparent or X-Request-ID can be spoofed to merge unrelated requests into one trace. It can be stuffed with newlines or oversized strings to poison logs. A client that reuses one value can also collide with other callers' IDs. Simply regenerating everything at the gateway is safer, but it breaks the join with upstream services and partners who already hold their side of the trace.
A practical middle ground:
- Validate format strictly. A
traceparentmust have a 32-hex-character trace ID and a 16-hex-character parent ID, and neither may be all zeros. Request IDs should match a fixed length and character set. If a value fails, drop it and generate a new one rather than sanitizing it. - Trust by source. Continue the trace for authenticated internal callers and known partners. For anonymous public traffic, start a new trace and record the inbound value as a span link or attribute, so it can still be looked up without being trusted.
- Return an ID in a response header so users can quote it in bug reports. A raw trace ID is the simplest choice, but it exposes internal identifiers and ties your support process to your tracing backend. A separate public ID avoids both problems, but you then have to store and maintain the mapping from public ID to trace ID.
Sampling adds a second source of confusion. With head-based sampling, an unsampled request still has a valid trace context, so its logs carry a trace_id even though no spans were exported. Keep logging that ID, because it still joins log lines across services. Also log the sampled flag, so that when an engineer finds no trace for a given ID, they know it was dropped by design and not lost to a broken pipeline.
import logging, random, re, secrets
log = logging.getLogger("gateway")
TRACEPARENT_RE = re.compile(r"^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$")
def parse_traceparent(value):
# Strict validation: reject rather than sanitize anything malformed
m = TRACEPARENT_RE.match(value or "")
if not m:
return None
version, trace_id, parent_id, flags = m.groups()
if version == "ff" or trace_id == "0" * 32 or parent_id == "0" * 16:
return None
return trace_id, parent_id, bool(int(flags, 16) & 0x01)
def handle_edge(headers, caller_is_trusted, sample_rate=0.1):
inbound = parse_traceparent(headers.get("traceparent"))
untrusted_ref = None
if inbound and caller_is_trusted:
# Authenticated internal caller or known partner: continue their trace
trace_id, _, sampled = inbound
else:
# Anonymous traffic: start fresh, keep the valid inbound ID only for lookup
untrusted_ref = inbound[0] if inbound else None
trace_id = secrets.token_hex(16)
sampled = random.random() < sample_rate # head-based decision
span_id = secrets.token_hex(8)
outbound = f"00-{trace_id}-{span_id}-{'01' if sampled else '00'}"
# Always log trace_id, even when unsampled; log the flag so a missing
# trace reads as "dropped by design", not "pipeline broken"
log.info("request accepted", extra={
"trace_id": trace_id,
"trace_sampled": sampled,
"inbound_trace_id_untrusted": untrusted_ref,
})
# Expose an ID users can quote in bug reports (here: the raw trace ID)
response_headers = {"X-Request-ID": trace_id}
return {"traceparent": outbound}, response_headers
A gateway helper that strictly validates inbound traceparent headers, continues traces only for trusted callers, records untrusted IDs as a reference, logs the trace ID and sampled flag on every request, and returns an ID in a response header.
What are the alternatives to trace IDs and correlation IDs?
There are a few other ways to connect log lines across services. Each one works in narrow cases and breaks down as traffic or service count grows.
- Timestamp-window correlation. You find the error, note its time, and search each downstream service's logs in a window around it, filtering by endpoint, user or order ID. This needs no code changes, but under concurrent load many requests overlap in the same window. Clock skew between hosts also makes the window unreliable, and every hop needs its own search.
- Business identifiers as join keys. Searching by order ID, user ID or account ID often works for one-off investigations, because those values already appear in logs. They are not unique per request, though. One user generates many requests, and some services never see the business ID at all, such as an auth service or a shared cache layer.
- Custom IDs only. A home-grown
X-Request-IDwith no tracing gives you log correlation without spans. It is simple to start, but you maintain propagation in every service and client library yourself, and you get no call tree or timing data. - Vendor-specific or older propagation formats. Some tracing tools have their own header formats. They work within that tool's ecosystem, but mixing them with W3C Trace Context means configuring multiple propagators, and switching tools means rework. OpenTelemetry can run several propagators side by side during a transition, which is the usual way to bridge them.
- Centralized request logging at the gateway only. Logging every request at the edge tells you what came in and what went out, but nothing about what happened between services.
For most systems, W3C trace context plus structured log fields is the default, with a correlation ID added only where business actions span several traces. The migration table in the next section compares the main options on interoperability, ongoing cost and lock-in.
How do you migrate from X-Request-ID to OpenTelemetry trace context?
Migration steps
- Turn on W3C propagation without removing anything. Services start accepting and emitting
traceparentwhile still readingX-Request-ID. When the legacy header arrives, record it as a span attribute and as acorrelation_idlog field next totrace_id. Existing dashboards and support runbooks keep working while the new IDs spread. - Measure coverage per service. For each service, track the share of log lines with a valid
trace_idand the share of inbound requests that carrytraceparent. A fleet-wide average can hide the one legacy gateway that drops the header and breaks every trace that passes through it. - Decide the header's fate. Once coverage is high enough for on-call to rely on, choose one of two options. You can stop reading
X-Request-IDentirely. Or you can keep it with a narrower, written meaning, such as a customer-facing support reference that survives retries and async hops.
| Approach | Interoperability | Ongoing cost | Lock-in |
|---|---|---|---|
| OTel trace context, legacy header bridged | Standard across libraries and proxies | Low, mostly one-time | None |
| Custom IDs only | Only your own code understands them | You maintain propagation in every service | Internal |
| Vendor-specific headers | Limited to that vendor's tooling | Rework if you switch tools | High |
Naming and schema checklist
- Use the same field names everywhere:
trace_id,span_idandcorrelation_id. Do not mix in variants liketraceIdorrequest_id. - Use one casing convention for field names across all services.
- Format IDs as lowercase hex, without dashes or prefixes, so text searches match across services.
- Write down what each ID means, who creates it and how long it lives, so on-call engineers do not have to guess.
How can trace IDs in logs save engineering time?
The time savings come from replacing a search per hop with one search per request. Without a shared ID, an engineer reconstructs a request by hand: find the first error, guess a time window, search the next service, decide which of several overlapping requests is the right one, and repeat. With trace_id in every log line, one query returns every line from every instrumented service, and the trace view shows which hop was slow or failed.
The concrete places time is saved:
- Incident triage. On-call goes from an alert or error log straight to the full request path instead of asking each owning team to search their own logs.
- Fewer cross-team handoffs. When the trace shows the failing span, the investigation goes to the right team first.
- Support tickets. A response-header ID quoted in a bug report leads directly to the relevant logs, without back-and-forth about timestamps and account details.
- Async and retry debugging. A correlation ID recorded on every hop finds all traces for one business action, which otherwise means stitching jobs and retries together manually.
Illustrative estimate, with assumed inputs rather than measured results:
- Assumed: a typical failing request touches 6 services.
- Assumed: without a shared ID, narrowing down the right log lines takes about 5 minutes per service, so about 30 minutes per investigation.
- Assumed: with
trace_idin logs, the same investigation takes about 5 minutes, one search plus reading the trace. - Assumed: the team does 10 such investigations a month.
Under those assumptions, the savings are about 25 minutes per investigation, or roughly 4 hours a month. That has to be weighed against the setup cost: per-language logging configuration, gateway and proxy header allowlists, and coverage tracking. Most of that cost is paid once, while the per-investigation savings recur. Your real numbers depend on how many hops your requests cross, how much concurrent traffic your logs contain, and how complete your instrumentation coverage is. Uneven coverage erodes the savings quickly, because every uninstrumented service brings back a manual search.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to correlation id vs trace id, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.