Why does distributed tracing break at Kafka?
Trace context propagation is the mechanism that carries a trace ID and the current span ID from one process to the next, so spans recorded in different services join a single trace. Over HTTP this feels automatic. A synchronous call has one request lifecycle: the client injects context into request headers, the server extracts it, and the caller waits on the same connection for the response. Instrumentation libraries hook both ends of that exchange, so engineers rarely see the handoff.
Kafka removes every part of that arrangement. The producer writes a record to a topic and moves on. A consumer may read it milliseconds or hours later, in a different process, possibly after a restart or rebalance. There is no shared call stack, no open connection, and no response path. Any context the consumer needs has to be serialized into the record itself.
Kafka record headers serve as that carrier. They are key-value pairs stored alongside the record's key and value. The standard format is W3C Trace Context:
traceparentholds the version, trace ID, parent span ID, and trace flags.tracestate(optional) holds vendor-specific data.baggage(optional) holds application key-value pairs, defined by the separate W3C Baggage specification.
[order-service] [topic: orders] [billing-service]
span: "orders publish" ──► record ──► span: "orders process"
inject context headers: extract context
traceparent: 00-<trace-id>-<span-id>-01
value: {...}
Resulting trace: orders publish ── child of / linked from ──► orders process
(same trace ID; consumer references producer span ID)
This matters to backend and platform engineers who run event-driven systems. In those systems, a single user action passes through several producers, topics, and consumers, and without propagated context it shows up as a set of unrelated traces.
+------------------+ +------------------------+ +------------------+
| order-service | | topic: orders | | billing-service |
| (producer) | | partition P, offset N | | (consumer) |
| | | | | |
| span: | send | record | poll | span: |
| orders publish |----->| key: order-123 |----->| orders process |
| kind=PRODUCER | | value: {...} | | kind=CONSUMER |
| | | headers: | | |
| inject() writes | | traceparent | | extract() reads |
| context into | | tracestate (opt) | | context from |
| record headers | | baggage (opt) | | record headers |
+------------------+ +------------------------+ +------------------+
traceparent: 00-<trace-id>-<span-id>-01
| | | |
| | | +-- trace flags (01 = sampled)
| | +-- parent span ID = producer span
| +-- trace ID, shared by both spans
+-- version
Resulting trace (same trace ID end to end):
orders publish [PRODUCER, order-service]
|
+--> orders process [CONSUMER, billing-service]
child of (or linked to) the span ID from traceparent
The producer injects W3C Trace Context into the Kafka record headers, and the consumer extracts it later so its span joins the same trace as the producer's span.
Who should use OpenTelemetry Kafka tracing?
Header-based context propagation is worth adopting if you fit one or more of these descriptions:
- Teams running event-driven services on Kafka. If a user-facing request triggers work that crosses one or more topics before it completes (order placement, payment, notification, fulfilment), you need propagated context to see that work as one trace.
- Platform and SRE teams responsible for incident response. When latency or failures show up downstream of a topic, these teams have to determine which upstream request caused them. Without propagated context, that means correlating timestamps across logs by hand.
- Teams that already use OpenTelemetry for HTTP and gRPC. If your synchronous calls are traced but every Kafka hop starts a new trace, header propagation closes the gap using the same API and backend you already run.
- Service owners with consume-transform-produce pipelines. Services that read from one topic and write to another are where traces fork silently. They benefit most from deliberate context handling.
It is a weaker fit for systems where Kafka mostly carries bulk data with no request behind it, such as log shipping, metrics pipelines, or analytics fan-out. The last section covers when to scale back.
How does Kafka trace context propagation work end to end?
The mechanism has four steps, and each one maps to a section below:
- The producer starts a span and injects context. The producing service starts a
PRODUCER-kind span around the send and makes it current. The configured OpenTelemetry propagator serializes the current context into W3Ctraceparent(plustracestateandbaggagewhen present). A setter writes each field into the record's headers as UTF-8 bytes. - Kafka stores the headers with the record. The broker does not interpret the headers. It persists them alongside the key and value, so the context survives restarts, rebalances, and arbitrary delays between produce and consume.
- The consumer extracts context per record. After polling, the consuming service passes each record's headers to the propagator with a getter that decodes the bytes. This rebuilds a remote span context containing the producer's trace ID, span ID, and sampled flag.
- The consumer span attaches to that context. The consumer starts a
CONSUMER-kind span either as a child of the extracted context (single-record processing) or with a span link to it (batches, replays). It makes that span current while the handler runs, so database calls, HTTP calls, and any onward Kafka sends nest under it and continue the trace.
OpenTelemetry provides the format and the span model. Kafka provides the carrier. Your code, or an instrumentation library, connects the two with a setter, a getter, and correct scoping.
How does the OpenTelemetry propagator API work with Kafka headers?
OpenTelemetry does not know what a Kafka record is. It defines a TextMapPropagator that works against an abstract carrier through two small adapters. On the producer side, inject takes the current context and calls your setter once per field: traceparent, tracestate if present, and baggage. On the consumer side, extract asks your getter for those same keys and rebuilds a context from whatever it finds. The propagator owns the format. Your code only moves strings in and out of the carrier.
For Kafka, the carrier is the record's headers. Header values are byte arrays, not strings. The setter therefore encodes each value as UTF-8 and adds it to the headers. The getter decodes the bytes back to a string and returns null when the key is absent. The getter's keys method should list the header names so propagators that scan for fields can find them.
Kafka allows several headers with the same key, so you need a defined read rule. Re-sending a consumed record, or a client library that also injects, can leave two traceparent entries on one record. A sensible convention has two parts. The setter removes any existing header with that key before adding the new one. The getter reads the last occurrence, which is the most recent write.
Configure the global propagator once at startup, typically as a composite of W3C Trace Context and Baggage (for example through OTEL_PROPAGATORS=tracecontext,baggage). Avoid hand-building traceparent strings. Code that calls the global propagator keeps working when you add a propagator, change formats, or bridge to a legacy header scheme, because none of those changes touch the setter, the getter, or the call sites.
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.propagation.TextMapGetter;
import io.opentelemetry.context.propagation.TextMapSetter;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashSet;
import java.util.Set;
import org.apache.kafka.common.header.Header;
import org.apache.kafka.common.header.Headers;
// The global propagator is configured once at startup (e.g. OTEL_PROPAGATORS=tracecontext,baggage).
// This class only moves strings in and out of Kafka headers; it never builds traceparent itself.
final class KafkaHeaderPropagation {
// Write rule: replace instead of append, so a re-sent record never carries two traceparent values.
static final TextMapSetter<Headers> SETTER = (headers, key, value) -> {
if (headers == null) return;
headers.remove(key);
headers.add(key, value.getBytes(StandardCharsets.UTF_8)); // header values are bytes
};
static final TextMapGetter<Headers> GETTER = new TextMapGetter<>() {
@Override public Iterable<String> keys(Headers headers) {
Set<String> names = new LinkedHashSet<>(); // dedupe repeated keys
for (Header h : headers) names.add(h.key());
return names;
}
@Override public String get(Headers headers, String key) {
if (headers == null) return null;
Header h = headers.lastHeader(key); // read rule: last occurrence is the most recent write
return (h == null || h.value() == null) ? null : new String(h.value(), StandardCharsets.UTF_8);
}
};
static void inject(Headers headers) { // producer: call before send()
GlobalOpenTelemetry.getPropagators().getTextMapPropagator().inject(Context.current(), headers, SETTER);
}
static Context extract(Headers headers) { // consumer: parent context for the processing span
return GlobalOpenTelemetry.getPropagators().getTextMapPropagator().extract(Context.current(), headers, GETTER);
}
}
A Java TextMapSetter and TextMapGetter for Kafka Headers that encode values as UTF-8 bytes, replace existing keys on write, read the last occurrence on extract, and delegate the wire format to the globally configured propagator.
How do you instrument a Kafka producer and consumer step by step?
Producer
Start a span with kind PRODUCER immediately before the send, and make it current. Then call the propagator's inject with a setter that writes into the record's headers. The order matters: injecting after the span exists puts the producer span's ID in the headers, so the consumer connects to the send itself rather than to whatever request triggered it. Do not end the span when send returns. The send has only been queued in a local buffer at that point. End it in the delivery callback, which is where you learn whether the broker accepted the record. Record the exception and set error status there too, because enqueue-time success says nothing about delivery.
Consumer
For each record in a polled batch, call extract on that record's headers, start a CONSUMER-kind span using the extracted context, and make that span current while your handler runs. When the span is current, the HTTP and database spans that auto-instrumentation creates inside the handler nest under it. End the span after processing completes, or after the commit if that is part of your unit of work.
Common mistakes
- Not making the extracted context current. Extracting returns a value. It does not activate anything, so downstream spans start new traces.
- Leaking context across records. Close the scope at the end of each iteration of the poll loop. Otherwise record N+1 inherits record N's trace.
- Ending spans before async work finishes. If the handler hands work to a future or thread pool, end the span when that work completes.
Attributes
Add the messaging system, destination topic, operation type, partition, and offset. Attribute names such as messaging.system and messaging.destination.name have changed over time and are still evolving. Check the current OpenTelemetry semantic conventions spec before you hard-code any of them.
Tracer tracer = GlobalOpenTelemetry.getTracer("orders-service");
TextMapPropagator propagator = GlobalOpenTelemetry.getPropagators().getTextMapPropagator();
// Producer: create the span first, then inject, so headers carry the PRODUCER span's ID.
Span send = tracer.spanBuilder("orders publish").setSpanKind(SpanKind.PRODUCER)
.setAttribute("messaging.system", "kafka")
.setAttribute("messaging.destination.name", "orders")
.setAttribute("messaging.operation.type", "send").startSpan(); // names evolve: check current semconv
try (Scope s = send.makeCurrent()) {
ProducerRecord<String, String> rec = new ProducerRecord<>("orders", "order-42", payload);
propagator.inject(Context.current(), rec.headers(),
(h, k, v) -> h.add(k, v.getBytes(StandardCharsets.UTF_8)));
producer.send(rec, (meta, ex) -> { // delivery outcome is only known here, so end here
if (ex != null) { send.recordException(ex); send.setStatus(StatusCode.ERROR); }
send.end();
});
}
// Consumer: read trace headers back out of Kafka's byte[] headers.
TextMapGetter<Headers> getter = new TextMapGetter<>() {
public Iterable<String> keys(Headers h) {
List<String> ks = new ArrayList<>(); h.forEach(x -> ks.add(x.key())); return ks; }
public String get(Headers h, String k) {
Header x = (h == null) ? null : h.lastHeader(k);
return x == null ? null : new String(x.value(), StandardCharsets.UTF_8); }
};
for (ConsumerRecord<String, String> r : consumer.poll(Duration.ofMillis(500))) {
// Extract against root so nothing from the previous record can leak in.
Context parent = propagator.extract(Context.root(), r.headers(), getter);
Span span = tracer.spanBuilder("orders process").setParent(parent)
.setSpanKind(SpanKind.CONSUMER).setAttribute("messaging.system", "kafka")
.setAttribute("messaging.destination.name", r.topic())
.setAttribute("messaging.operation.type", "process")
.setAttribute("messaging.destination.partition.id", String.valueOf(r.partition()))
.setAttribute("messaging.kafka.offset", r.offset()).startSpan();
try (Scope s = span.makeCurrent()) { handle(r); } // HTTP/DB spans nest under this span
catch (RuntimeException e) { span.recordException(e); span.setStatus(StatusCode.ERROR); throw e; }
finally { span.end(); } // scope closes per iteration; if handle() goes async, end on completion instead
}
A Java sketch using the OpenTelemetry API and the Kafka client (imports omitted). The producer injects context after starting its span and ends that span in the delivery callback. The consumer extracts context per record, makes the span current, and closes its scope on every loop iteration.
Should a consumer span be a child of the producer or linked to it?
Once you have extracted a context from the headers, you must decide how the consumer span relates to it. There are two patterns, and the right one depends on how many records a single unit of work handles.
Pattern one: parent-child for single-record processing. If each record gets its own processing span, start that span with the extracted context as its parent. The trace then reads as one tree from the HTTP request, through the producer send, to the consumer's database write. This is the easiest trace to debug.
Pattern two: links for batch consumption. When one span covers a polled batch, choosing any single record's context as the parent misattributes the work done for the other records. Instead, start one batch span with no remote parent and attach a span link to each record's extracted context. Some SDKs only accept links at span creation, so collect all the contexts before you start the span.
Parenting has a cost. A record can sit in a topic for hours or be replayed days later. The resulting trace spans that whole gap, which distorts duration metrics, falls outside tail-based sampling buffers, and clutters trace UIs. Links avoid all of this, but following the path means jumping between traces.
Sampling also differs between the two. With head-based sampling, the producer's decision travels in the traceparent sampled flag, and a parent-based sampler on the consumer inherits it. A batch span with many links has no single decision to inherit, so the consumer's sampler makes its own decision.
| Concern | Parent-child | Span links |
|---|---|---|
| Readability | One continuous tree | Separate traces; navigate via links |
| Trace duration | Includes queue wait and replay delay | Each trace stays short |
| Sampling behavior | Inherits producer's head decision | Consumer decides independently |
| Batch fit | Misrepresents all but one record | One link per record, accurate |
| Replay handling | Extends old traces by days | New trace linked to original |
from opentelemetry import trace, propagate
from opentelemetry.context import Context
from opentelemetry.propagators.textmap import Getter
from opentelemetry.trace import Link, SpanKind
tracer = trace.get_tracer("orders-consumer")
class KafkaHeaderGetter(Getter):
# Kafka headers arrive as a list of (str, bytes) tuples, or None
def get(self, carrier, key):
vals = [v.decode("utf-8") for k, v in (carrier or []) if k == key and v is not None]
return vals or None
def keys(self, carrier):
return [k for k, _ in (carrier or [])]
getter = KafkaHeaderGetter()
def handle_one(msg):
# Pattern one: the producer's context becomes the parent, so this span
# joins the producer's trace and inherits its sampled flag
ctx = propagate.extract(msg.headers(), getter=getter)
with tracer.start_as_current_span("orders process", context=ctx, kind=SpanKind.CONSUMER):
process(msg)
def handle_batch(msgs):
# Pattern two: collect every record's context first, because links
# are passed at span creation
links = []
for msg in msgs:
ctx = propagate.extract(msg.headers(), getter=getter)
sc = trace.get_current_span(ctx).get_span_context()
if sc.is_valid:
links.append(Link(sc))
# Empty Context() means no parent: a new trace with its own sampling decision
with tracer.start_as_current_span("orders process", context=Context(), kind=SpanKind.CONSUMER,
links=links, attributes={"messaging.batch.message_count": len(msgs)}):
for msg in msgs:
process(msg)
Python example using the OpenTelemetry API. It parents a single-record consumer span to the extracted producer context, and starts a batch span as a new root with one span link per record.
What are the alternatives to manual context propagation?
Manual inject and extract is one of four options, and usually not the cheapest. OpenTelemetry's language agents and instrumentation libraries wrap common Kafka clients and handle header injection and extraction for you. That removes the propagator plumbing from every producer and consumer and keeps the conventions identical across teams. The cost is that you rely on spans you did not design.
Auto-instrumentation can hide three things:
- Short consumer spans. A consumer span may end when the poll returns or when deserialization finishes, so your business logic never appears in the trace.
- Inconsistent batch handling. A batch can show up as one span per poll with links, one span per record, or no consumer span at all.
- Version drift. Span names, attributes and parenting can change when you upgrade the client library or the instrumentation.
A hybrid approach lets the agent propagate context and record transport spans, while you add one manual processing span per record.
| Approach | Setup cost | Processing time covered | Batch handling | Maintenance | Failure visibility |
|---|---|---|---|---|---|
| Auto-instrumentation | Low | Often not | Library-defined | Re-verify on upgrades | Transport errors only |
| Manual propagation | High | Yes | You define it | Every client call site | Full, if you set status |
| Hybrid | Medium | Yes | You define it | Moderate | Full |
| Correlation IDs only | Very low | No timing | None | Low | Log search only |
Before you trust the emitted spans, check the following:
- Add a deliberate delay to a staging handler and confirm the consumer span grows by that amount.
- Confirm the consumer span shares or links to the producer's trace ID.
- Count the spans produced for a batch of N records.
- Throw an exception in the handler and check that the span is marked as an error.
- Repeat these checks after every client or agent upgrade.
How do you keep trace context across multi-hop, retry, and dead-letter flows?
Consume-transform-produce
A service that reads from one topic and writes to another is the most common place for a trace to fork without any error. If the outgoing send runs after the consumer span has ended, or on a thread where it was never made current, the producer instrumentation finds no active context. It then starts a new root trace, and the backend shows two unrelated traces where there should be one. The fix is ordering. Extract context from the incoming record, start the consumer span, make it current, and call send inside that scope. The propagator then injects the consumer span's context into the outgoing headers, and the next hop continues the same trace. If the send is handed to an executor, pass the Context object to it explicitly. Thread-local state does not follow the work to another thread.
Retry and dead-letter topics
When you republish a failed record to a retry or dead-letter topic, inject the context of the span that handled the failure. Do not let the original headers pass through unchanged and become stale. Store the attempt number as a header and as a span attribute. Use a namespaced attribute of your own, such as app.retry.attempt. This shows each retry as a separate step in the same trace.
Replays
Rewinding offsets or reprocessing an archived topic re-extracts context that may be days old. The backend may have already expired that trace, so new spans become orphans attached to a trace that no longer exists. Detect replays by checking the record timestamp against a threshold or by setting an explicit replay flag. For a replay, start a new root span and add a link to the extracted context instead of making it the parent.
Intermediaries
Older clients that predate header support, connectors, cross-cluster mirroring tools, and custom serializers can drop, filter, or rewrite headers. Test each hop individually. Confirm that traceparent arrives intact on the far side before you rely on it.
How can Kafka tracing save engineering time?
The time savings come from replacing manual correlation work with a single query. Without propagated context, investigating a slow or failed event usually goes like this: find the consumer error in logs, guess which producer sent the record, search that producer's logs by timestamp or business key, and repeat for every upstream hop. Each step depends on someone knowing the topology and on log formats lining up.
With context in the headers, the same investigation starts from one trace ID and gives you:
- The full causal path in one view. The originating HTTP request, the producer send, the consumer processing, and any onward sends appear in one trace or in traces joined by links. Nobody has to reconstruct the topology from memory or documentation.
- Where the time went. Gaps between the producer span ending and the consumer span starting show queue wait and consumer lag for a specific record. Long consumer spans point at slow handlers or downstream calls. This separates "Kafka is slow" from "our handler is slow" without extra instrumentation.
- Failures attributed to their source. When a consumer span is marked as an error, its parent or link identifies the request and service that produced the bad record. Retry and dead-letter hops that carry the handling span's context show how many attempts a record took and where it ended up.
- Less cross-team escalation. Producer and consumer teams can look at the same trace rather than exchanging log excerpts. This matters most when the services are owned by different teams.
- Faster verification of changes. After a deploy, you can check that a new consumer, topic, or retry path is wired correctly by looking at a trace instead of adding temporary logging.
The saving is not free. It depends on the setup work described above being correct, and the integration test in the next section is what stops a silent regression from taking the benefit away. Using auto-instrumentation or the hybrid approach for the propagation plumbing cuts the up-front cost further, because individual teams do not each write and maintain their own setters and getters.
When should you avoid full Kafka tracing, and how do you verify it works?
When to scale back
Per-message spans are not free. On a topic carrying a high, steady message rate, one producer span and one consumer span per record, each with partition and offset attributes, can multiply span volume faster than the business traffic justifies. Storage and ingest cost grow linearly with it. Head sampling at the producer helps, because consumers inherit the sampled flag from traceparent. For bulk consumers, emitting one span per poll batch with links to a sample of producer contexts is often enough.
Baggage needs explicit governance. Every entry is copied into every message header downstream, so bytes accumulate per hop, and any value is readable by every consumer of the topic, including teams you do not control. Allow-list baggage keys, and never put user identifiers or tokens there.
Some pipelines gain little from parented traces. Examples include nightly reprocessing, compaction-driven state rebuilds, and fan-out to analytics sinks. These produce enormous, long-lived traces that nobody reads. Links or no tracing at all are reasonable choices there.
The development cost is mostly in correctness, not volume. A header getter is a few lines, but a missed Context scope or a serializer that drops headers fails silently. The return on that effort is faster cross-service incident diagnosis.
Verifying it works
Start with the raw record. Use the console consumer with header printing enabled, or any debugging tool that shows headers, and confirm that traceparent is present. Take the trace ID from that header and look it up in your tracing backend. It should match the trace that contains the consumer span.
Then lock the behavior in with an integration test. Register an in-memory span exporter and produce to an embedded or containerized broker. After consuming, assert one of two things: the consumer span's trace ID equals the producer span's, or the consumer span holds a link to the producer's span context, depending on the model you chose.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to opentelemetry kafka tracing, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.