Why is async tracing over Kafka different from HTTP tracing?
Kafka trace context propagation means carrying a trace's identity, the W3C traceparent (and optionally tracestate), inside each Kafka record so that a consumer's work joins the same trace as the producer that wrote it. With HTTP, a client sends headers and waits for a response, so parent and child spans overlap in time and the call stack defines the relationship. Kafka removes that structure in four ways:
- No live caller. The producer's span usually ends once the broker acknowledges the write, often before any consumer has read the record.
- Fan-out. Several consumer groups can read the same record, each running its own processing branch from one producer span.
- Time shifts. Consumption can lag by seconds or days. Resetting offsets replays old records, which can attach new work to traces that ended long ago.
- Batching. One
poll()returns records from many producers, so a single consumer operation may have many parents.
[order-svc] --produce--> topic: orders (headers: traceparent=00-<trace-id>-<span-id>-01)
|
+----------------+----------------+
v v
group: billing group: shipping
| (fails)
v
topic: orders.retry --(fails again)--> topic: orders.dlq
(traceparent copied on each hop)
Without the header, each consumer starts an unrelated trace, and the path from order-svc to orders.dlq can't be reconstructed. Backend and platform engineers running event-driven systems benefit most. Propagation pays off when you need to explain end-to-end latency across topics, find which hop added lag, or trace a dead-lettered record back to the request that produced it.
+-----------+ produce +-------------------------------------+
| order-svc |------------->| topic: orders |
| span: P | | headers: |
+-----------+ | traceparent=00-<trace-id>-<P>-01 |
(P ends on ack) +-------------------------------------+
| |
poll() | | poll()
v v
+---------------------+ +---------------------+
| group: billing | | group: shipping |
| span: B (parent P) | | span: S (parent P) |
+---------------------+ +---------------------+
|
| fails: re-produce,
| copy traceparent
v
+-------------------------------------+
| topic: orders.retry |
| headers: traceparent=<same trace> |
+-------------------------------------+
|
| fails again: re-produce,
| copy traceparent
v
+-------------------------------------+
| topic: orders.dlq |
| headers: traceparent=<same trace> |
+-------------------------------------+
One producer span fans out to two consumer groups, and the billing branch's retry and dead-letter hops keep the original trace because each re-produced record carries the traceparent header forward.
How does the W3C traceparent map onto Kafka record headers?
A traceparent value is a 55-character ASCII string of four hyphen-separated, lowercase-hex fields. On a Kafka record it goes into a header with the key traceparent. Kafka header keys are strings, and values are raw byte arrays, so the string is UTF-8 encoded into bytes on produce and decoded on consume. Because the content is pure ASCII, UTF-8 and ASCII produce the same bytes.
| Field | Length (hex chars) | Example | Purpose |
|---|---|---|---|
| version | 2 | 00 |
Format version; parsers reject unknown layouts |
| trace-id | 32 (16 bytes) | 4bf92f3577b34da6a3ce929d0e0e4736 |
Identifies the whole trace; all zeros is invalid |
| parent-id | 16 (8 bytes) | 00f067aa0ba902b7 |
Span ID of the producer-side span that sent the record |
| trace-flags | 2 | 01 |
Bit field; 01 means the upstream sampled this trace |
| tracestate (separate header) | variable | vendorA=abc,vendorB=x1 |
Optional vendor-specific key=value list |
A sample header dump for one record:
traceparent→00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01tracestate→vendorA=abc
Kafka headers differ from HTTP headers in one important way. They are an ordered list, and the same key can appear more than once. Kafka does not deduplicate them. This becomes a problem when a service copies all inbound headers onto an outbound record and then injects its own context. The record now carries two traceparent entries, one stale and one current.
Some extractors read the first occurrence and others read the last. A downstream consumer may therefore attach to the wrong parent without raising any error. To avoid this, remove existing trace headers before injecting new ones.
import java.nio.charset.StandardCharsets;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.header.Header;
import org.apache.kafka.common.header.Headers;
public final class TraceHeaderForwarder {
// Builds an outbound record that keeps the inbound business headers
// but carries exactly one traceparent and at most one tracestate.
static ProducerRecord<String, String> forward(ConsumerRecord<String, String> in,
String outTopic,
String newTraceparent, // e.g. 00-4bf92f3577b34da6a3ce929d0e0e4736-a1b2c3d4e5f60718-01
String tracestate) { // e.g. vendorA=abc, or null
ProducerRecord<String, String> out = new ProducerRecord<>(outTopic, in.key(), in.value());
Headers headers = out.headers();
// Naive copy: this also copies the upstream traceparent/tracestate.
for (Header h : in.headers()) {
headers.add(h);
}
// remove(key) drops every header with that key, so stale trace
// context cannot sit next to the new one in the ordered list.
headers.remove("traceparent");
headers.remove("tracestate");
// Header values are raw bytes; the W3C strings are ASCII, so UTF-8 encoding is byte-identical.
headers.add("traceparent", newTraceparent.getBytes(StandardCharsets.UTF_8));
if (tracestate != null && !tracestate.isEmpty()) {
headers.add("tracestate", tracestate.getBytes(StandardCharsets.UTF_8));
}
// Debug dump: print each header in order, decoding values as UTF-8.
for (Header h : headers) {
System.out.println(h.key() + " -> " + new String(h.value(), StandardCharsets.UTF_8));
}
return out;
}
}
A Java helper using the Kafka clients Headers API that copies inbound headers, removes every existing traceparent and tracestate entry, and then writes exactly one fresh UTF-8-encoded value for each, so downstream extractors cannot pick up a stale parent.
How do you inject and extract trace context with OpenTelemetry for Kafka?
On the producer, start a span with SpanKind.PRODUCER around the send() call, make it current with makeCurrent(), and pass the ProducerRecord's Headers to the configured TextMapPropagator's inject(Context.current(), headers, setter). The setter is a few lines. It calls headers.remove(key) and then headers.add(key, value.getBytes(StandardCharsets.UTF_8)), which applies the deduplication rule above.
Inject at send time, on the calling thread. The Kafka client batches records and flushes them from its own I/O thread, which has no access to your request's context. Header injection delayed until then would capture an empty or unrelated context. Attach headers to the record before handing it to send(), then end the span in the send callback so the span covers the broker acknowledgment.
On the consumer, the getter implements keys() by iterating the record's headers and get() by decoding headers.lastHeader(key). For each record, call extract(Context.root(), record.headers(), getter). Use Context.root(), not Context.current(). Otherwise, a record with no trace headers inherits whatever context the previous record left on the thread. Open the resulting scope in try-with-resources for each record so it closes before the next record runs.
You then have two ways to relate the SpanKind.CONSUMER span to the producer:
- Child: Call
setParent(extracted). The producer and consumer spans appear in one trace. This fits when one span processes one record. - Link: Call
setNoParent()and thenaddLink(Span.fromContext(extracted).getSpanContext()). This starts a new trace that references the producer span. The OpenTelemetry messaging semantic conventions favor links when a single span receives a batch, because one span cannot have several parents.
// Imports omitted: io.opentelemetry.api.trace.*, io.opentelemetry.context.*,
// io.opentelemetry.context.propagation.*, org.apache.kafka.clients.*, org.apache.kafka.common.header.*
static final TextMapSetter<Headers> SETTER = (headers, key, value) -> {
headers.remove(key); // one value per key; never stack duplicate traceparent headers
headers.add(key, value.getBytes(StandardCharsets.UTF_8));
};
static final TextMapGetter<Headers> GETTER = new TextMapGetter<>() {
public Iterable<String> keys(Headers h) { List<String> k = new ArrayList<>(); for (Header x : h) k.add(x.key()); return k; }
public String get(Headers h, String key) {
Header x = (h == null) ? null : h.lastHeader(key);
return (x == null || x.value() == null) ? null : new String(x.value(), StandardCharsets.UTF_8);
}
};
void publish(ProducerRecord<String, String> rec) {
Span span = tracer.spanBuilder("orders publish").setSpanKind(SpanKind.PRODUCER).startSpan();
try (Scope s = span.makeCurrent()) {
propagator.inject(Context.current(), rec.headers(), SETTER); // at send time, on the caller's thread
producer.send(rec, (meta, ex) -> { // callback runs on the Kafka I/O thread
if (ex != null) span.setStatus(StatusCode.ERROR).recordException(ex);
span.end(); // span covers the broker ack
});
}
}
void consume(ConsumerRecords<String, String> records) {
for (ConsumerRecord<String, String> rec : records) {
// Context.root(): a record without headers must not inherit the previous record's context
Context extracted = propagator.extract(Context.root(), rec.headers(), GETTER);
Span span = tracer.spanBuilder("orders process").setSpanKind(SpanKind.CONSUMER)
.setParent(extracted) // child: producer and consumer spans share one trace
// link instead: .setNoParent().addLink(Span.fromContext(extracted).getSpanContext())
.startSpan();
try (Scope s = span.makeCurrent()) { handle(rec); } // scope closes before the next record
finally { span.end(); }
}
}
A minimal Java setter and getter over Kafka headers that inject W3C trace context at send time and extract it per record from Context.root(), with a comment showing the span-link alternative to a parent-child relationship.
How should you trace batch polls, retries, and dead-letter topics?
A single span wrapped around poll() and everything that follows is the most common mistake. One poll can return records from dozens of unrelated traces, so a span around the loop has no correct parent, runs as long as the slowest record, and hides per-record failures. It also never ends cleanly in a long-running consumer.
There are two correct patterns:
- Span per record. Extract each record's
traceparentand start a processing span with that context as its parent or link. Use this when records are handled independently. - Batch span with links. If the code really does process the batch as one unit, such as a bulk database write, create one span and attach one link per record's extracted context. Large batches can produce many links, so cap or sample them.
Pipelines, retries and DLQs
In consume-transform-produce loops and stream processors, every outbound record needs fresh context injected from the current processing span. This applies equally to retry and dead-letter topics. Code often copies the inbound headers onto the outbound record to preserve metadata. Kafka headers allow duplicate keys, so a plain add leaves two traceparent entries, and consumers may read the stale one. Remove the existing traceparent and tracestate keys before injecting. Record the retry attempt count as a span attribute so retries stay visible.
Context leakage
OpenTelemetry context is usually thread-local. When you hand records to a thread pool, wrap the tasks so they carry the captured context. Otherwise the worker threads inherit whatever context a previous task left behind. Reactive consumers need the same care: propagate context through the operator chain, not through ambient thread state.
// Forward a failed record to a retry or DLQ topic without leaving a stale traceparent.
TextMapPropagator propagator = GlobalOpenTelemetry.getPropagators().getTextMapPropagator();
TextMapGetter<Headers> getter = new TextMapGetter<>() {
public Iterable<String> keys(Headers h) {
List<String> keys = new ArrayList<>();
h.forEach(x -> keys.add(x.key()));
return keys;
}
public String get(Headers h, String key) {
Header last = (h == null) ? null : h.lastHeader(key);
return last == null ? null : new String(last.value(), StandardCharsets.UTF_8);
}
};
void forwardToRetry(ConsumerRecord<String, String> in, String topic, int attempt) {
// Per-record span, parented on the upstream producer's context.
Context upstream = propagator.extract(Context.root(), in.headers(), getter);
Span span = tracer.spanBuilder(topic + " publish")
.setParent(upstream)
.setSpanKind(SpanKind.PRODUCER)
.startSpan();
span.setAttribute("app.retry.attempt", attempt); // keeps retries visible in the trace
try (Scope ignored = span.makeCurrent()) {
// Copy business headers, then drop every key this propagator owns
// (traceparent, tracestate) so the injected value is the only one.
Headers out = new RecordHeaders(in.headers().toArray());
propagator.fields().forEach(out::remove);
propagator.inject(Context.current(), out,
(carrier, key, value) -> carrier.add(key, value.getBytes(StandardCharsets.UTF_8)));
producer.send(new ProducerRecord<>(topic, null, in.key(), in.value(), out));
} finally {
span.end();
}
}
A Java retry/DLQ forwarder that copies inbound Kafka headers, removes the propagator's existing trace fields, and injects fresh context from the current span so the outbound record carries exactly one traceparent.
Should consumer spans be children or links, and when should you avoid continuing the trace?
Parent-child is the default most instrumentations produce: the consumer's process span becomes a child of the producer's send span. Traces stay connected, but they get misleading under two conditions. When a record waits hours in a lagging partition, the trace's duration reflects queue time, not work. When one event fans out to dozens of consumer groups, the single trace grows without bound. Links keep each consumer trace bounded and still record where it came from. The cost is navigation: some backends render links as clickable jumps, and others barely surface them.
| Approach | Trace size | Latency reading | Backend dependency |
|---|---|---|---|
| Child of producer span | Unbounded under fan-out | Includes queue delay | Works everywhere |
| New root + link | Bounded per consumer | Consumer work only | Needs link support |
| New root, no link | Bounded | Consumer work only | None, but lineage lost |
Sometimes you should not continue the trace at all:
- Untrusted producers. At ingress topics fed by external partners or other tenants, a
traceparentis attacker-controlled input. It can force sampling or pollute someone else's trace. Start a new root there, and optionally link to the incoming context. - Replays. Topic retention often outlives trace retention. Reprocessed records reference trace IDs that no longer exist in your backend, so prefer links or fresh roots during replay.
Sampling only stays consistent if every hop agrees. A parent-based sampler honors the incoming sampled flag. Mismatched root samplers, or an uninstrumented producer that drops headers, silently break the chain. Head sampling decides once at the producer. Tail sampling must reassemble spans arriving hours apart, which strains its buffering window.
What are the alternatives to manual Kafka instrumentation?
Manual instrumentation is not the only option, and often not the first one to reach for.
Auto-instrumentation. The OpenTelemetry Java agent patches the Kafka client at load time and injects traceparent with no code changes. OpenTelemetry contrib projects in several languages offer Kafka instrumentation libraries that wrap producers and consumers in a similar way. Both approaches only cover the APIs they hook. A custom client wrapper, a batch listener that hands the whole poll to your code, or a stream-processing framework with its own threading model can produce orphaned consumer spans. Library upgrades are the quieter risk. If the instrumentation stops matching a changed client method, it can stop producing spans without raising any error.
Manual setters and getters. These give you full control over span kind, links, and batch handling. In exchange, you own the code in every service.
Composite propagators. During a migration from B3, configure both formats, for example with OTEL_PROPAGATORS=tracecontext,b3multi. The risk comes when a record carries traceparent and B3 headers that disagree, such as after a proxy rewrote only one of them. The extractor then picks one context and drops the other. The trace forks without any visible error.
| Approach | Code changes | Coverage gaps | Main risk |
|---|---|---|---|
| Java agent / contrib libraries | None or minimal | Batch listeners, stream frameworks, custom clients | Silent breakage after upgrades |
| Manual setters/getters | Every service | Only what you skip | Inconsistent implementations |
| Composite (W3C + B3) | Configuration | Inherits from the underlying method | Conflicting contexts |
Verifying behavior
Run the Kafka console consumer with --property print.headers=true. Confirm that each record carries one consistent traceparent. Then check the tracing backend: producer spans should have kind PRODUCER, consumer spans should have kind CONSUMER, and batch spans should show the links you expect.
How do you test propagation, and what does it cost to run?
Run this check in CI rather than by hand. Start a single-node Kafka broker in a container from the test harness. Register an in-memory span exporter with a simple (synchronous) span processor so spans are available as soon as they end. Then:
- Open a parent span with a trace ID you can read back.
- Produce one record inside it.
- Consume the record and wait for the consumer span.
- Assert that the consumer span either shares the parent's trace ID or carries a link to the producer span's context, depending on the model you chose in section 5.
Also assert that the raw record header holds exactly one traceparent. This test catches the usual regressions: an interceptor dropped during a client upgrade, a serializer wrapper that discards headers, or a retry path that re-injects a second header.
Cost
The first service takes the most instrumentation work. After that, a shared producer/consumer wrapper keeps the cost per service small. Maintenance shows up mainly on client and SDK upgrades, which is where the CI test pays for itself.
Illustrative estimate (assumed numbers, not a benchmark). Assume 200-byte payloads at 50,000 records/s. The header costs about 68 bytes per record: an 11-byte key, a 55-byte value, and varint length prefixes.
- Per record: 68 / 200 = 34% before compression.
- Per day: 68 × 50,000 × 86,400 ≈ 294 GB.
The random IDs compress poorly, so treat this as close to the upper bound. Adding tracestate increases it.
In return, you get one trace query in place of joining log lines across services by timestamp and guessed keys. That manual correlation gets less reliable once consumer lag, retries and dead-letter topics reorder events.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kafka trace context propagation, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.