What is Kafka consumer lag, and why is the aggregate number misleading?
Kafka consumer lag is measured per partition: the log end offset (the offset the next produced record will receive) minus the consumer group's committed offset for that partition. A lag of zero means the group has committed everything the broker has accepted. A positive value is the number of records written but not yet confirmed as processed.
Most dashboards sum these values across every partition in the group, and that sum hides problems. As an illustrative estimate, take a topic with 12 partitions. Eleven of them have a lag of about 50 records and one has a lag of 40,000. The total is roughly 40,550, which can look like normal backlog during a traffic peak. In practice, eleven consumers are keeping up and one partition is falling further behind with every poll. An alert on the sum fires late, and it points to the wrong cause, because the problem is not general throughput.
Offset lag also describes only one dimension of the problem:
| Offset lag | Time lag | |
|---|---|---|
| Measures | Records behind the log end | Age of the oldest unprocessed record |
| Unit | Count of offsets | Seconds or minutes |
| Depends on produce rate | Yes. 10,000 records can mean seconds of delay or hours. | No |
| Best for | Capacity planning, spotting backlog growth | SLOs and user-facing freshness alerts |
If you own a consumer service or its alerting, base pages on per-partition time lag, because that maps directly to how stale your data is. Keep offset lag for spotting trends and sizing capacity.
Per-partition offsets (illustrative estimate, 12 partitions, not to scale)
P0 |===================C.E| lag 50
P1 |===================C.E| lag 50
... P2..P10 same pattern lag ~50 each
P11 |=====C.................................E| lag 40,000
C = committed offset E = log end offset lag = E - C
==== processed and committed .... written, not yet committed
per-partition view aggregate view
+---------------------------+ sum +-------------------------+
| 11 partitions: ~50 each |----------->| total lag ~= 40,550 |
| P11: 40,000 !! | | looks like peak backlog |
+---------------------------+ +-------------------------+
| |
v v
+---------------------------+ +-------------------------+
| time lag(P11) = now - | | alert on sum: fires |
| timestamp of oldest | | late, blames general |
| unprocessed record | | throughput |
| -> page on this (SLO) | | |
+---------------------------+ +-------------------------+
Lag is log end offset minus committed offset for each partition. Summing it across the group buries one hot partition (40,000 behind) among eleven healthy ones, and per-partition time lag is the signal that should page.
Who should use this approach?
The techniques in this article are for engineers who run Kafka consumers in production and get paged when data goes stale. That usually means one of these roles:
- Service owners whose application consumes from Kafka and writes to a database, calls APIs, or feeds another topic. You control the poll loop, commit strategy and per-record processing code, which is where most lag originates.
- On-call and SRE engineers who respond to lag alerts and need to decide quickly whether the cause is a hot partition, a bad host, a rebalance loop, or not enough total capacity.
- Platform teams who maintain shared consumer libraries, dashboards and alert rules for many services. The instrumentation and alerting advice applies most directly to you.
It is less relevant if your consumers are fully managed by a framework that hides partition assignment and commits from you. Even then, the diagnosis patterns (one partition, one consumer, or everything) still apply. It is also more than you need for low-volume topics where lag never builds, though per-partition time-lag alerting is cheap insurance.
How does consumer lag tracking work?
Lag comes from two numbers that live in different places on the broker side:
- The log end offset. Each partition is an append-only log. When a producer writes a record, the partition leader assigns it the next offset. The log end offset is the offset the next record will receive. (Strictly, consumers using the default isolation level read up to the high watermark, the offset replicated to all in-sync replicas. Lag tools typically report against the end offset the broker exposes to consumers.)
- The committed offset. A consumer group is coordinated by one broker, the group coordinator. When a consumer commits, either automatically on a timer or explicitly with
commitSync()/commitAsync(), the coordinator stores the offset for that group and partition in the internal__consumer_offsetstopic. By convention, the committed offset is the offset of the next record to read, not the last one processed.
Lag for a partition is the difference between those two values. Tools such as kafka-consumer-groups.sh ask the coordinator for the group's committed offsets and ask the partition leaders for end offsets, then subtract.
Three consequences follow from this design:
- Lag reflects commits, not progress. The consumer's in-memory position advances on every
poll(), but nothing outside the process sees it until a commit succeeds. Delayed or failing commits make lag look worse than it is. - Lag is only as fresh as the last commit and the last read of end offsets. Two snapshots taken seconds apart can differ just because a commit landed in between, which is why the trend matters more than any single value.
- Assignment determines who can reduce lag. Within a group, each partition is assigned to exactly one member at a time. Only that member can make progress on it, which is why a slow partition or a slow host shows up as concentrated lag rather than being absorbed by the rest of the group.
Time lag is derived differently: it needs the timestamp of a record (the oldest unprocessed one, or as an approximation the last committed one), compared against the current time. Kafka records carry either a producer-set CreateTime or a broker-set LogAppendTime, depending on topic configuration.
How do you read per-partition lag with kafka-consumer-groups.sh?
Running kafka-consumer-groups.sh --describe --group <group> against a broker prints one row per partition the group has committed offsets for. Four columns matter most:
- CURRENT-OFFSET: the last offset the group committed, not the position the consumer has actually reached in memory. Between commits, lag can look higher than it really is.
- LOG-END-OFFSET: the offset the next record written to the partition will receive.
- LAG: LOG-END-OFFSET minus CURRENT-OFFSET, measured in records.
- CONSUMER-ID: the group member that currently owns the partition. A dash means no member owns it, which you'll see when the group is empty or in the middle of a rebalance.
A single snapshot tells you little. Run the command several times a minute or so apart and compare. What you are looking for is direction: whether lag is growing, shrinking or flat, and how it is spread across partitions and consumers.
Three patterns worth recognizing
| Pattern | What you see across runs | Likely cause |
|---|---|---|
| One hot partition | One partition's LAG climbs while the others stay near zero | Key skew: a heavy key or tenant hashes to that partition, or its records are unusually expensive to process |
| Lag follows a consumer | Every partition with growing lag shares the same CONSUMER-ID | A slow host (CPU throttling, GC pauses, noisy neighbour) or a dependency that only that instance calls slowly |
| Even growth everywhere | LAG rises at a similar rate on all partitions and all consumers | Total consumer throughput is below the produce rate, or a shared downstream system (database, API) is the bottleneck |
The second pattern is the one most often misread as key skew. To tell them apart, check whether the lag moves with the partition or stays with the consumer after a rebalance. If it moves with the partition, the data is the cause. If it stays with the consumer, the host is.
#!/usr/bin/env bash
set -euo pipefail
BOOTSTRAP="kafka-1.internal:9092" # placeholder broker address
GROUP="orders-processor" # placeholder consumer group
RUNS=5
INTERVAL=60 # seconds between snapshots
for i in $(seq 1 "$RUNS"); do
echo "=== run $i $(date -u +%H:%M:%S) ==="
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
--describe --group "$GROUP" 2>/dev/null |
awk '
# Find columns by header name instead of assuming a fixed order.
$1 == "GROUP" { for (c = 1; c <= NF; c++) col[$c] = c; next }
# Skip anything before the header and any blank or short lines.
!("LAG" in col) || NF < col["LAG"] { next }
{
tp = $col["TOPIC"] "-" $col["PARTITION"]
lag = $col["LAG"]
owner = $col["CONSUMER-ID"] # "-" means unowned (empty group or rebalance)
printf " %-30s lag=%-12s owner=%s\n", tp, lag, owner
# Roll lag up per owner. If growth tracks one owner across runs,
# suspect that host or its dependencies, not key skew.
if (lag != "-") byowner[owner] += lag
}
END { for (o in byowner) printf " TOTAL owner=%s lag=%.0f\n", o, byowner[o] }
'
if [ "$i" -lt "$RUNS" ]; then sleep "$INTERVAL"; fi
done
A bash loop that snapshots kafka-consumer-groups.sh --describe several times and totals lag per partition and per consumer, which lets you tell whether lag follows a partition, a consumer, or the whole group.
Why does one partition fall behind? Key skew and slow per-record processing
The default partitioner hashes each record's key and maps it onto one partition, so every record with the same key lands in the same place. When key cardinality is low (a handful of region codes) or one key dominates (a single large tenant), one partition receives a disproportionate share of traffic. To confirm this, compare per-partition produce rates and average message sizes over the same window. A hot partition shows higher inbound bytes, not just higher lag.
Skew is not the only cause. If each record triggers a database write or an HTTP call, that I/O usually dominates processing time. Within a partition, records are processed in offset order, so one slow call holds up every record behind it. This is head-of-line blocking. A single tenant with slow downstream dependencies can stall a partition even at normal volume.
Scaling out does not fix either case. Each partition is assigned to at most one consumer in a group, so instances beyond the partition count sit idle and consumer throughput stays the same.
| Remedy | What it fixes | Trade-off |
|---|---|---|
| Re-key or salt hot keys | Spreads a heavy key across partitions | Loses strict per-key ordering |
| Add partitions | Raises the parallelism ceiling | Changes key-to-partition mapping, so ordering assumptions can break during the transition |
| Concurrent processing within a partition | Hides slow I/O | Offset commits get harder, and a crash can cause duplicates or reordering |
How does partition rebalancing create lag, and how do you reduce it?
Every rebalance stops consumption. With the default eager protocol, all members revoke all their partitions, wait for a new assignment, then resume from the last committed offsets. Lag grows on every partition during that window, including partitions that were healthy. Records that were processed but not committed are processed again.
The slow-consumer rebalance loop
If the gap between two poll() calls exceeds max.poll.interval.ms, the client concludes the consumer is stuck and leaves the group. Heartbeats on the background thread do not prevent this. The group rebalances, and the slow partition moves to another member. That member resumes from the last commit and hits the same expensive records. It also exceeds the interval, gets removed, and triggers another rebalance. In logs, this looks like repeated join/sync cycles and commit failures, while lag rises across the whole group. The fix is to bound the work done per poll. Lower max.poll.records or speed up processing first. Raising the interval only makes real hangs take longer to detect.
Reducing churn
- Cooperative rebalancing (the cooperative sticky assignor) revokes only the partitions that actually move. Other members keep consuming during the rebalance.
- Static membership (
group.instance.id) lets a restarted instance rejoin under the same identity and get its old partitions back without a rebalance, provided it returns withinsession.timeout.ms. The trade-off is that a member that crashes for good keeps its partitions unconsumed until the session timeout expires. A long timeout means a long stall.
Autoscaling on lag
Every scale-out or scale-in event triggers a rebalance. That pause adds lag, and the added lag can cause more scaling. Use cooldowns and scale in coarse steps. Once the instance count equals the partition count, extra consumers sit idle and add nothing except rebalance overhead.
bootstrap.servers=broker-1.example.internal:9092
group.id=orders-processor
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
max.poll.records=100
max.poll.interval.ms=300000
group.instance.id=orders-processor-0
session.timeout.ms=45000
heartbeat.interval.ms=3000
enable.auto.commit=false
Consumer properties that cap per-poll work under max.poll.interval.ms, use the cooperative sticky assignor for incremental rebalances, and enable static membership with a session timeout sized to cover a normal restart.
How should you tune the poll loop and commit strategy?
Size the poll loop from measurements. A batch must finish within max.poll.interval.ms, or the consumer is removed from the group and a rebalance starts. The worst-case batch time is roughly max.poll.records × p99 per-record processing latency. Illustrative estimate: with an assumed p99 of 40 ms and 250 records per poll, a batch takes about 10 s. A 30 s interval gives 3× headroom. If p99 spikes to 150 ms during a downstream slowdown, the same batch needs 37.5 s and the consumer gets evicted. Lower max.poll.records before you raise the interval.
Fetch sizing (fetch.max.bytes, max.partition.fetch.bytes) sets how much data is buffered client-side. Larger fetches reduce round trips but increase heap use per consumer.
| Aspect | Auto-commit | Manual commit after processing |
|---|---|---|
| When offsets commit | On a timer, during poll() |
After the batch is handled |
| Risk with async processing | Offsets can commit before work finishes (loss) | Commit tracks completed work |
| Redone work after a crash | Up to one commit interval | Up to one uncommitted batch |
Lag tools compute lag from committed offsets. If commits fail or are delayed, the consumer's real position can be far ahead of what the tool shows. Before scaling out, check the commit error rate.
Every increase involves a trade-off. Larger batches raise memory use and per-batch latency, and they add more redone work after a failure. A higher max.poll.interval.ms prevents spurious rebalances, but the group takes longer to detect a consumer that is actually stuck.
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.*;
public class OrdersConsumer {
public static void main(String[] args) {
// Size from measurements: assumed p99 = 40 ms/record, 3x headroom (illustrative).
long p99Ms = 40; int maxRecords = 250; int headroom = 3;
long pollIntervalMs = maxRecords * p99Ms * headroom; // 250 * 40 * 3 = 30000 ms
Properties p = new Properties();
p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1.example.internal:9092");
p.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-processor");
p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
p.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // commit only after work is done
p.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, String.valueOf(maxRecords)); // lower this first if p99 rises
p.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, String.valueOf(pollIntervalMs));
p.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, "1048576"); // client-side buffer per partition
p.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, "52428800"); // cap per fetch response (heap use)
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(p)) {
consumer.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> r : records) {
process(r); // must finish the whole batch within max.poll.interval.ms
}
if (records.isEmpty()) continue;
try {
consumer.commitSync(); // lag tools read this offset, not the in-memory position
} catch (CommitFailedException e) {
// Usually means the group rebalanced (batch exceeded the interval). Count it:
// a rising rate makes reported lag look worse than real progress.
System.err.println("commit failed: " + e.getMessage());
}
}
}
}
static void process(ConsumerRecord<String, String> r) { /* business logic */ }
}
A Java consumer that derives max.poll.interval.ms from max.poll.records and an assumed p99 per-record latency, disables auto-commit, and commits synchronously after each processed batch while surfacing commit failures that would otherwise inflate reported lag.
What should you instrument to diagnose lag faster?
Lag is easier to diagnose from dashboards than from ad hoc CLI runs. Four metrics cover most incidents, and each should be tagged with topic, partition and consumer instance ID:
- Per-partition offset lag. This is the log-end offset minus the committed offset.
- Per-partition time lag. This is the current time minus the timestamp of the last committed record.
- Per-record processing duration, recorded as a histogram. Averages hide the slow tail that stalls a single partition.
- Rebalance count and duration per consumer instance.
The tags carry most of the diagnostic value. If lag is concentrated on one partition while other partitions on the same instance are healthy, key skew is the likely cause. If every partition owned by one instance is lagging, look at that host or pod. If lag rises in step with rebalance counts, the problem is group membership rather than the processing code.
| Alert basis | Cost to build | Comparable across topics | Main weakness |
|---|---|---|---|
| Offset lag | Low; available from broker-side offsets | No; 10,000 records may be seconds on one topic and hours on another | Thresholds need per-topic tuning |
| Time lag | Higher; requires reading record timestamps | Yes; "five minutes behind" means the same thing everywhere | Producer-set CreateTime timestamps can be skewed or wrong; LogAppendTime is more reliable |
The development cost is modest. Offset lag needs no application changes. The histogram and rebalance listener metrics take a small, one-time change to the shared consumer wrapper. Without them, responders spend each incident reconstructing which partition, which instance and which phase is slow, and that time recurs with every page. The main ongoing cost is metric cardinality on topics with many partitions, so set a retention or aggregation policy before rolling out the tags.
When should you add partitions or consumers, and when should you avoid it?
A worked capacity estimate
Illustrative estimate, not a measurement. Assume a topic receives 6,000 records/s at peak and each record takes 5 ms to process sequentially. One consumer thread then handles about 200 records/s, so you need at least 30 consumers. Running at 70% utilization to leave headroom for rebalances and catch-up gives 6,000 / (200 × 0.7) ≈ 43 instances. A partition is assigned to at most one consumer in a group, so you need at least 43 partitions. Choosing 48 leaves room to grow without repartitioning.
When scaling out does not help
The calculation assumes load is spread evenly across partitions. With a skewed key, the hot partition still has one consumer, and adding instances only adds idle ones. Adding partitions also changes which partition each key hashes to. Records for a key can then sit on two partitions during the transition, which breaks strict per-key ordering. If the bottleneck is a slow database or API, more consumers send more concurrent requests to it. That usually raises its latency instead of increasing throughput.
| Fix | Dev effort | Ops complexity | Best when |
|---|---|---|---|
| Add consumers/partitions | Low | Medium (repartitioning, rebalance cost) | Load is even and the per-record cost is fixed |
| Fix the hot key | Medium (key redesign) | Low | One partition dominates the lag |
| Batch or cache downstream calls | Medium | Medium (cache invalidation) | Downstream round trips dominate processing time |
| Concurrent processing within a partition | High (per-key ordering, offset tracking) | Medium | Records are I/O-bound and largely independent |
| Poll-loop tuning | Low | Low | Lag comes from rebalances or commit overhead |
What are the alternatives to the CLI for measuring lag?
kafka-consumer-groups.sh is the right tool during an incident, but it is a poor basis for continuous monitoring. There are three common alternatives, and each sees lag from a different vantage point.
Consumer client metrics. The Java consumer exposes fetch metrics over JMX, including records-lag per partition and records-lag-max. These are computed from the consumer's in-memory fetch position rather than the committed offset, so they show real progress and are not inflated by delayed commits. The weakness is that they exist only while the consumer is running. If an instance crashes or the group is empty, the metric disappears exactly when lag is growing fastest, so client metrics alone cannot drive an alert for a dead consumer.
The Admin API. The Java Admin client can fetch a group's committed offsets (listConsumerGroupOffsets) and partition end offsets (listOffsets with OffsetSpec.latest()). Subtracting them gives the same numbers the CLI prints, without parsing text output. This is the usual building block for an in-house lag exporter or a small service that publishes per-partition lag to your metrics system. Because it reads from the brokers, it keeps reporting when consumers are down.
External lag monitors. Several open-source exporters and monitoring services poll committed and end offsets from outside the consumer and publish per-partition lag, some with time-lag estimates or trend-based status evaluation. They follow the same broker-side approach as the Admin API, so they share its strengths. Evaluate them on whether they report per partition (not only per group), whether they can estimate time lag, and how they behave for groups with no active members.
| Approach | Source of truth | Survives consumer outage | Shows real progress between commits | Effort |
|---|---|---|---|---|
kafka-consumer-groups.sh |
Committed offsets | Yes | No | None, but manual |
| Consumer client metrics | In-memory fetch position | No | Yes | Low; already emitted by the client |
| Admin API exporter | Committed offsets | Yes | No | Medium; you write and run it |
| External lag monitor | Committed offsets | Yes | No | Low to medium; deploy and configure |
A practical combination is a broker-side source (Admin API or an external monitor) for alerting, because it keeps working when consumers fail, plus client metrics and processing-time histograms for diagnosis, because they show what the consumer is actually doing.
The alternatives to scaling are covered in the fix table above. In most cases, re-keying, batching downstream calls, or tuning the poll loop addresses the cause more directly than adding instances does.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kafka consumer lag, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.