Kafka Consumer Lag Troubleshooting: Slow Consumers and Rebalances

What is Kafka consumer lag, and why is it a symptom rather than a diagnosis?

Consumer lag for a partition is the log-end offset minus the consumer group's committed offset. It counts the records that have been written but not yet acknowledged as processed. Lag is a rate imbalance. It grows when producers append faster than the consumer commits, and it shrinks when the consumer catches up. A lag number does not explain why. The cause could be slow processing, a stalled partition, a rebalance that paused the group, or a traffic spike. That is why lag is a symptom. Finding the cause requires breaking the number down.

The first breakdown is by partition. Aggregate group lag sums every partition, so a flat total of 50,000 can hide 49,000 records stuck behind one hot key on a single partition while the rest sit near zero. Lag should be read per partition, alongside which consumer instance owns each partition.

The second breakdown is time. Offset counts are simple to collect, but their meaning depends on message rate. A lag of 10,000 is seconds of delay on a busy topic and hours on a quiet one. Time-based lag, the age of the oldest unprocessed record, maps directly to what the business cares about, such as how stale an order status is. Measuring it requires trustworthy record timestamps and extra tooling.

This deep dive is for backend and platform engineers who run consumer services. Some lag is normal. A consumer that batches work will show a sawtooth pattern of lag building up and then draining. Lag needs investigation when it trends upward across several cycles or when record age breaches a service-level objective.

Who should use this troubleshooting approach?

The approach in this article fits teams that own Kafka consumers in production and are responsible when those consumers fall behind:

• Backend engineers who write consumer services and control the poll loop, batch sizes, commit logic and downstream calls.
• Platform and SRE engineers who run the deployment, autoscaling and alerting around consumer groups and need to tell a capacity problem from a membership problem.
• On-call engineers who receive a lag alert and need a fixed order of checks rather than guessing.

It is most useful when consumers run in containers with frequent deploys or autoscaling, when processing involves calls to databases or external APIs, and when key-based partitioning makes skew likely. It matters less for a single consumer reading a low-volume topic with no downstream dependencies. In that setting lag is rare, and when it appears the cause is usually obvious from logs.

The approach assumes you can run the Kafka command-line tools against the cluster, read consumer client configuration, and see application metrics such as per-record processing time and downstream latency.

How does consumer group tracking work?

Understanding where the lag number comes from makes the rest of the diagnosis easier.

A consumer group is a set of consumer instances that share a group.id and a topic subscription. One broker acts as the group coordinator for that group. It tracks which members are in the group and runs rebalances when membership or the set of subscribed partitions changes. During a rebalance, partitions are assigned so that each partition belongs to exactly one member of the group. With the classic protocol, one member acts as group leader and computes that assignment using the configured partition.assignment.strategy.

Each member then calls poll() in a loop. The client fetches records from the partitions it owns, the application processes them, and the consumer commits offsets, either automatically or explicitly with calls such as commitSync(). Committed offsets are stored by the brokers in the internal __consumer_offsets topic. A committed offset marks the position the group resumes from after a restart or reassignment.

Lag is computed by comparing two values the brokers already hold: the log-end offset of each partition and the group's committed offset for it. kafka-consumer-groups.sh --describe asks the brokers for both and prints the difference. Consumer clients also expose lag metrics from their own fetch position, such as records-lag-max, but these report only while the consumer is running. That distinction matters for alerting, as the runbook section explains.

Two timers decide whether a member stays in the group: session.timeout.ms, which covers liveness through background heartbeats, and max.poll.interval.ms, which covers progress through poll() calls. Most of the rebalance problems below come from one of those timers expiring.

How do you read lag patterns to find the bottleneck?

Lag is easier to diagnose from its shape over time than from any single reading. Plot it per partition across at least an hour before you decide anything.

To confirm the shape, run kafka-consumer-groups.sh --describe --group <group>. It lists each partition's current committed offset, log-end offset, lag, and the consumer ID and host that own it. Group the lagging partitions by owner. If they all belong to one consumer instance, suspect that host: a noisy neighbour, CPU throttling, or a stuck thread. If they are spread across instances but are always the same partition numbers, the cause is in the data. Hot keys or a dependency that is slow for certain tenants are the usual suspects.

Then compare the timeline with application metrics. If per-record processing time stays flat while lag climbs, the input rate has grown. If downstream call latency rises at the same moment lag does, the bottleneck is that dependency. In that case, adding consumers increases the load on the struggling service and can make the lag worse.

Copy#!/usr/bin/env bash
set -uo pipefail

BOOTSTRAP="kafka-1.internal:9092"   # placeholder broker address
GROUP="orders-processor"            # placeholder consumer group
INTERVAL=60                         # seconds between samples
OUT="lag-${GROUP}.csv"

describe() {
  # Columns: GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
  kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --describe --group "$GROUP" 2>/dev/null
}

describe | awk '$1 != "GROUP" && $6 ~ /^[0-9]+$/ {
    lag[$8] += $6; parts[$8] = parts[$8] " " $2 "-" $3
  }
  END { for (h in lag) printf "%-22s total_lag=%d partitions:%s\n", h, lag[h], parts[h] }' \
  | sort -t= -k2 -nr

[ -f "$OUT" ] || echo "timestamp,topic,partition,lag,consumer_id" > "$OUT"
while true; do
  ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  describe | awk -v ts="$ts" '$1 != "GROUP" && $6 ~ /^[0-9]+$/ {
      print ts "," $2 "," $3 "," $6 "," $7
    }' >> "$OUT"
  sleep "$INTERVAL"
done

A bash script that parses kafka-consumer-groups.sh --describe output to sum lag per owning host. It then appends timestamped per-partition lag to a CSV, so you can plot the lag's shape and line it up with your processing-time and downstream-latency metrics.

How does the poll loop turn slow processing into a rebalance storm?

Kafka tracks two different things about each consumer, and they fail independently. Liveness means the process is alive. A background heartbeat thread sends a heartbeat every heartbeat.interval.ms. If the group coordinator receives none within session.timeout.ms, it assumes a crash or network partition. Progress means the application keeps calling poll(). If the gap between two calls exceeds max.poll.interval.ms, the client concludes the poll loop is stuck. The heartbeat thread then stops heartbeating and leaves the group, even though the process is still running.

The eviction path for a slow batch:

1. The poll thread fetches up to max.poll.records records and starts calling a slow downstream service.
2. The heartbeat thread keeps the session alive, so liveness looks healthy.
3. max.poll.interval.ms elapses. The consumer leaves the group, and the coordinator triggers a rebalance.
4. The partitions move to other members. Those members resume from the last committed offset, so the evicted consumer's uncommitted work is processed again.
5. The original consumer finishes, and its offset commit fails because it no longer owns the partitions. It rejoins, which causes another rebalance.

If the dependency is still slow, every member repeats this cycle. That is a rebalance storm, and while it runs, lag grows and very little gets committed.

The levers, in order of preference:

• Lower max.poll.records so the worst-case batch time fits inside the interval.
• Put timeouts on downstream I/O so a single slow call cannot consume the whole poll budget.
• Move slow work off the poll thread, but only with care. You must pause() the affected partitions, keep polling, commit only offsets whose processing has completed, and drain or cancel in-flight work when a rebalance listener reports revoked partitions.

CopyProperties props = new Properties();
props.put("bootstrap.servers", "kafka-1.internal:9092");
props.put("group.id", "orders-enricher");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
props.put("enable.auto.commit", "false");      // commit only after work is done
// Progress: max gap allowed between poll() calls before the client leaves the group.
props.put("max.poll.interval.ms", "300000");
// Bound the batch so worst case fits: 100 records x 2 s downstream timeout = 200 s < 300 s.
props.put("max.poll.records", "100");
// Liveness: background heartbeats; detects a dead process or network partition.
props.put("session.timeout.ms", "45000");
props.put("heartbeat.interval.ms", "3000");    // well under session.timeout.ms

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
  consumer.subscribe(List.of("orders"));
  while (running) {
    // Each call proves progress; the poll budget restarts here.
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, String> record : records) {
      // Downstream call must enforce its own timeout (e.g. 2 s) so one slow
      // request cannot consume the whole max.poll.interval.ms budget.
      enrich(record);
    }
    try {
      consumer.commitSync();  // commits offsets of the batch returned by the last poll()
    } catch (CommitFailedException e) {
      // We were evicted mid-batch: partitions moved, and the new owner will
      // reprocess from the last committed offset. Log it and keep polling to rejoin.
      log.warn("Commit failed after rebalance; batch will be reprocessed", e);
    }
  }
}

A Java consumer configuration and poll loop that sizes max.poll.records and downstream timeouts to fit inside max.poll.interval.ms, keeps liveness settings separate, and shows where an eviction surfaces as a CommitFailedException.

When should you avoid raising timeouts or going asynchronous?

Each of these levers trades one failure mode for another, so treat none of them as a default.

Raising max.poll.interval.ms stops the eviction, but it also removes the check that catches a consumer that is truly stuck. The background heartbeat thread keeps the session alive while the processing thread is blocked on a deadlocked lock or a downstream call with no timeout. The group coordinator therefore sees a healthy member, and that instance keeps its partitions. Lag on those partitions grows, and nothing reassigns them until the larger interval expires. If you raise it, alert on per-partition lag and on time since the last committed offset, not only on group membership.

Lowering max.poll.records shortens each loop iteration, but fixed per-batch costs are now spread over fewer records. Those costs include the poll call, the offset commit, and any batched database write or downstream request. Throughput can fall, which makes the lag you were trying to fix worse.

Asynchronous or thread-pool processing separates polling from work. However, a committed offset means that everything before it is done. You must track the highest contiguous completed offset per partition. You also give up per-partition ordering unless you route records by key to a fixed worker. If you commit too early, you lose data on a crash. If you fail to drain work on revocation, you produce duplicates.

What are the alternatives to tuning the consumer?

Tuning the poll loop and group membership is not the only response to lag. The alternatives below are sometimes the right call, and each has its own cost.

• Scale out without diagnosis. Adding instances is quick and works when the cause is plain under-capacity and you are below the partition count. It does nothing for key skew, makes a downstream bottleneck worse, and each scale event causes a rebalance.
• Skip the backlog by resetting offsets. kafka-consumer-groups.sh --reset-offsets can move a group's committed offsets, for example to the latest offset, when the backlog has no value, such as stale metrics or cache invalidations. The group must have no active members when you run it, and every skipped record is lost to that consumer. This is a business decision, not a performance fix.
• Divert problem records. If one poison record or one slow tenant blocks a partition, routing failures to a separate retry or dead-letter topic keeps the main partition moving. This adds a second topic and consumer to operate, and changes ordering for the diverted records.
• Split the workload. Moving expensive processing for a subset of events into its own topic and consumer group isolates it, so a slow path cannot hold back a fast one. This is a design change with producer-side work.
• Use the newer consumer group protocol. Recent Kafka releases include a consumer rebalance protocol in which the broker computes assignments incrementally, enabled on the client with group.protocol=consumer. It reduces the cost of membership changes, but both brokers and clients must support it, so check your versions before planning around it.
• Rely on client-side metrics instead of the CLI. Consumer metrics such as records-lag-max give lag without running a command. They are a good complement but stop reporting when the consumer dies, so they cannot replace a broker-side view of committed offsets.

None of these removes the need to know the cause. Scaling, skipping or splitting the workload without knowing why lag grew tends to move the problem rather than fix it.

Why do microservice deployments trigger so many consumer rebalances?

Each pod start or stop changes group membership, and in a containerised fleet that happens far more often than engineers expect. A rolling deploy of ten replicas can trigger twenty membership changes. Autoscaling adds and removes members, and crash-looping pods do the same. Two less obvious triggers are long GC pauses that miss the session timeout, and liveness probes that kill a pod because it is busy processing rather than because it is unhealthy.

Under the default eager protocol, every member revokes all its partitions during each rebalance, so the whole group stops consuming. Setting partition.assignment.strategy to org.apache.kafka.clients.consumer.CooperativeStickyAssignor limits revocation to the partitions that actually move. Switching strategies on a live group takes two rolling bounces. The first adds the cooperative assignor alongside the old one. The second removes the old one.

Static membership fixes restarts by giving each pod a stable group.instance.id, such as the StatefulSet pod name. A restarted pod rejoins with the same identity and gets its old partitions back without triggering a rebalance, as long as it returns within session.timeout.ms.

Autoscaling on raw lag can create a feedback loop. Each scale event triggers a rebalance, the rebalance raises lag, and the higher lag triggers another scale event. To break the loop, scale on lag smoothed over several minutes, add cooldowns longer than a rebalance takes to settle, and cap replicas at the partition count.

Copygroup.id=orders-processor
bootstrap.servers=kafka-0.kafka:9092,kafka-1.kafka:9092

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor,org.apache.kafka.clients.consumer.RangeAssignor

group.instance.id=orders-processor-0

session.timeout.ms=60000
heartbeat.interval.ms=5000

max.poll.interval.ms=300000
max.poll.records=200

A consumer configuration that migrates to CooperativeStickyAssignor in two rolling bounces and enables static membership with a per-pod group.instance.id and a session timeout sized to cover a normal pod restart.

Can you ever catch up? Partition limits and an illustrative capacity estimate

Consumer scaling stops at the partition count, because that is the hard ceiling on parallelism. Partition assignment gives each partition to exactly one consumer in the group, so consumers beyond the partition count sit idle. Key-based partitioning adds skew. If one hot key fills a partition, that partition's single consumer is the bottleneck, and adding instances cannot split it.

When you hit the ceiling, these are the options:

Illustrative estimate (assumed numbers, not a benchmark). Assume:

• Produce rate: 2,000 records/s, spread evenly across keys
• Processing time: 4 ms per record, handled serially per partition
• 12 partitions
• 6 consumer instances

The capacity formulas are:

• Per-partition capacity = 1 / processing time = 1 / 0.004 s = 250 records/s
• Group capacity = min(consumers, partitions) × per-partition capacity

With 6 instances, group capacity is 6 × 250 = 1,500 records/s. That is below the 2,000 records/s produce rate, so lag grows by 500 records/s.

With 12 instances, group capacity is 12 × 250 = 3,000 records/s, a surplus of 1,000 records/s.

Drain time = backlog / (capacity − produce rate). A backlog of 3.6 million records therefore takes 3,600,000 / 1,000 = 3,600 s, or about one hour.

A 13th instance adds nothing because there are only 12 partitions. Past that point, only lower per-record cost or more partitions raises capacity.

What should a lag monitoring and diagnostic runbook include?

In the estimate above, cheaper per-record processing or more efficient downstream calls shorten that hour. That makes it important to know which constraint you are actually hitting before you act on it.

Diagnostic sequence

1. Per-partition or global? Lag on one partition points to key skew or a poison record. Lag spread evenly across partitions points to throughput.
2. Rebalance activity in the window? If the group is churning, stabilise membership first. In that case lag is a side effect.
3. Processing time versus max.poll.interval.ms. A batch that approaches the limit is a rebalance waiting to happen.
4. Downstream latency. If a database or dependency slowed down, adding consumers only adds load to the real bottleneck.
5. Only then scale or repartition.

The order runs from cheapest check to most expensive change. The first four steps take minutes using dashboards you already have. Repartitioning changes the key-to-partition mapping, breaks per-key ordering during the transition, and is hard to undo. A fixed sequence helps on-call engineers rule out the common causes before they commit to that kind of work.

How does this approach save engineering time?

Most of the time lost on lag incidents goes into fixes that target the wrong cause. A structured diagnosis cuts that waste in several concrete ways:

• Fewer wrong fixes. Checking downstream latency before scaling avoids adding consumers that only increase load on a struggling database. Reading lag per partition avoids scaling a group whose real problem is one hot key.
• Avoiding irreversible work. Adding partitions cannot be undone and disrupts per-key ordering. Ruling out skew, rebalances and slow dependencies first means that change is only made when the capacity ceiling is the actual limit.
• Shorter incidents. The diagnostic sequence uses the CLI and dashboards you already have, so an on-call engineer can narrow the cause without reading consumer code or escalating to the service owner straight away.
• Fewer recurring incidents. Static membership and the cooperative assignor remove the rebalance pauses that occur on every deploy, and a bounded max.poll.records with downstream timeouts prevents rebalance storms. Those fixes are configuration changes made once rather than incidents handled repeatedly.
• Alerts that fire for the right reason. Alerting on time lag, sustained growth, rebalance frequency and broker-side committed offsets reduces pages for normal sawtooth patterns and catches crashed consumers that in-process metrics would miss.
• Planning without load tests for every change. The capacity formula in the estimate above lets a team check whether a proposed change, such as more instances or a faster code path, can close the gap before anyone builds or deploys it.

How much time this saves depends on how often your groups lag and how they are deployed. The largest gains usually come in containerised fleets with frequent deploys and autoscaling, where membership churn is the most common hidden cause of lag.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka consumer lag troubleshooting, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.