Kafka Consumer Lag Monitoring: Alerting on Time and Scaling Safely

What consumer lag is and where the numbers come from

This guide is for backend and platform engineers who run consumer groups that must process data within a freshness target. Consumer lag is the per-partition difference between a partition's log end offset (the offset the next produced record will receive) and the offset the consumer group last committed for that partition. Lag is reported per partition because offsets only have meaning inside a single partition. A group-wide total adds counts from partitions with different write rates, so it can hide one stalled partition behind many healthy ones.

The committed offset usually trails what the consumer has actually processed. Consumers commit periodically, either through auto-commit on an interval or through explicit commits after a batch. Records that are processed but not yet committed still count as lag to any observer that reads committed offsets.

External exporters can make a healthy consumer look lagged when they sample just before a commit. They can also miss a short stall entirely if the stall begins and recovers between two polls. Before you trust a lag graph, find out how often its source polls and how often the consumer commits.

Copy#!/usr/bin/env bash
set -euo pipefail

BOOTSTRAP="broker1.example.internal:9092"  # placeholder
GROUP="orders-processor"                   # placeholder consumer group
SAMPLES=6
INTERVAL_SECONDS=5  # keep this shorter than the commit interval to see lag rise and drop

for i in $(seq 1 "$SAMPLES"); do
  ts=$(date -u +%H:%M:%S)
  # The Apache Kafka distribution names this script kafka-consumer-groups.sh.
  kafka-consumer-groups --bootstrap-server "$BOOTSTRAP" \
      --describe --group "$GROUP" 2>/dev/null |
    awk -v ts="$ts" '
      # Find columns by header name instead of fixed positions.
      $1 == "GROUP" {
        for (c = 1; c <= NF; c++) col[$c] = c
        next
      }
      # Print one line per partition. Do not sum across partitions,
      # because a group total can hide a single stalled partition.
      col["LAG"] && NF >= col["LAG"] {
        printf "%s %s-%s committed=%s log_end=%s lag=%s\n", ts,
          $col["TOPIC"], $col["PARTITION"],
          $col["CURRENT-OFFSET"], $col["LOG-END-OFFSET"], $col["LAG"]
      }'
  sleep "$INTERVAL_SECONDS"
done

Repeated kafka-consumer-groups --describe samples printed per partition, showing how committed-offset lag steps with commit frequency and how a stalled partition differs from normal commit sawtooth.

Who should use time-based lag monitoring and lag-driven scaling

The approach in this guide fits teams that meet most of these conditions:

• A freshness promise exists. Something downstream depends on data arriving within a bound, such as a dashboard, a fraud check, a search index or a billing pipeline. If nobody would notice an hour of delay, a simple offset threshold or a daily check is enough.
• Topics differ in throughput. When one cluster carries both high-rate and low-rate topics, a single offset threshold is wrong for at least one of them. Time-based alerts let you use one rule shape everywhere.
• On-call load matters. Teams that get paged on lag spikes during deploys and rebalances benefit most from the trend, duration and suppression conditions described below.
• Consumers run on elastic infrastructure. If consumers run as containers that an autoscaler can add or remove, you need the rebalance and partition-ceiling material before turning autoscaling on.

It is a poor fit for batch-style consumers that intentionally run once an hour and drain everything, where large lag between runs is expected. For those, alert on whether the run finished and how long it took, not on lag.

Why offset lag alone misleads, and how to measure time lag

A lag of 100,000 messages has no fixed meaning. How long that backlog takes to clear depends on how fast the consumer drains it, so the same count can be harmless on one topic and a serious outage on another.

Illustrative estimate (assumed numbers, not a benchmark)

Formula: time lag ≈ offset lag ÷ recent consume rate

• Topic A: 100,000 ÷ 50,000 msg/s = 2 seconds
• Topic B: 100,000 ÷ 20 msg/s = 5,000 seconds, about 83 minutes

Suppose the freshness SLO is "events processed within 60 seconds of being produced." Topic A is well within it. Topic B has been breaching it for over an hour. A single offset threshold cannot express both cases, which is why alerts should be stated in seconds against the SLO.

The rate-based method fails exactly when you need it most. A stalled consumer has zero throughput, so the estimate either breaks or gets clamped to some value that hides the problem. Measuring the age of the oldest unconsumed message avoids this.

Raw offset lag is still acceptable for coarse dashboards and for topics with a stable, steady rate. On those topics the count converts to time predictably, so it tracks real delay closely enough.

How time-lag monitoring works end to end

A time-lag monitor is a small loop that runs outside the consumers, so it keeps reporting when they crash or hang. Each cycle does the following for every partition the group consumes:

1. Read the committed offset. The monitor asks the group coordinator for the group's committed offsets. In the Java client this is AdminClient.listConsumerGroupOffsets. This is the same data kafka-consumer-groups --describe shows in its CURRENT-OFFSET column.
2. Read the log end offset. The monitor asks each partition leader for its latest offset (AdminClient.listOffsets with the latest offset spec). Offset lag is log end minus committed.
3. Read the timestamp of the oldest unconsumed record. If offset lag is greater than zero, the monitor fetches the single record at the committed offset and reads its timestamp. Time lag is the current time minus that timestamp. If offset lag is zero, there is nothing unconsumed and time lag is reported as zero.
4. Export per-partition gauges. The monitor publishes offset lag and time lag per group, topic and partition. Alert rules then take the maximum across partitions, so one stalled partition cannot be averaged away.

Two details decide whether the number is trustworthy:

• Which timestamp the record carries. The topic setting message.timestamp.type is either CreateTime (set by the producer) or LogAppendTime (set by the broker when it writes the record). CreateTime reflects when the event happened, but producer clock skew or replayed historical data can make records look much older or newer than they are. LogAppendTime reflects when Kafka received the record, which is usually what a freshness SLO about the pipeline means.
• Commit cadence. Because the monitor reads committed offsets, time lag includes the gap between processing and committing. A consumer that commits every 30 seconds will show time lag of up to roughly 30 seconds even when it is fully caught up. Set alert thresholds well above the commit interval.

The monitor's poll interval bounds how quickly it detects a problem. Polling every 30 seconds means a stall can go unseen for up to 30 seconds plus the alert's for duration.

Designing lag alerts around SLO, trend and duration

A useful page answers one question: will this consumer miss its freshness promise if nobody acts? Three conditions together answer it.

Primary alert: SLO breach that is getting worse

Fire when time lag exceeds a fraction of the SLO budget, the lag trend over the window is positive, and both have held for a sustained period. For example: lag is above 60% of a 5-minute freshness SLO, it is rising, and this has been true for 10 minutes. The growth condition filters out a consumer that is behind but recovering. The duration condition filters out brief bursts. Each condition alone is noisy. Together they describe a backlog that will not clear without help.

Secondary alert: stalled partition

Fire when a partition's committed offset has not advanced for N minutes while its log-end offset keeps moving. This catches a poison message or a hung consumer thread on a low-traffic partition. Group-level lag aggregates can hide that case for a long time.

Suppression

Silence or route to a ticket during deploys and rebalances, using deploy annotations or a rebalance-in-progress signal. Expire the suppression automatically so it cannot hide a rebalance that never settles.

Thresholds and simpler options

Tight thresholds catch problems early but page on noise, and on-call engineers learn to ignore them. Loose thresholds stay quiet but can miss lag that creeps upward over days. Pair a loose page with a tighter, non-paging warning.

A fixed offset-lag threshold is enough for steady-rate topics with generous SLOs. Each false page costs someone an investigation and, at night, sleep. Cutting false pages is where the multi-condition design pays off.

Copygroups:
  - name: kafka-consumer-lag
    rules:
      # Page: lag above 60% of a 300s freshness SLO AND still rising, held for 10m.
      - alert: ConsumerLagSLOBreachGrowing
        expr: |
          max by (group, topic) (consumer_time_lag_seconds) > 0.6 * 300
          and
          deriv(max by (group, topic) (consumer_time_lag_seconds)[10m:1m]) > 0
          unless on() (deploy_in_progress == 1)  # set by deploy pipeline with a short TTL
        for: 10m
        labels:
          severity: page

      # Warning only: tighter threshold to catch slow creep without waking anyone.
      - alert: ConsumerLagElevated
        expr: max by (group, topic) (consumer_time_lag_seconds) > 0.3 * 300
        for: 30m
        labels:
          severity: ticket

      # Page: committed offset frozen while new data keeps arriving on that partition.
      - alert: ConsumerPartitionStalled
        expr: |
          changes(kafka_consumergroup_current_offset[15m]) == 0
          and on (topic, partition)
          delta(kafka_topic_partition_current_offset[15m]) > 0
        for: 5m
        labels:
          severity: page

Prometheus rules that page only when time lag breaches a share of the SLO and keeps growing, page separately on a stalled partition, suppress during deploys, and send a tighter non-paging warning for slow creep.

Diagnose before you scale: consumer-bound, skewed or stuck

When lag fires, the first question is which of three failures you have. They produce similar lag graphs, but each has a different fix.

Adding replicas only helps the first case, and only while there are fewer consumers than partitions. Kafka assigns each partition to exactly one consumer in the group, so a hot partition stays on one consumer however many you add. A slow dependency stays slow, and more callers can overload it further. A poison message blocks its partition on whichever consumer owns it. In all three cases, scaling adds rebalances and makes the lag graph look like a capacity problem when the real cause is something else.

Inside a consumer, batching and parallel processing both raise throughput, and each has a cost. Larger batches take longer per poll loop and can push you past max.poll.interval.ms. They also mean more records get replayed after a crash, because more work sits between commits. Parallel workers break per-partition ordering unless you route records by key to fixed workers, and you can only commit an offset once every record before it has finished.

Copy#!/usr/bin/env bash
set -euo pipefail

BOOTSTRAP="kafka-1.internal:9092"   # placeholder broker address
GROUP="orders-processor"             # placeholder consumer group
INTERVAL=60

snapshot() {
  # Locate columns by header name so the parsing does not depend on column order.
  kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --describe --group "$GROUP" \
  | awk 'NF && $1=="GROUP" { for (i=1; i<=NF; i++) col[$i]=i; next }
         NF && col["PARTITION"] { print $col["TOPIC"] "-" $col["PARTITION"], $col["CURRENT-OFFSET"], $col["LAG"] }'
}

before=$(mktemp); after=$(mktemp)
trap 'rm -f "$before" "$after"' EXIT

snapshot | sort > "$before"
sleep "$INTERVAL"
snapshot | sort > "$after"

join "$before" "$after" | awk '
  {
    moved = ($2 == $4) ? "NO" : "yes"   # NO while lag > 0 points to a stuck partition
    printf "%-35s lag=%-10s committed_offset_moved=%s\n", $1, $5, moved
    total += $5
    if ($5 > max) { max = $5; hot = $1 }
  }
  END {
    if (total > 0)
      # Roughly even shares suggest consumer-bound lag. One dominant partition
      # suggests key skew, and more replicas will not help in that case.
      printf "\nlargest: %s holds %.0f%% of total lag (%d)\n", hot, 100 * max / total, total
  }'

A shell script that takes two snapshots of per-partition offsets for a consumer group, a set interval apart, and shows whether lag is spread evenly (consumer-bound), concentrated on one partition (skewed), or sitting on a partition whose committed offset has stopped moving (stuck).

How consumer group scaling works and where it stops

Within a consumer group, each partition is assigned to at most one consumer at a time. That rule is what preserves per-partition ordering, and it also sets a hard ceiling: a group can never have more active consumers than the topic has partitions. Replicas beyond that count join the group, receive no assignment, and sit idle. They still take part in every rebalance.

The partition count therefore has to be planned for peak load plus backlog recovery, not for today's traffic. Raising it later is possible but not free. The default partitioner maps a key to a partition by hashing it modulo the partition count, so a new count sends many existing keys to different partitions. For a while, older records for a key sit in one partition and newer ones in another, and consumers can process them out of order.

Over-provisioning has its own costs. Each partition replica keeps log segments and index files open on its broker and adds replication traffic. It also adds a leader election when a broker fails, so recovery after a broker restart takes longer as the count grows.

Illustrative estimate. These inputs are assumed, not measured:

• Produce rate: 30,000 msg/s
• Per-consumer throughput: 4,000 msg/s
• Partitions: 24
• Backlog: 18 million messages

Keeping up requires 30,000 / 4,000 = 7.5 consumers, so 8. With 8 consumers, net drain is 2,000 msg/s, and the backlog takes 9,000 s (2.5 hours) to clear. With 12 consumers, net drain is 18,000 msg/s, about 17 minutes. With 24 consumers, the partition ceiling, net drain is 66,000 msg/s, about 4.5 minutes. This assumes throughput scales linearly with consumer count and that load is spread evenly across partitions.

Partition rebalancing costs and how to reduce them

That linear-scaling assumption also ignores the work lost each time a consumer joins or leaves the group.

Under the eager rebalance protocol, every member revokes all of its partitions, waits for a new assignment, and only then resumes. The whole group stops consuming, including members whose partitions don't move. Records that were fetched but not yet committed get re-delivered to whichever consumer receives the partition next. Cooperative incremental rebalancing revokes only the partitions that actually move, so unaffected consumers keep working. Static membership (group.instance.id) lets a restarting pod rejoin under the same identity within session.timeout.ms without triggering a rebalance.

max.poll.interval.ms is a separate limit. If a handler takes longer than this between polls, the consumer is evicted from the group. Under an autoscaler, that eviction looks like scale churn.

Commit offsets after processing to get at-least-once delivery. Make handlers idempotent so that replays after a rebalance are harmless. Auto-commit commits on a timer during poll(), regardless of whether your processing has finished. If processing is asynchronous, committed offsets can run ahead of the work actually done, so lag reads low. After a crash, the same records are replayed.

CopyProperties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-0.kafka:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-processor");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
// Cooperative incremental: only partitions that actually move are revoked
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, CooperativeStickyAssignor.class.getName());
// Static membership: a stable per-pod ID (e.g. StatefulSet pod name) lets a quick restart rejoin without a rebalance
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, System.getenv("HOSTNAME"));
// Trade-off: longer tolerance for restarts means a truly dead pod's partitions sit idle this long
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "45000");
// Must exceed worst-case processing time for one poll batch, or the consumer is evicted
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "300000");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "200");
// Disable timer-based commits so committed offsets reflect finished work, not fetched work
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("orders"));
    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> record : records) {
            // Idempotent handler: e.g. upsert keyed on order ID, so replays after a rebalance are harmless
            handleIdempotently(record);
        }
        if (!records.isEmpty()) {
            // Commit only after the whole batch is processed: at-least-once delivery
            consumer.commitSync();
        }
    }
}

A Java consumer configured with the cooperative sticky assignor, static membership, explicit session and poll timeouts, and manual commits issued only after idempotent processing finishes.

Autoscaling consumers on lag: when to use it, when to avoid it, and what it costs

Choose the autoscaling signal carefully. Scale on time lag or on the rate at which lag is growing. Raw offset counts are a poor signal: a backlog of a million small messages may drain in seconds. Set the replica ceiling to the partition count. Consumers beyond that number receive no partitions, but they still take part in every rebalance.

Each scaling action triggers a rebalance, and consumption pauses while partitions move. If an autoscaler adds one replica at a time during a backlog, each step pauses the group again. Lag rises, the autoscaler adds another replica, and the group can end up oscillating. To prevent this:

• Scale up in larger steps.
• Wait for the rebalance to finish and the new consumers to warm up before evaluating again.
• Use hysteresis, with a scale-down threshold well below the scale-up threshold.

Scale-down needs more caution than scale-up because it revokes partitions while work is in flight. Require a longer period of sustained low lag before removing replicas.

Avoid lag-based autoscaling in three cases:

• Skew-driven lag: one hot partition is processed by one consumer, so extra replicas don't help.
• Dependency-bound lag: more consumers put more load on a database that is already saturated.
• Low-partition topics: there is little room to scale.

Illustrative estimate (assumed numbers, not a benchmark):

• Idle cost = idle replicas × hourly price × hours. With 4 × $0.10 × 730, that is $292/month.
• Incident cost = engineers × hours × loaded rate. With 3 × 4 × $100, that is $1,200 for one misdiagnosed rebalance storm.

With these assumed figures, a single incident costs about four months of idle replicas.

Alternatives to lag-based monitoring

Committed-offset lag is not the only way to know whether a pipeline is fresh. These alternatives measure different things and are often used alongside it:

A practical combination is an external committed-offset monitor for stalls and dead consumers, plus end-to-end freshness at the sink for SLO reporting. On the scaling side, the alternatives to lag-based autoscaling are fixed peak provisioning, scheduled scaling and per-consumer tuning, summarized in the table in the previous section. For skewed or dependency-bound lag, the alternative to scaling is to fix the key distribution or the downstream bottleneck directly.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka consumer lag monitoring, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.