How does Kafka consumer rebalancing work, and why do eager assignors stop the world?

A rebalance is how a consumer group decides which member owns which partition. Each group is managed by a group coordinator, a broker selected by hashing the group ID. When membership or subscriptions change, through a join, a leave, a missed heartbeat, or new partitions, the coordinator starts a new generation of the group in two steps:

  1. JoinGroup. Every member sends its subscription. The coordinator picks one member as group leader and gives it the full member list.
  2. SyncGroup. The leader runs the configured assignor on the client side and sends the result back. The coordinator then hands each member its own share.

With an eager assignor such as range or round-robin, every member revokes all of its partitions before it rejoins. The protocol has no way to say "keep what you have," so the whole group stops consuming until SyncGroup completes. A cooperative incremental rebalance lets members keep their partitions. Only the partitions that need to move are revoked, and a short follow-up rebalance assigns them to their new owners.

EAGER (C3 joins)                 COOPERATIVE (C3 joins)
C1: P0 P1 -> revoke all -> P0    C1: P0 P1 -> keep P0, revoke P1
C2: P2 P3 -> revoke all -> P2    C2: P2 P3 -> keep P2, revoke P3
C3: --    ->            -> P1 P3 C3: --    -> (2nd round) -> P1 P3
    [ all consumption paused ]       [ only P1, P3 paused ]

In microservices this causes two problems. The first is stop-the-world revocation during every deploy or autoscaling event. The second is pod restarts. A restarted pod comes back with a new member ID, so the group sees one member leave and a new one join, which can trigger two full rebalances for what was the same consumer.

 CONSUMERS                          GROUP COORDINATOR (one broker)
 +--------------+                  +-----------------------------+
 | C1           |--JoinGroup------>| 1. JoinGroup phase          |
 | C2           |--JoinGroup------>| collect subscriptions,      |
 | C3 (new)     |--JoinGroup------>| pick leader, new generation |
 | C1 = leader  |<--member list----| full list to leader only    |
 | runs assignor|                  |                             |
 | C1 (plan)    |--SyncGroup------>| 2. SyncGroup phase          |
 | C2, C3       |--SyncGroup------>| store leader's plan         |
 | C1, C2, C3   |<--own assignment-| send each member its share  |
 +--------------+                  +-----------------------------+

 EAGER, C3 joins                     COOPERATIVE INCREMENTAL, C3 joins
 +--------------------------------+  +--------------------------------+
 | C1: P0 P1 --revoke all--> P0   |  | C1: P0 P1 --keep P0-----> P0   |
 | C2: P2 P3 --revoke all--> P2   |  | C2: P2 P3 --keep P2-----> P2   |
 | C3: --    --------------> P1 P3|  | C3: --    --2nd round---> P1 P3|
 +--------------------------------+  +--------------------------------+
 | every member revokes first     |  | movers revoke; rest keep       |
 | ALL partitions paused          |  | only P1, P3 paused             |
 +--------------------------------+  +--------------------------------+

 Pod restart, no static ID: old member leaves + new ID joins = 2 rebalances

In a rebalance, the coordinator collects subscriptions in JoinGroup and the leader computes the assignment on the client for SyncGroup. An eager assignor pauses every partition, while a cooperative incremental one revokes only the partitions that move.

What is the cooperative sticky assignor and who should use it?

With incremental cooperative rebalancing, consumers do not drop their whole assignment when the group changes. In the first round, each member reports the partitions it currently owns. The group leader computes a new target assignment and tells members to revoke only the partitions that must move to someone else. A second round then assigns those freed partitions to their new owners. Every partition that stays put keeps being fetched and processed the whole time.

That is why rebalance-driven lag drops. Under an eager assignor, a single pod restart pauses consumption on every partition in the group. Under cooperative sticky, the pause applies only to the few partitions changing hands. The assignor also tries to keep partitions on their previous owners, so local caches and state stay useful.

Aspect Eager (e.g. range, round-robin) Cooperative sticky
Partitions paused All Only those that move
Rounds per change One Two or more
Time to settled assignment Usually shorter Can be longer
Load rebalancing Redistributes freely Can preserve skew

The teams that gain most are those that deploy often, run stateful processing where moving a partition means rebuilding state, or consume many partitions, where a full stop-the-world pause costs the most.

There are trade-offs. The extra rounds can make the full rebalance take longer to settle. Stickiness can keep an uneven distribution in place if partitions differ in throughput. Support, configuration names, and defaults vary across client libraries and versions, so check what your own client actually implements before you rely on it.

How do you migrate a running consumer group to cooperative sticky safely?

The constraint that makes this a two-step process is how the group picks an assignor. When members join, the coordinator selects a strategy that every member lists as supported. If you roll out a config that replaces the eager assignor with cooperative sticky, then partway through the deploy, old members support only the eager assignor and new members support only cooperative sticky. There is no common strategy, so new members are rejected when they try to join (the Java client surfaces an inconsistent group protocol error). As the deploy continues, members keep bouncing in and out, and every join and leave triggers another rebalance. The only way to switch in one step is to stop the whole group, which means downtime.

Step 1: advertise both assignors

Roll out a config to every member that lists cooperative sticky first and keeps the current eager assignor as a fallback. In the Java client, that is partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor,org.apache.kafka.clients.consumer.RangeAssignor, with RangeAssignor replaced by whatever you use today. Old and new members always share the eager assignor, so the group stays healthy. Because the list still contains an eager assignor, the client continues to use the eager protocol and revokes all partitions on each rebalance. Expect no improvement yet.

Step 2: remove the eager assignor

After every member runs the step 1 config, roll out a second change: partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor. All members already support cooperative sticky, so the group never loses its common strategy. Once the last eager entry is gone, members switch to incremental rebalancing. Do not start step 2 until step 1 has fully finished. Confirm it by checking the configuration of each running instance, not just the deploy status.

# consumer.properties -- STEP 1 (roll out to every member first)
# List cooperative sticky first, keep the current eager assignor as a fallback.
# Old members (Range only) and new members still share RangeAssignor,
# so the group always has a common strategy during the rolling deploy.
# The eager protocol stays in use while any eager assignor is listed.
group.id=orders-processor
bootstrap.servers=kafka-1.internal:9092,kafka-2.internal:9092
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor,org.apache.kafka.clients.consumer.RangeAssignor

# ---------------------------------------------------------------

# consumer.properties -- STEP 2 (only after EVERY member runs step 1)
# Every member already supports cooperative sticky, so dropping the
# eager fallback never leaves the group without a common strategy.
# Once the eager entry is gone, rebalances become incremental.
group.id=orders-processor
bootstrap.servers=kafka-1.internal:9092,kafka-2.internal:9092
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

# DO NOT jump straight from the original config to step 2 while the
# group is running. Range-only members and cooperative-only members
# share no strategy, so new members are rejected and the group keeps
# rebalancing until the deploy finishes.

Java consumer configs for the two rolling deploys. Step 1 advertises CooperativeStickyAssignor with the existing RangeAssignor as a fallback. Step 2 removes the eager assignor so members switch to incremental rebalancing.

Writing a correct rebalance listener for revoked, assigned, and lost partitions

With the migration finished, the listener carries most of the correctness burden. In the Java client, ConsumerRebalanceListener exposes three callbacks, and each needs a different response.

onPartitionsRevoked: finish the work you own

Under cooperative sticky, this callback receives only the partitions actually moving away, not your whole assignment. For each one, drain or complete any in-flight records, flush buffered writes to downstream stores, and then commit offsets synchronously with commitSync. The callback runs on the poll thread before the partition is released, so this is your last chance to record progress while you still own it. An async commit here can lose the race with the handover.

onPartitionsAssigned: build state for the new partitions

The callback lists only newly added partitions and may be empty. Initialize per-partition state here: caches, counters, and any seek to an externally stored offset. Leave existing partitions alone, because they kept processing through the rebalance.

onPartitionsLost: drop the state, do not commit

Partitions are lost when the member was fenced, typically after missing the session or poll timeout. Another member may already own them and be processing from the last committed offset. A commit now will fail or overwrite valid progress. Discard the local state and abandon in-flight work. Override this callback explicitly. Its default implementation delegates to onPartitionsRevoked, which would attempt the commit.

Duplicates still happen

Incremental revocation reduces how much work stops during a rebalance. It does not remove duplicates. Records processed after the last commit get redelivered to the new owner, so handlers must be idempotent. Use upserts keyed on a business ID, or deduplicate on partition and offset.

// PartitionState is your own class: buffered writes, caches, counters per partition.
class RebalanceHandler implements ConsumerRebalanceListener {
  private final Consumer<String, String> consumer;
  private final Map<TopicPartition, OffsetAndMetadata> pending = new HashMap<>();
  private final Map<TopicPartition, PartitionState> state = new HashMap<>();

  RebalanceHandler(Consumer<String, String> consumer) { this.consumer = consumer; }

  // Poll loop calls this after a record's (idempotent, upsert-based) write succeeds.
  void markProcessed(ConsumerRecord<String, String> r) {
    pending.put(new TopicPartition(r.topic(), r.partition()), new OffsetAndMetadata(r.offset() + 1));
  }

  @Override
  public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
    // Cooperative sticky: only the partitions moving away are listed here.
    Map<TopicPartition, OffsetAndMetadata> toCommit = new HashMap<>();
    for (TopicPartition tp : partitions) {
      PartitionState s = state.remove(tp);
      if (s != null) s.flush();            // finish in-flight work and flush downstream first
      OffsetAndMetadata o = pending.remove(tp);
      if (o != null) toCommit.put(tp, o);
    }
    if (!toCommit.isEmpty()) consumer.commitSync(toCommit); // sync: we still own these partitions
  }

  @Override
  public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
    // Only newly added partitions (possibly none); existing ones keep their state.
    for (TopicPartition tp : partitions) state.put(tp, new PartitionState(tp));
  }

  @Override
  public void onPartitionsLost(Collection<TopicPartition> partitions) {
    // Fenced: another member may own these already. Do NOT delegate to onPartitionsRevoked.
    for (TopicPartition tp : partitions) {
      PartitionState s = state.remove(tp);
      if (s != null) s.discard();          // abandon in-flight work; new owner resumes from last commit
      pending.remove(tp);                  // never commit these offsets
    }
  }
}

A ConsumerRebalanceListener that flushes and synchronously commits only revoked partitions, initializes state only for newly assigned ones, and discards state without committing when partitions are lost.

How does static membership stop restarts from triggering rebalances?

Static membership gives each consumer a persistent identity through group.instance.id. When a static member shuts down, it does not send a leave request, and the coordinator keeps its partition assignment reserved. If a member with the same ID rejoins before session.timeout.ms expires, the coordinator returns the cached assignment. There is no rebalance, and the other members keep consuming without interruption.

The ID must survive restarts, and it must be unique at any moment. A StatefulSet satisfies both conditions because its pods have stable ordinal names such as orders-consumer-0 and orders-consumer-1. Inject the pod name through the Downward API and use it as the instance ID. Deployments generate a random suffix for each pod, so a restarted pod comes back with a new ID and triggers a rebalance anyway. If you work around this by mapping Deployment pods to fixed slots, a surge during a rolling update can briefly run two pods with the same slot. When two live members share an ID, the newer one fences the older one, and the older one receives a fenced-instance error and stops. Two pods that both restart on failure can then fence each other repeatedly.

Setting What it detects Static-membership guidance
session.timeout.ms Missed heartbeats (process gone) Longer than a typical restart. Capped by the broker's group.max.session.timeout.ms
heartbeat.interval.ms Heartbeat cadence About one-third of the session timeout or less
max.poll.interval.ms A stuck processing loop Unchanged. Set it from the worst-case batch processing time

The cost of a longer session timeout is slower failure detection. If a pod actually crashes and does not return, its partitions stay unassigned until the full timeout elapses, and consumer lag builds on those partitions for that whole period.

# StatefulSet pods get stable ordinal names (orders-consumer-0, -1, ...),
# so each pod reclaims the same group.instance.id after a restart.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: orders-consumer
spec:
  serviceName: orders-consumer
  replicas: 3
  selector:
    matchLabels:
      app: orders-consumer
  template:
    metadata:
      labels:
        app: orders-consumer
    spec:
      containers:
        - name: consumer
          image: registry.example.com/orders-consumer:1.0.0
          env:
            # Downward API: inject the pod name, which is stable per ordinal
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            # The app maps this to the consumer property group.instance.id
            - name: KAFKA_GROUP_INSTANCE_ID
              value: "$(POD_NAME)"
            # Longer than a typical restart; must not exceed the broker's
            # group.max.session.timeout.ms
            - name: KAFKA_SESSION_TIMEOUT_MS
              value: "60000"
            # About one-third of the session timeout or less
            - name: KAFKA_HEARTBEAT_INTERVAL_MS
              value: "15000"
            # Unchanged: sized from worst-case batch processing time
            - name: KAFKA_MAX_POLL_INTERVAL_MS
              value: "300000"

A StatefulSet that derives group.instance.id from the stable pod name via the Downward API, with session timeout set above restart time and heartbeat at a third of it.

How do you diagnose frequent Kafka consumer rebalances?

Start by finding out who is leaving the group and why. Each rebalance begins with a member joining or leaving, and both sides record a reason. On the broker, the group coordinator logs when it prepares a rebalance and names the triggering member. On the client, look for messages about leaving the group, heartbeat failures, or an expired poll timeout. Correlate the member IDs with pod names and timestamps. Restarts during deploys point to static membership configuration. Heartbeat failures point to network or GC pauses. Poll timeouts point to your processing code.

To inspect group state from the CLI, use kafka-consumer-groups.sh --bootstrap-server <host:port> --describe --group <group> for per-partition lag. Add --members --verbose to see which member owns which partitions, and --state to check whether the group is Stable or stuck in PreparingRebalance. Run it several times. If ownership keeps shifting or the state rarely reads Stable, the group is churning.

On the client side, chart the coordinator rebalance metrics, such as rebalance rate and time since the last rebalance, next to consumer lag. If lag rises in a sawtooth pattern that lines up with rebalance spikes, the rebalances are the cause and not just a symptom.

The most common culprit is a slow poll loop. When processing a batch takes longer than max.poll.interval.ms, the client proactively leaves the group, even though its heartbeats are healthy. Watch time-between-poll-max and last-poll-seconds-ago. If they approach the configured interval, neither the cooperative sticky assignor nor static membership will help. The member is genuinely leaving. To fix it, lower max.poll.records, move slow work off the poll thread with careful offset handling, or raise the interval deliberately.

When should you avoid these features, and what are the alternatives?

Static membership trades failover speed for stability. When a static member crashes, the broker keeps its partitions reserved until session.timeout.ms expires, and that timeout is usually raised so restarts can finish. If your service cannot tolerate those partitions sitting idle for that long, static membership is the wrong fit. It also requires a stable group.instance.id. Autoscaled Deployments with generated pod names cannot provide one, and the broker fences a member when a second one joins with the same ID. Check that every client library in the group supports it before you enable it.

Cooperative sticky assignment balances partitions by count, not by traffic, and it preserves that placement across rebalances. If a few partitions carry most of the load, stickiness keeps the hot ones on the same consumers. In that case, fix the key distribution or use a custom assignor instead.

Alternative What it addresses Trade-off
Eager assignment with tuned timeouts Few, short pauses in small groups Every rebalance still stops the whole group
Decoupling processing from polling Slow handlers exceeding max.poll.interval.ms Offset commits and ordering need careful handling
Reducing partition or consumer churn Rebalances triggered by topology changes Requires capacity planning up front
Rolling updates with low surge, clean close() on shutdown, autoscaler cooldowns Rebalances caused by deploys and flapping Deploys take longer

The configuration and listener changes in this guide take a few days of engineering time at most. That is small compared with the time teams lose to recurring lag incidents and reprocessing duplicates after every deploy.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka consumer rebalancing, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.