Kafka Partition Key Ordering: Key Design and Hot Partition Fixes

What does Kafka actually guarantee about message ordering?

Kafka guarantees order only within a partition. A topic with twelve partitions is twelve independent logs, and there is no ordering relationship between records in different partitions. Per-key ordering is a consequence of placement, not a separate guarantee. The default partitioner hashes the serialized key (murmur2) and takes the result modulo the partition count, so every record with the same key lands in the same log. Records with a null key are spread across partitions and have no ordering relationship to each other at all.

The producer can still break order inside a single partition. Suppose batches A and B for the same partition are in flight at once. A fails with a retriable error, B is written, and then A is retried and appended after B. Two conditions together allow this: retries enabled and more than one in-flight request per connection.

Enabling idempotence fixes it. The broker tracks a producer ID and a per-partition sequence number, and it rejects batches that arrive out of sequence until the gap is filled. With idempotence on, ordering holds even with up to five in-flight requests. The minimal settings are enable.idempotence=true, acks=all, and max.in.flight.requests.per.connection no higher than 5.

This protection has a limit. Sequence numbers are scoped to one producer session. If the process restarts and gets a new producer ID, the broker has no way to order its writes against the previous instance's writes. In the diagram, order holds only along each partition's lane and only for writes from one producer session.

Copybootstrap.servers=broker1:9092,broker2:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

enable.idempotence=true

acks=all

max.in.flight.requests.per.connection=5

retries=2147483647

delivery.timeout.ms=120000

A minimal producer configuration that turns on idempotence with acks=all and at most five in-flight requests, which keeps retried batches in order within each partition for a single producer session.

How does key-based partitioning work end to end?

A keyed record goes through the same path from producer to consumer every time. Each step either preserves per-key order or is a place where it can be lost.

1. Serialization. The producer runs the key through the configured key serializer. The partitioner hashes these bytes, not the original object. Two producers that serialize the same logical key differently, for example a string versus a JSON-encoded string, will send it to different partitions.
2. Partition selection. If the application sets an explicit partition on the ProducerRecord, that partition is used. Otherwise, if the key is non-null, the default partitioner computes a murmur2 hash of the key bytes, makes it non-negative and takes it modulo the topic's current partition count. If the key is null, the producer picks a partition itself, and records with null keys get no ordering relationship. A custom class set through partitioner.class replaces this logic entirely.
3. Batching and sending. Records for the same partition are collected into batches and sent to that partition's leader broker. The idempotence settings above keep retries from reordering these batches.
4. Append and replication. The leader appends each batch to the partition log and assigns increasing offsets. Followers copy the log in the same order. With acks=all, the producer gets an acknowledgment only after the in-sync replicas have the write.
5. Consumption. Within a consumer group, each partition is assigned to exactly one consumer at a time. That consumer receives the partition's records in offset order. From here on, order depends on how the application processes and commits those records, which is covered below.

The chain only holds if every step stays consistent: the same serializer, the same partitioner, a stable partition count, and a consumer that does not reorder records itself.

Who should rely on partition key ordering?

Per-key ordering is a good fit for teams whose consumers apply events as a sequence of state changes to a known entity, and who also need more throughput than a single partition can give:

• Entity state machines. Order, shipment or subscription lifecycles, where a Cancelled event processed before Created leaves the entity in an invalid state.
• Ledgers and balances. Account debits and credits, where applying events out of order can let a balance check pass when it should fail.
• Change data capture. Row-level change events keyed by primary key, where the last write applied must be the last write made.
• Event-sourced aggregates. Consumers that rebuild an aggregate by replaying its events in order.
• Device and session streams. Telemetry or command sequences per device, where steps must run in the order issued.

In each case the ordering need is per entity, not global. That is what makes the pattern scale: different entities run in parallel across partitions while each entity's events stay in sequence.

How do you choose a Kafka partition key?

The partition key is where you decide which events must stay in order relative to each other. Kafka keeps that order only within a partition, so the key defines your consistency boundary. This matters most to teams whose consumers apply state changes in sequence, such as balance updates, order state machines, or device firmware steps. For those consumers, a reordered pair of events produces wrong state rather than a slow query.

The rule is to key on the narrowest aggregate whose events must be applied in order. Common examples are an order ID, an account ID or a device ID. Tenant and region are too coarse. They place unrelated entities in one sequence, which limits parallelism and lets one large tenant overload a single partition.

Consider a payments service that emits OrderPlaced, PaymentCaptured and RefundIssued, where each payment debits a customer account.

Account ID keeps the balance correct because every debit for an account is applied in sequence. Order ID spreads load best, but two orders on the same account can then be processed in parallel, and a balance check can pass for both. Finer keys spread load more evenly and give a narrower ordering scope. Once a key is finer than an invariant, the invariant has to be enforced somewhere else, usually by the consumer or the database.

Why do consumers break per-key ordering, and how do you prevent it?

The broker keeps records for a key in order within a partition. The consumer is where that order is usually lost. Three patterns cause it:

• Parallel processing of one partition. A thread pool runs record 41 and record 42 for the same key at once, and 42 finishes first.
• Async handlers. The poll loop fires off non-blocking calls, and those calls complete in whatever order the downstream service replies.
• Retry topics and DLQs. A failed event moves to a retry topic. Later events for the same key keep flowing and apply first.

Per-key serial execution

You can keep parallelism and still preserve order by hashing each record's key to one of N worker queues. Every record for a key lands on the same worker, so each key is processed serially while different keys run concurrently.

Offsets need separate tracking. For each partition, keep the set of in-flight offsets. Commit only the offset just past the lowest one that has not finished. Committing anything higher risks skipping an unprocessed record after a rebalance.

Costs

• Bookkeeping. Every record adds an in-flight entry for each partition.
• Head-of-line blocking. One slow key holds back commits for its whole partition, and its worker queue fills up. Consumer lag rises even though other keys are healthy.

Retries

Retry topics and DLQs trade ordering for availability. A key-aware retry keeps a "blocked keys" set. When an event for a key goes to retry, later events for that key are parked behind it until the retry succeeds or is dead-lettered.

Copy// N single-threaded workers: a key always hashes to the same worker, so it runs serially.
ExecutorService[] workers = new ExecutorService[N];
for (int i = 0; i < N; i++) workers[i] = Executors.newSingleThreadExecutor();

// Per partition: offsets dispatched but not yet finished, kept sorted.
Map<TopicPartition, ConcurrentSkipListSet<Long>> inFlight = new ConcurrentHashMap<>();
// Per partition: next offset after the last dispatched record (poll thread only).
Map<TopicPartition, Long> dispatchedUpTo = new HashMap<>();

while (running) {
  ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
  for (ConsumerRecord<String, String> r : records) {
    TopicPartition tp = new TopicPartition(r.topic(), r.partition());
    ConcurrentSkipListSet<Long> pending =
        inFlight.computeIfAbsent(tp, k -> new ConcurrentSkipListSet<>());
    pending.add(r.offset());
    dispatchedUpTo.put(tp, r.offset() + 1);

    int w = Math.floorMod(Objects.hashCode(r.key()), N);
    workers[w].submit(() -> {
      handle(r);                // in-place retries here stall this key (head-of-line)
      pending.remove(r.offset());
    });
  }

  // A committed offset is the next record to read, so commit the lowest
  // unfinished offset; if nothing is pending, commit past everything dispatched.
  Map<TopicPartition, OffsetAndMetadata> commits = new HashMap<>();
  for (Map.Entry<TopicPartition, Long> e : dispatchedUpTo.entrySet()) {
    Long lowest = inFlight.get(e.getKey()).ceiling(Long.MIN_VALUE); // null if empty
    long safe = (lowest != null) ? lowest : e.getValue();
    commits.put(e.getKey(), new OffsetAndMetadata(safe));
  }
  consumer.commitSync(commits);
  // Omitted for brevity: pausing partitions when worker queues fill,
  // and draining or discarding state for partitions revoked in a rebalance.
}

A Java sketch of key-hashed dispatch to single-threaded workers. For each partition, the poll loop commits only up to the lowest offset that is still unfinished.

What happens to ordering when you change the partition count?

The default partitioner places a keyed record by hashing the key and taking the result modulo the partition count. Adding partitions changes that divisor, so most keys map to a different partition from that point on. A key's older events stay on partition 3 while its new events land on partition 7. Kafka orders records only within a partition, so a consumer may process the new event before an unprocessed old one. Kafka cannot rehash existing data in place, and it cannot reduce the partition count either.

Repartitioning is disruptive, so size the topic up front. Base the count on the peak consumer parallelism you expect, with headroom, because one partition is consumed by at most one consumer in a group.

Over-partitioning has its own costs. Each partition adds broker file handles, replication traffic and leader elections. It also adds more producer batches that fill less fully, and longer consumer group rebalances.

How do you diagnose a hot partition?

Start with three numbers per partition, collected over the same window: message rate, byte rate and consumer lag. Each one alone is ambiguous. Read together, they point to a cause.

For lag, kafka-consumer-groups.sh --bootstrap-server <broker> --describe --group <group> lists CURRENT-OFFSET, LOG-END-OFFSET and LAG for each partition. To get message rate, run it twice a few minutes apart and diff LOG-END-OFFSET. For bytes on disk per partition, use kafka-log-dirs.sh --describe. If one partition receives far more messages or bytes than the others, the skew is on the input side.

To find out which keys cause it, sample the topic with kafka-console-consumer.sh --property print.key=true --max-messages <n>. Pipe the output through cut, sort | uniq -c | sort -rn | head to get a frequency list. Check two things: the share held by the top key, and the total count of distinct keys.

Only the third row is a tuning problem. The first two mean the key doesn't match the real consistency boundary. Either the key groups together entities that don't need to be ordered relative to each other, or it has too little cardinality to spread the load. Adding partitions or broker capacity won't change where a hot key hashes.

Copy#!/usr/bin/env bash
set -euo pipefail

BROKER="broker1.example.internal:9092"
GROUP="orders-processor"
TOPIC="orders"
WINDOW=300        # seconds between snapshots
SAMPLE=100000     # messages to sample for key frequency

snapshot() {
  kafka-consumer-groups.sh --bootstrap-server "$BROKER" --describe --group "$GROUP" \
    | awk -v t="$TOPIC" '$2 == t {print $3, $5, $6}' | sort -n
}

snapshot > /tmp/snap1.txt
sleep "$WINDOW"
snapshot > /tmp/snap2.txt

echo "partition msgs_per_sec lag_before lag_after"
join /tmp/snap1.txt /tmp/snap2.txt \
  | awk -v w="$WINDOW" '{printf "%s %.1f %s %s\n", $1, ($4-$2)/w, $3, $5}'

kafka-console-consumer.sh --bootstrap-server "$BROKER" --topic "$TOPIC" \
  --from-beginning --max-messages "$SAMPLE" --timeout-ms 60000 \
  --property print.key=true --property key.separator=$'\t' 2>/dev/null \
  | cut -f1 > /tmp/keys.txt || true

echo "top keys (count key):"
sort /tmp/keys.txt | uniq -c | sort -rn | head -10
echo "distinct keys: $(sort -u /tmp/keys.txt | wc -l) of $(wc -l < /tmp/keys.txt) sampled"

A shell script that takes two consumer-group snapshots to compute per-partition message rate next to lag, then samples the topic to report the top keys and the distinct key count.

How do you fix a hot key without breaking correctness?

The fix has to change what the key means. Each option gives up a specific kind of ordering.

Sub-key by a finer entity. If consumers only need per-order sequence, key by tenantId:orderId instead of tenantId. You lose ordering across a tenant's orders, which is often acceptable.

Salt deterministically. Derive the bucket from a sub-entity, such as tenantId + ":" + (hash(orderId) % 8), and never from a random value. Random salt scatters one entity's events across partitions. A derived bucket keeps each order's events together.

Illustrative estimate (assumed inputs, not measured): with 12 partitions and one tenant producing 40% of traffic keyed by tenant ID, that 40% is capped at what one partition and one consumer can handle. With 8 buckets, the tenant spreads across up to 8 partitions at roughly 5% each. Order then holds only within each bucket.

Dedicated topic. Moving the heavy tenant to its own topic lets you size its partitions independently.

Custom partitioner. This couples every producer to the partition topology. If producers run different logic or see different partition counts, ordering breaks silently.

Over-partitioning to dilute hash skew raises broker metadata, file handle and rebalance costs. It also does nothing for a single heavy key, because that key still hashes to one partition.

Copyimport java.nio.charset.StandardCharsets;
import java.util.zip.CRC32;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

public final class SaltedKeys {
  // Shared contract: every producer must use the same bucket count and hash.
  // Changing BUCKETS remaps orders to new buckets and breaks in-flight ordering.
  static final int BUCKETS = 8;

  // Bucket is derived from the order ID, never from a random value,
  // so all events for one order land in the same bucket and partition.
  static String saltedKey(String tenantId, String orderId) {
    CRC32 crc = new CRC32(); // stable across JVMs and easy to reproduce in other languages
    crc.update(orderId.getBytes(StandardCharsets.UTF_8));
    long bucket = crc.getValue() % BUCKETS; // getValue() is non-negative
    return tenantId + ":" + bucket;
  }

  static void sendOrderEvent(KafkaProducer<String, String> producer,
                             String tenantId, String orderId, String payload) {
    // The default partitioner hashes this key, so order holds per bucket only.
    // Events from different buckets of the same tenant can interleave.
    producer.send(new ProducerRecord<>("orders", saltedKey(tenantId, orderId), payload));
  }

  public static void main(String[] args) {
    // Same order always yields the same key, e.g. "tenant-42:<bucket>".
    System.out.println(saltedKey("tenant-42", "order-1001"));
    System.out.println(saltedKey("tenant-42", "order-1001"));
    System.out.println(saltedKey("tenant-42", "order-1002"));
  }
}

Deterministic salting in a Java producer, where the bucket comes from a stable hash of the order ID, so a heavy tenant spreads across up to 8 keys while each order's events stay in one bucket.

What are the alternatives when strict ordering can't be kept?

Partition order is one way to get correct state, not the only way. The alternative is to make consumers tolerate disorder, so correctness no longer depends on where an event landed.

Per-entity versions. The producer stamps each event with a monotonically increasing version or sequence number for its entity. The consumer applies an event only if its version is newer than the stored one, using a single conditional write such as UPDATE accounts SET balance = ?, version = ? WHERE id = ? AND version < ?. If zero rows change, the event is stale and is dropped. Late arrivals and redeliveries become no-ops.

Idempotent and commutative updates. Set-membership changes, max-wins timestamps, and counter increments with deduplicated event IDs produce the same result in any order. If your events already have this shape, strict key ordering only buys you skew.

Bounded reordering buffers. The consumer holds events per entity until the next expected sequence number arrives, or until a time or size window expires. The bound is required. Without one, a single missing event stalls that entity forever and grows memory without limit.

Skip strict ordering on high-skew workloads where one key would saturate a partition. Be cautious about retrofitting versions, though. Every write path, including backfills, admin scripts and other services, must carry and check the version. One unversioned writer undoes the guarantee.

The trade-off is extra design effort now against incidents later. Choosing the narrowest correct key and adding version fields while the system is young costs a schema field and some review time. Waiting means repartition migrations, slow debugging of silent reordering, and repeated rework across every consumer.

When should you avoid relying on partition key ordering?

Partition key ordering is the wrong tool in several situations:

• You need a total order across all events. Kafka can only give this with a single partition, which caps throughput at one partition and one consumer per group. If global order is a real requirement, a single partition is an honest choice for low volume. At higher volume, redesign so the requirement becomes per entity, or order events at the consumer using timestamps or sequence numbers assigned elsewhere.
• Invariants span multiple keys. A transfer between two accounts touches two keys that may live on different partitions. Partition order cannot make that atomic. Enforce it with a database transaction, a conditional write, or a single coordinating entity that owns both sides.
• One key carries a large share of traffic and cannot be split. If the entity really must be ordered and its volume exceeds what one partition and one consumer can handle, no key change will help. You need consumers that tolerate disorder, as described above, or a different processing design for that entity.
• Events are already commutative or idempotent. Keying adds skew risk and consumer constraints for no benefit. Use null keys or a high-cardinality key chosen only for load spread.
• Partition counts must change often. Each expansion remaps keys, as covered above. If your topology changes frequently, design consumers to reject stale writes rather than trust placement.
• Consumers cannot process a key serially. If the processing model fans out each record to async calls with no per-key sequencing, partition order is lost at the consumer anyway. Fix the consumer or switch to version checks before depending on key placement.

How can good key design save engineering time?

Most of the time spent on ordering problems goes into finding them, not fixing them. Reordering bugs rarely raise errors. They show up as a wrong balance, an order stuck in an impossible state, or a CDC replica that disagrees with its source, often long after the events that caused it. Tracing that back to two events landing on different partitions, or a consumer thread pool finishing them out of order, means correlating offsets, partitions and timestamps across services.

Getting the design right early removes whole categories of that work:

• Choosing the narrowest correct key up front avoids a later repartition migration, with its new topic, dual configuration, replay and coordinated consumer cutover.
• Sizing partition counts for peak parallelism at creation avoids live expansions that remap keys and introduce disorder.
• Building per-key serial execution and key-aware retries into a shared consumer library means each team does not rediscover head-of-line blocking and offset-commit bugs on its own.
• Adding version fields to events from the start turns redeliveries, late retries and cross-partition races into dropped stale writes instead of incidents.
• Running the diagnosis script as a routine check separates key skew, hash skew and slow consumers in minutes, so teams do not spend time scaling brokers or consumers for a problem that more capacity cannot fix.

Illustrative estimate (assumed inputs, not measured): assume a repartition migration takes two engineers five working days for the new topic, backfill and consumer cutover, and that each silent-reordering incident takes one engineer three days to trace and correct data. If a poor key choice leads to one migration and two incidents in a year, that is 10 + 6 = 16 engineer-days. Against that, assume choosing the key, adding a version field and reviewing the consumer's commit logic takes one engineer two days at design time. Under those assumptions, the up-front work costs about an eighth of the later rework. Replace the inputs with your own team's figures before using this to make a decision.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka partition key ordering, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.