What is a Kafka partition key, and what ordering does it actually guarantee?
A partition key is a declaration of an ordering domain. When you key a record by order_id, you are saying that every event for one order must be consumed in the order it was produced, and that events for different orders carry no ordering relationship. Kafka enforces only that much. Records are ordered within a single partition by offset. Across partitions there is no ordering at all, so a consumer can process an event from partition 3 before an earlier-produced event from partition 7.
The default partitioner hashes the serialized key with murmur2 and takes the result modulo the partition count. The same key always lands on the same partition, but only while the partition count stays the same. If you add partitions, the modulo changes and most keys move. New events for an entity can then arrive on a different partition than its older, still-unconsumed events.
Records with a null key have no entity affinity. The producer spreads them across partitions to balance load, so they have no per-entity ordering.
Ordering can also break before a record reaches the broker. If a batch fails and is retried while a later batch to the same partition has already succeeded, the two batches are written out of order. This happens when retries is above zero and max.in.flight.requests.per.connection is greater than one. Setting enable.idempotence=true fixes this. It requires acks=all and limits in-flight requests to at most five. The broker then uses producer IDs and sequence numbers to reject duplicates and out-of-sequence writes.
bootstrap.servers=broker1.example.internal:9092,broker2.example.internal:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5
retries=2147483647
delivery.timeout.ms=120000
A producer configuration that enables idempotence with acks=all and at most five in-flight requests, so retried batches cannot reorder records within a partition.
How does it work? Following a keyed record from producer to consumer
A partition key strategy works because each stage of the pipeline preserves a narrow, well-defined guarantee. Chaining those guarantees gives you per-key ordering from end to end:
- Serialization. The producer serializes the key with the configured key serializer. The partitioner hashes these bytes, not the object in your code. Two producers that serialize the same logical key differently, for example a string and an integer, will route it to different partitions.
- Partition selection. The partitioner maps the key bytes to a partition. With the default partitioner, that is the murmur2 hash modulo the partition count. Records with a null key are not tied to any partition and are distributed for load balance instead.
- Batching and sending. The producer groups records per partition into batches and sends each batch to that partition's leader broker. With idempotence enabled, sequence numbers stop retries from reordering batches within a partition.
- Append. The leader appends the batch to the partition log, assigns increasing offsets, and replicates it to followers. With
acks=all, the producer gets an acknowledgement only after the in-sync replicas have the data. - Assignment. The consumer group protocol assigns each partition to exactly one member of the group. That one-owner rule is why a partition is consumed sequentially, and why partition count limits group parallelism.
- Fetch and process. The owning consumer fetches records in offset order and commits offsets as it makes progress. From this point, ordering depends on your code. If the consumer processes records concurrently, it must keep records for the same key in sequence, as described in the consumer section below.
Every later section is about one of these links breaking. Changing the partition count breaks step 2, retries without idempotence break step 3, hot keys overload step 5, and a careless thread pool breaks step 6.
Who should use a deliberate partition key strategy?
Any team whose consumers apply state changes should choose keys on purpose rather than accepting whatever the producer code happens to set. In practice this includes:
- Services that maintain state from events. Examples are order lifecycles, account balances, inventory counts, and any consumer that builds a materialized view. These depend on per-entity ordering.
- Change data capture pipelines. Row-level changes must be applied in sequence per primary key, or the downstream copy diverges from the source.
- Multi-tenant platforms. Tenant size varies widely, so key choice determines whether one large tenant can stall a partition. The skew sections below apply directly.
- Teams planning for growth in consumer count. Partition count and key distribution set the ceiling on how far a consumer group can scale out, and both are hard to change later.
- Schema owners and platform teams. They define keys for topics that many services consume, so a mistake affects every downstream team.
It matters much less for independent events such as log lines, metrics, or clickstream data that is only aggregated over windows. There, null keys or loosely chosen keys are fine, and load balance matters more than ordering. If no consumer cares about the relative order of two events, you do not need an ordering domain.
How do you choose the right partition key from your domain model?
Idempotence protects ordering within a partition. The key decides which records share a partition in the first place. Records without a key are spread across partitions with no ordering relationship between them. That is fine for independent events, such as log lines or metrics. It is not fine when a consumer applies state changes. Whoever owns the event schema should make this choice deliberately, because changing the key later means migrating data.
A method: key by the narrowest ordered entity
List the state transitions your consumers apply. For each one, ask which entity's history must be replayed in sequence. "Order shipped before order paid" is wrong, so the order is an ordered entity. Then key by the narrowest entity that still covers every invariant.
The failure modes sit on either side of that choice:
- Keying too broadly caps parallelism. If you key by tenant and one tenant produces most of the traffic, one consumer does most of the work.
- Keying too narrowly breaks invariants without any visible error. Suppose a fraud rule enforces "at most three open orders per customer," and events are keyed by order ID. That customer's orders now land on different partitions and are processed concurrently. The rule worked earlier only because traffic was low enough that the races rarely happened.
| Candidate key | Ordering scope | Cardinality | Skew risk | Invariants protected / broken |
|---|---|---|---|---|
| Order ID | Single order | Very high | Low | Protects order lifecycle; breaks per-customer limits |
| Customer/account ID | All of a customer's events | High | Medium (power users) | Protects balances and per-customer rules |
| Device ID | Single device stream | High | Low–medium | Protects telemetry sequence; breaks fleet-level aggregates |
| Tenant ID | Entire tenant | Low | High | Protects tenant-wide rules; caps parallelism |
How many partitions do you need for consumer parallelism?
A consumer group assigns each partition to at most one member at a time, so partition count is a hard ceiling on parallelism. With 12 partitions, a 13th consumer sits idle. Size the partition count from expected peak throughput and per-message processing cost, as capacity planning.
Worked example (illustrative estimate). Assumed inputs:
- Peak rate: 6,000 messages/s
- Processing cost: 4 ms per message, handled sequentially within a partition, so 250 messages/s per partition
- Headroom factor: 1.5× for traffic spikes and catch-up after an outage
- Key distribution: the hottest key carries 5% of traffic
The base requirement is 6,000 ÷ 250 = 24 partitions. With headroom, that becomes 36. However, the hottest key produces 300 messages/s, and all of it lands on one partition that one consumer drains at 250 messages/s. That partition falls behind by 50 messages/s at peak, whether the topic has 36 partitions or 360. The real ceiling is per-key throughput. Adding partitions cannot raise it. Only cheaper processing or a different key strategy can.
Adding more partitions than you need also has costs:
| Cost | Why it grows with partition count |
|---|---|
| Broker overhead | Each partition replica carries its own log, index, and in-memory state |
| Open file handles | Every log segment and index file per replica stays open on the broker |
| Replication traffic | More leader–follower pairs to keep in sync, with more fetch requests |
| Rebalance and recovery time | More assignments to move, and more leaders to re-elect after a broker failure |
What causes partition skew, and how do you detect it?
Skew is rarely the hash function's fault. The default partitioner spreads distinct keys across partitions well enough. What it cannot fix is the shape of your key distribution. Two properties matter:
- Key cardinality. With only a few dozen distinct keys across many partitions, some partitions get several keys and others get none.
- Key frequency. Even with millions of keys, a single key that produces a large share of traffic sends all of it to one partition, because that is exactly what keying promises.
The typical case is a multi-tenant system keyed by tenant_id, where one enterprise tenant generates more events than thousands of small tenants combined. A flash sale on one product_id causes the same problem. All of that traffic lands on one partition. Within a consumer group, one partition is owned by one consumer, so that consumer falls behind while the others sit mostly idle. Adding consumers does nothing, because the hot partition still has only one owner.
Detecting it
Measure lag and throughput per partition. Topic-level totals average the hot spot away. The quickest check is kafka-consumer-groups.sh --describe --group <group>, which prints one row per partition with CURRENT-OFFSET, LOG-END-OFFSET, LAG, and the owning CONSUMER-ID.
Signs of a hot partition:
- One or two partitions show lag far above the rest.
- That lag keeps growing across repeated runs while the other partitions stay near zero.
LOG-END-OFFSETon the lagging partition advances much faster than on its siblings. This means the producer side is uneven, so the problem is the keys and not a slow consumer.
If lag is high but the offsets advance evenly, the cause is consumer performance, not skew.
#!/usr/bin/env bash
set -euo pipefail
BOOTSTRAP="broker1.example.internal:9092"
GROUP="orders-processor"
INTERVAL=60
snapshot() {
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --describe --group "$GROUP" \
| awk '$3 ~ /^[0-9]+$/ && $5 ~ /^[0-9]+$/ && $6 ~ /^[0-9]+$/ {print $2"-"$3, $5, $6}' \
| sort
}
snapshot > /tmp/lag_t0.txt
sleep "$INTERVAL"
snapshot > /tmp/lag_t1.txt
join /tmp/lag_t0.txt /tmp/lag_t1.txt \
| awk -v t="$INTERVAL" 'BEGIN {printf "%-24s %12s %12s %12s\n", "PARTITION", "PRODUCED/S", "LAG", "LAG_DELTA"}
{printf "%-24s %12.1f %12d %12d\n", $1, ($4-$2)/t, $5, $5-$3}' \
| sort -k3 -n -r
Takes two snapshots of kafka-consumer-groups.sh --describe a minute apart and computes the per-partition produce rate and lag growth. A partition with both a high produce rate and growing lag points to key skew.
What are the alternatives when a single key gets too hot?
Once you have confirmed the skew comes from one key, each fix trades away part of the ordering guarantee or moves it somewhere else.
Composite keys such as tenantId:orderId spread one tenant across partitions. Ordering now holds per order, not per tenant. This only works if nothing downstream needs to see one tenant's events in sequence.
Salting appends a bucket to the hot key, for example key + "#" + (hash(eventId) % N), so it lands on up to N partitions. Ordering breaks across buckets. If consumers need the original sequence, you must add a merge step that buffers by the original key and reorders by a producer-assigned sequence number. That step holds state, has to tolerate gaps, and is where latency builds up.
Dedicated topics for heavy tenants keep ordering intact and isolate the noise. The cost is that every consumer, ACL, and alert now has to handle a second topic.
Custom partitioners couple every producer to the same logic. A service in another language that falls back to its client's default partitioner sends the same key to a different partition. For example, librdkafka-based clients do not default to the Java client's murmur2 hash. When that happens, ordering breaks without any error.
| Option | Load balance | Ordering impact | Operational cost | Failure risk |
|---|---|---|---|---|
| Composite key | Good | Narrowed to sub-entity | Low | Hidden cross-entity assumptions |
| Salting | Tunable via N | Lost across buckets; restored only by merge | Medium–high | Merge bugs, buffering stalls |
| Dedicated topic | Isolated | Preserved | Medium | Topic sprawl, missed subscriptions |
| Custom partitioner | Flexible | Preserved only if all producers agree | High | Silent cross-client mismatch |
import json, zlib
from collections import defaultdict
from confluent_kafka import Producer
N_BUCKETS = 8 # salt fan-out for the hot key
producer = Producer({"bootstrap.servers": "kafka-1.internal:9092"})
seq = defaultdict(int) # per-original-key counter; assumes ONE producer owns each key
def send(key: str, event_id: str, payload: dict) -> None:
seq[key] += 1
# Stable hash (not Python's per-process randomized hash()) so a retried
# event lands in the same bucket every time.
bucket = zlib.crc32(event_id.encode()) % N_BUCKETS
value = json.dumps({"key": key, "seq": seq[key], "data": payload})
producer.produce("orders", key=f"{key}#{bucket}", value=value)
next_seq = defaultdict(lambda: 1)
pending = defaultdict(dict) # original key -> {seq: event}; dict dedupes redeliveries
def on_message(raw: bytes, emit) -> None:
ev = json.loads(raw)
k, s = ev["key"], ev["seq"]
if s < next_seq[k]:
return # duplicate of an already-emitted event
pending[k][s] = ev
# Release only a contiguous run; one missing seq stalls the whole key.
while next_seq[k] in pending[k]:
emit(pending[k].pop(next_seq[k]))
next_seq[k] += 1
# Still needed in production: a gap timeout policy, bounded buffers,
# and committing offsets only after events are emitted, not when buffered.
A salted producer key that spreads one hot key across eight buckets, paired with the stateful consumer-side merge that buffers by original key and reorders by producer-assigned sequence number to restore per-key order.
When should you avoid changing partition counts or relying on partition ordering?
The default partitioner hashes the key and takes the result modulo the partition count. If you add partitions to a live keyed topic, that modulo changes, so many keys start landing on a different partition. Existing records stay where they were written. For a while, events for order-42 can sit on both partition 3 (older) and partition 9 (newer). Different consumers drain those partitions at different speeds, so the newer event can be processed first. Kafka also cannot reduce a topic's partition count, so the change is one-way.
| Approach | Ordering during change | Operational cost | Main risk |
|---|---|---|---|
| Add partitions in place | Broken for remapped keys until old partitions drain | Low | Silent out-of-order processing |
| Over-provision up front | Unaffected | Higher broker and file-handle overhead | Paying for capacity you never use |
| New topic with dual writes | Preserved if consumers switch per key after draining | High | Duplicates and divergence between topics |
| Controlled cutover | Preserved if producers pause and the old topic drains | Medium | Brief write pause, coordinated deploys |
Partition ordering also breaks down outside a clean happy path. A retry topic reorders by design. A replay from an earlier offset re-emits old state after new state. A flow that spans two topics has no ordering across them. When any of these apply, don't depend on arrival order. Carry a per-entity version or sequence number in the payload, and have consumers reject or reconcile stale updates. Ordering then becomes a property of the data rather than of the transport.
How should consumers be implemented to preserve ordering and survive redelivery?
Kafka delivers a partition's records to a consumer in offset order, but that guarantee ends once the records reach your process. If the poll loop hands records to a shared thread pool, two updates for the same order can run on different threads and commit their effects in the wrong sequence. To parallelize safely, serialize work per key. Hash each record's key to one of N in-process lanes, each a single-threaded queue. Records for one key stay in order, and unrelated keys run concurrently. This gives you more throughput without adding partitions.
Per-key lanes complicate offset commits. You can only commit offset n after every record at or below n in that partition has finished, across all lanes. Track the lowest incomplete offset per partition and commit up to it. When a rebalance revokes a partition, stop accepting work for it, drain or abandon its lanes, and commit what has completed before releasing it. Otherwise the new owner reprocesses records you already applied, or skips records you never finished.
Delivery is at-least-once, so redelivery will happen. Handlers must be idempotent. The worker in the sketch below loads the entity's stored version, discards any event whose version is not newer, and writes the state and the new version atomically. A replayed or late event then has no effect.
This design takes some up-front work: choosing the key deliberately, adding version fields, and building lane-aware commits. That work is cheaper than migrating a live topic, investigating intermittent ordering bugs in production, or re-partitioning under load.
import queue, threading, zlib
N_LANES = 8
lanes = [queue.Queue() for _ in range(N_LANES)]
store = {} # order_id -> (version, state); stand-in for a DB table
store_lock = threading.Lock()
def apply_if_newer(key, version, state):
# Check and write must be one atomic step in the real store,
# e.g. UPDATE orders SET ... WHERE id = :key AND version < :version
with store_lock:
current = store.get(key, (0, None))[0]
if version <= current:
return False # stale or duplicate event: redelivery is a no-op
store[key] = (version, state)
return True
def lane_worker(q, on_done):
while True:
rec = q.get() # dict: key, version, state, partition, offset
if rec is None: # sentinel used on shutdown or partition revoke
break
apply_if_newer(rec["key"], rec["version"], rec["state"])
# Report completion so the poll loop can commit only up to the
# lowest offset that is still incomplete in this partition.
on_done(rec["partition"], rec["offset"])
def dispatch(rec):
# Stable hash: same key -> same lane -> applied in offset order.
# Different keys spread across lanes and run concurrently.
lane = zlib.crc32(rec["key"].encode()) % N_LANES
lanes[lane].put(rec)
def start(on_done):
for q in lanes:
threading.Thread(target=lane_worker, args=(q, on_done), daemon=True).start()
start(lambda p, o: None)
for v, off in [(2, 10), (3, 11), (2, 10)]:
dispatch({"key": "order-1042", "version": v, "state": f"v{v}", "partition": 0, "offset": off})
A per-key serialized worker that hashes each record key to a single-threaded lane and applies an update only if its version is newer than the stored one, so out-of-order or redelivered events have no effect.
How can a deliberate key strategy save engineering time?
Most of the time saved comes from avoiding work that is expensive to do after a topic is live. The savings come from a few places:
- Fewer data migrations. Changing a key or partition count on a live topic usually means dual writes or a coordinated cutover, plus reprocessing and validation. If you derive the key from your invariants and size partitions from peak load before launch, you often avoid that project entirely.
- Less time debugging intermittent ordering bugs. A key that is too narrow, a non-idempotent producer, or a shared thread pool produces failures that appear only under load and are hard to reproduce. Idempotence, version checks, and per-key lanes remove whole classes of these bugs, so you do not have to find them one incident at a time.
- Faster incident triage. When per-partition lag and produce rate are already monitored, an on-call engineer can tell in minutes whether lag comes from a hot key or a slow consumer. Without that, teams often scale consumers first, which does not help a hot partition.
- Scaling without redesign. Per-key lanes add processing parallelism inside each consumer without touching the topic. When load grows, the first step can be a configuration change instead of re-partitioning.
- Consistent behavior across services. Documenting the key and serializer in the event schema, and keeping the default partitioner unless you have a strong reason not to, prevents the cross-client mismatches described above. Teams adding a new producer have less to coordinate.
The trade-off is up-front design time: modeling ordering domains, adding version fields, and writing lane-aware commit logic. That cost is paid once, during design and review. The costs it replaces tend to come back with every migration, scaling event, and production incident.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kafka partition key strategy, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.