What does Kafka message ordering actually guarantee?
Kafka makes one ordering promise. Within a single partition, records are appended in the order the broker accepts them, and each gets a monotonically increasing offset. A consumer reading that partition sees records in offset order. Kafka makes no promise about ordering across partitions of the same topic, and none across topics. If two records land in different partitions, a consumer may process the later one first.
This rules out global ordering on any topic with more than one partition, and that is usually acceptable because most business requirements don't need it. "Process all events in order" almost always means "process all events for this entity in order." The shipping service does not care whether order 123 was paid before order 456. It does care that order 123's Created, Paid and Shipped events arrive in that sequence. If every event for an entity goes to the same partition, per-entity ordering follows directly from the partition guarantee.
Several kinds of system depend on this property:
- State machines reject or corrupt state when a transition arrives before its predecessor.
- Event-sourced entities rebuild state by replaying events, so the replay order must match the write order.
- CDC consumers apply row-level changes, and an
UPDATEapplied before itsINSERT, or a stale value overwriting a newer one, silently diverges from the source.
Ordering does not extend across topics or services. Suppose commands go to one topic and the resulting events go to another. Kafka gives you no relationship between the two streams. For workflows like this, you need explicit entity versions, sagas that track progress, or reconciliation jobs, because Kafka's ordering won't cover them.
How does per-key ordering work end to end?
Per-entity ordering is not a single Kafka setting. It depends on four layers, and each one can break it:
- Partitioning. The producer sets the entity ID as the record key. The partitioner hashes the key to choose a partition, so every event for that entity goes to the same partition. Kafka's per-partition ordering then applies to that entity.
- Producer delivery. The producer has to write records to the partition in the order the application sent them, including when it retries after transient errors. Idempotence with
acks=alland a bounded number of in-flight requests provides this. - Consumer assignment and dispatch. A consumer group assigns each partition to one consumer at a time, so records are read in offset order. If you parallelize inside a consumer, records with the same key must still go to the same worker, one at a time.
- Applying side effects. Commits, rebalances and retries can redeliver or reorder records at the handler. A per-entity version check in the target store ensures that a duplicate or stale event cannot overwrite newer state.
The rest of this article covers these layers in order. The key gives ordering on the partition, producer settings preserve it on the write path, consumer design preserves it on the read path, and version checks protect it when the other layers fall short.
Who should use per-key ordering?
Use it if your consumers would produce wrong results when two events for the same entity are applied in the wrong order. In practice, these are the teams that need it:
- Teams building services around entity lifecycles, such as orders, payments, accounts, subscriptions or shipments, where each event is a state transition and depends on the one before.
- Teams running event-sourced systems, where rebuilding an aggregate replays its events and the result depends on their order.
- Teams operating CDC pipelines that copy database row changes into search indexes, caches or downstream databases, where applying an older row image after a newer one corrupts the copy.
- Platform teams that own shared Kafka clusters and client libraries and want application teams to inherit correct ordering behavior from defaults (keying conventions, producer configuration, consumer patterns) so each team doesn't have to work it out separately.
You probably don't need it if your events are commutative or independent, such as metrics, log lines, click streams aggregated by time window, or notifications where a swap in order is harmless. You also don't need it if every consumer already discards stale writes by version and you are willing to rely on that alone. The section on when to avoid strict ordering covers these cases in more detail.
How do you choose a partition key for per-key ordering?
With the default partitioner, a non-null key is hashed and mapped onto one of the topic's partitions. The same key always lands on the same partition, but only while the partition count stays fixed. Adding partitions changes the mapping, which is why repartitioning gets its own section later.
Choose the key from the entity whose state transitions must be applied in sequence. If an order moves through created → paid → shipped and consumers break when they see shipped before paid, the key is the order ID. Set it explicitly on every send by passing the order ID as the record key, not just inside the payload. The partitioner only sees the key.
Two mistakes break ordering quietly. A null key lets the producer spread records across partitions, so events for one entity can land on different partitions and no longer have a defined order relative to each other. A key that changes during the entity's lifecycle, such as keying by status, sends paid and shipped for the same order to different partitions. The key has to be stable for the entity's whole life.
Granularity is a trade-off:
| Candidate key | Ordering scope | Parallelism | Risk |
|---|---|---|---|
| Customer ID | All of a customer's orders | Low; busy customers create hot partitions | Unrelated orders wait on each other |
| Order ID | One order's lifecycle | High | None, if orders are independent |
| Order line ID | One line item | Highest | Order-level transitions lose ordering |
| Status | None per entity | Fixed by status count | Breaks ordering entirely |
Pick the coarsest key that covers the ordering your business needs, and no coarser.
import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
public class OrderEventProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1.internal:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// Avoid reordering or duplicates within a partition when sends are retried.
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
String orderId = "order-48213";
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
// Key = order ID: stable for the order's whole life, so every
// transition hashes to the same partition and stays in send order.
for (String status : new String[] {"created", "paid", "shipped"}) {
String payload = "{\"orderId\":\"" + orderId + "\",\"status\":\"" + status + "\"}";
producer.send(new ProducerRecord<>("order-events", orderId, payload));
}
// Broken: no key. The order ID is only in the payload, which the
// partitioner never reads, so these events can land on different partitions.
// producer.send(new ProducerRecord<>("order-events", payload));
// Broken: key changes with the lifecycle. "paid" and "shipped" for
// the same order can hash to different partitions.
// producer.send(new ProducerRecord<>("order-events", status, payload));
}
}
}
A Java producer that sets the order ID as the record key so all of one order's state transitions go to the same partition, with commented-out null-key and status-key sends that would break that ordering.
How do you stop the producer from reordering messages?
A correct key still doesn't guarantee order if the producer itself reorders writes. The producer pipelines requests: with max.in.flight.requests.per.connection above 1, it can send batch B while batch A is still unacknowledged. If A fails with a transient error and is retried, B may already be written, so the partition ends up with B before A even though both share a key.
Idempotence fixes this. With enable.idempotence=true, the broker assigns the producer an ID, and the producer tags each batch with a per-partition sequence number. The broker rejects out-of-sequence batches and drops duplicates, so a retried batch lands in its original position. Kafka documents this guarantee for up to 5 in-flight requests per connection. Idempotence requires acks=all. Without it, a leader could acknowledge a write and then fail before followers copy it, and the sequence state would be lost with it.
The producer settings that matter:
enable.idempotence=trueacks=allmax.in.flight.requests.per.connection=5(or lower)retriesleft at its large default, bounded bydelivery.timeout.ms
These guarantees cover only retries inside the producer. If your code catches a send failure and calls send() again, the record gets a new sequence number. The broker treats it as a new write. If the first attempt actually succeeded, you get a duplicate. If later records for the same key were already sent, you get them out of order. Tune delivery.timeout.ms instead of wrapping sends in your own retry loop.
The trade-off is latency. acks=all waits for in-sync replicas to confirm each write, and idempotence caps pipelining. Turning either off raises throughput but allows reordering on retries again.
bootstrap.servers=broker1.example.internal:9092,broker2.example.internal:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5
delivery.timeout.ms=120000
request.timeout.ms=30000
A producer configuration that uses idempotence, acks=all and a bounded in-flight limit to preserve per-partition order across retries, and leaves retrying to the producer through delivery.timeout.ms instead of application code.
How does Kafka consumer concurrency work without losing order?
Within a consumer group, each partition is assigned to exactly one consumer at a time. That rule is what preserves order, and it also caps parallelism. If a topic has 12 partitions, a 13th consumer sits idle. When lag builds because each record is slow to process (database writes, HTTP calls), adding consumers stops helping once you reach the partition count.
The key-hashed worker pool
Keys only need ordering relative to themselves, so you can parallelize inside a partition. The flow looks like this: partition → dispatcher → worker queues → commit tracker.
- A single poll loop fetches records.
- The dispatcher hashes each record's key modulo N and pushes the record onto that worker's bounded queue. Each key always lands on the same worker, so its records are processed in order. Different keys from the same partition run in parallel.
- Each worker reports completed offsets to a per-partition tracker.
- The tracker commits only up to the lowest offset that is still in flight. The committed value is the next offset to read, so you commit the end of the contiguous completed run plus one. Anything above a gap waits.
This is usually cheaper in engineering time than adding partitions. Adding partitions remaps keys and breaks per-key order during the transition.
The costs
- Offset tracking. If you commit past an unfinished record, a crash loses that record silently. If you commit too conservatively, a restart produces a burst of duplicates.
- Backpressure. When queues fill up, pause the affected partitions rather than blocking the poll loop. A blocked poll loop can exceed the poll interval and trigger a rebalance.
- Shutdown and rebalance. On partition revocation, drain or abandon the in-flight work for those partitions and commit only what is actually done.
import queue, threading, zlib
from collections import deque
N_WORKERS = 8
queues = [queue.Queue(maxsize=500) for _ in range(N_WORKERS)] # bounded: full queue = backpressure signal
class PartitionTracker:
"""Per-partition commit tracker. commit_at is the next offset to read."""
def __init__(self):
self.lock = threading.Lock()
self.pending = deque() # offsets in dispatch order (ascending within a partition)
self.done = set() # finished, but possibly above a gap
self.commit_at = None
def dispatched(self, offset):
with self.lock:
self.pending.append(offset)
def completed(self, offset):
with self.lock:
self.done.add(offset)
# Advance only across the contiguous finished prefix; stop at the first in-flight offset.
while self.pending and self.pending[0] in self.done:
off = self.pending.popleft()
self.done.discard(off)
self.commit_at = off + 1
trackers = {} # (topic, partition) -> PartitionTracker
def dispatch(record):
"""Called from the single poll loop for every fetched record."""
tp = (record.topic, record.partition)
trackers.setdefault(tp, PartitionTracker()).dispatched(record.offset)
# Stable hash (not Python's per-process hash()): same key -> same worker -> per-key order.
idx = zlib.crc32(record.key or b"") % N_WORKERS # null keys all share one worker
queues[idx].put(record) # sketch only: in production, pause the partition when full, don't block the poll loop
def worker(q):
while True:
rec = q.get()
process(rec) # your slow DB write / HTTP call
trackers[(rec.topic, rec.partition)].completed(rec.offset)
A library-agnostic sketch of key-hashed dispatch to bounded worker queues, with a per-partition tracker that commits only up to the end of the contiguous completed run plus one.
Offset commits, rebalances and idempotent consumers
Where you commit decides which failure you get. Committing an offset before processing means a crash mid-handler loses that message, because the next consumer starts after it. Committing after processing means a crash between the side effect and the commit replays the message. Most ordered pipelines commit after and accept redelivery, so redelivery has to be harmless.
Rebalances widen that window. When partitions are revoked, the revocation callback is your last chance to finish or abandon in-flight work for them and commit what actually completed. Work that is still running after the callback returns can overlap with the new owner, which then processes the same keys concurrently and can apply them out of order.
Track versions per entity
Redelivery is not the only source of stale events. Producer bugs, multiple writers for one entity, and outbox or CDC replays can all deliver an older state after a newer one. Put a monotonically increasing version or sequence on every event and store the last applied value next to the entity. The consumer applies an event only if its version is greater than the stored one, and updates both in the same database transaction, for example with UPDATE ... WHERE id = ? AND version < ?. A zero-row result means the event is a duplicate or stale, and the consumer skips it.
Kafka's idempotent producer and transactions narrow the duplicate window inside Kafka. They do not reach your database writes or outbound HTTP calls, so those side effects still need their own version checks or idempotency keys.
-- PostgreSQL: store the last applied version next to the entity state.
CREATE TABLE account_balance (
account_id text PRIMARY KEY,
balance_cents bigint NOT NULL,
version bigint NOT NULL -- last applied event version for this entity
);
-- Run once per consumed event, in the same transaction as any other writes
-- for that event. $1 = account_id, $2 = new balance, $3 = event version.
-- The upsert handles both the first event for an entity and later updates.
INSERT INTO account_balance (account_id, balance_cents, version)
VALUES ($1, $2, $3)
ON CONFLICT (account_id) DO UPDATE
SET balance_cents = EXCLUDED.balance_cents,
version = EXCLUDED.version
-- Apply only if the event is newer than the stored state.
-- Redeliveries, replays and stale writes match no row and change nothing.
WHERE account_balance.version < EXCLUDED.version;
-- The consumer checks the affected row count:
-- 1 row -> event applied; commit the transaction, then commit the offset.
-- 0 rows -> duplicate or stale event; skip it, but still commit the offset
-- so the partition keeps moving.
A PostgreSQL upsert applies an event only when its version is higher than the stored version for that entity, so redelivered, replayed or stale events affect zero rows and the consumer skips them.
What are the alternatives for retries and dead-lettering when order matters?
Retries raise the same issue for ordering. The common pattern is to send a failed record to a retry topic and keep consuming, but this reorders the entity. Suppose order-42 v3 fails and goes to the retry topic. Meanwhile v4 and v5 for the same key are processed on the main topic. When v3 succeeds on a later attempt, it is applied on top of v5. The partition still delivered records in order, but your handler did not process them in order.
There are three alternatives:
- Block the partition. Retry the failed record in place with backoff and pause the partition until it succeeds or someone intervenes. Order is preserved, but every other key on that partition waits behind one bad record.
- Park the key. Record the failed key in a durable store. Divert all later records for that key to a holding area, keep processing other keys, and replay the held records in order once the failure resolves. This keeps both order and throughput, at the cost of real state management. The parked set must survive rebalances.
- Accept the reorder. Dead-letter the record and move on. This is safe when events are independent, such as metrics or notifications. It is also safe when the version check from the previous section discards stale writes.
| Approach | Ordering | Throughput impact | Operational effort | Implementation cost |
|---|---|---|---|---|
| Block partition | Preserved | Stalls all keys on partition | High: needs alerting on stuck partitions | Low |
| Park key | Preserved per key | Only the failed key waits | Medium: inspect and release parked keys | High |
| Dead-letter / retry topic | Lost for that entity | None | Medium: replay DLQ | Low |
Block the partition when failures are rare and usually transient. Park the key when one poison record must not hold up other entities. Dead-letter only when the events tolerate being applied out of order.
When should you avoid strict per-key ordering, and how do you plan for repartitioning?
Strict per-key ordering is a cost, not a default. Skip it when events are commutative (counter increments, set additions) or independent (audit entries that are never replayed in sequence). In those cases a null or random key spreads load evenly. Also skip it for cross-topic workflows. Kafka never orders across topics, so a saga spanning orders and payments needs explicit state checks rather than a shared key.
Hot keys are the other limit. Illustrative estimate, assumed inputs, not a measurement: take a 12-partition topic where one key carries 30% of traffic. The average partition sees about 8.3%. The hot key alone is 30 / 8.3 ≈ 3.6× that, before counting the other keys hashed to the same partition. Because ordering pins the key to one partition and one consumer thread, that entity's throughput can never exceed what a single worker can process, no matter how many partitions you add.
Adding partitions is not free either. The default partitioner maps a key by hashing it modulo the partition count. Changing the count moves most keys to a different partition. Older events for a key then sit on the old partition while newer ones land on the new one, and two consumers can process them concurrently.
| Option | Development cost | Operational cost | Why it works |
|---|---|---|---|
| Over-provision partitions up front | Low | Medium: more files, more rebalance work | The mapping never changes |
| Drain-and-cutover to a new topic | Medium: dual config, switch logic | High: pause producers, wait for lag to reach zero, coordinate the switch | No old and new events overlap |
| Version-aware consumers | High: per-entity versions, discard stale writes | Low: repartition live | Stale events lose on version comparison |
How can per-key ordering save engineering time?
The main savings come from avoiding work: ordering bugs that are hard to debug, and compensating code that teams write when they don't trust the pipeline's order.
- Fewer intermittent ordering bugs. A null key, an application-level retry around
send(), or a commit that runs past unfinished work all cause failures that appear only under retries, rebalances or load. These are hard to reproduce in test environments and slow to diagnose in production. Setting a stable key, idempotent producer settings and contiguous-offset commits from the start removes these failure modes instead of leaving them to be found during incidents. - No custom resequencing layer. Without a dependable per-key order, consumers often end up buffering events, sorting them by timestamp and guessing how long to wait for stragglers. Relying on the partition guarantee, plus a version check for the cases it doesn't cover, replaces that buffering code with a single conditional write.
- One duplicate-handling mechanism instead of many. The per-entity version check handles redelivery after a crash, overlap during a rebalance, outbox or CDC replays, and stale retries from a dead-letter topic. Without it, each failure path tends to get its own special-case handling.
- Scaling without a migration project. The key-hashed worker pool raises consumer throughput without changing the partition count, so you avoid a drain-and-cutover. Choosing a generous partition count up front, or building version-aware consumers, also keeps a later repartition from becoming a coordinated multi-team change.
- Ordering only where it matters. Choosing the coarsest key that covers the business requirement, and skipping strict ordering for commutative or independent events, avoids hot partitions and the throughput investigations that come with them.
- Shared defaults. Once a team agrees on a keying convention, a standard producer configuration and a consumer template with offset tracking and version checks, new services inherit correct behavior. Each service team no longer needs to rediscover how Kafka ordering interacts with retries and rebalances.
The upfront cost is real: versions on every event, a version column in the target store, and more careful consumer code. That cost is paid once, during design. Ordering bugs found later have to be fixed in production, usually alongside cleanup of corrupted downstream state.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kafka message ordering, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.