What does Kafka actually guarantee about message ordering?

Kafka guarantees order within a single partition and nowhere else. Each partition is an append-only log. Records receive increasing offsets as the broker appends them, and a consumer reading that partition sees them in offset order. A topic is a set of independent partitions, so a topic with twelve partitions has twelve separate orderings and no ordering across them.

The default partitioner places a keyed record by hashing the key and taking the result modulo the partition count. All records with the same key land in the same partition, so their relative order holds, but only while the partition count stays the same. If you add partitions, the modulo changes and a key can move to a different partition. New records for that key can then be consumed before older ones still waiting in the old partition. Records sent without a key get no per-key placement.

Producer A ──► key=order-17 ─┐ hash % 3
Producer B ──► key=order-42 ─┤
                             ▼
   P0: [17:created][17:paid][17:shipped]   ← order holds for order-17
   P1: [42:created][42:paid]               ← order holds for order-42
   P2: [...]
        │            │
   Consumer 1   Consumer 2   ← no ordering between P0 and P1,
                               between topics, or between services

Kafka never orders events across topics or across services. A payment-captured event on one topic can be processed before the order-created event it depends on in another topic.

This makes ordering a design decision you have to make deliberately. It depends on which key you choose, how producers retry, and how consumers parallelize work. It matters to any team whose consumers apply state transitions that are only valid in sequence.

┌────────────┐ key=order-17  ┌────────────────────────────────┐
│ Producer A ├──────────────►│ default partitioner            │
└────────────┘               │ partition = hash(key) % 3      │
┌────────────┐ key=order-42  │ same key -> same partition,    │
│ Producer B ├──────────────►│ only while the count stays 3   │
└────────────┘               └───────────────┬────────────────┘
                                             │
 topic: orders   ┌───────────────────────────┘
                 │       offsets increase ──►
                 ├─► P0 [17:created][17:paid][17:shipped] ──► Consumer 1
                 │       └── order holds for order-17 ──┘
                 ├─► P1 [42:created][42:paid] ────────────► Consumer 2
                 │       └── order holds for order-42
                 └─► P2 [other keys ...] ─────────────────► Consumer 3

  NO ordering between P0, P1 and P2 (each partition is its own log)

 topic: payments
   P0 [payment-captured order-17] ──────────────────────► Payment consumer

  NO ordering across topics or services: payment-captured for
  order-17 can be processed before its order-created event

  Add a 4th partition -> hash % 4 -> order-17 may map elsewhere;
  its new records can be read before older ones still in P0

Keyed records keep their relative order only inside the partition their key hashes to, and only while the partition count is unchanged. Kafka provides no ordering across partitions, topics or services.

Who should use partition-key ordering?

Per-key ordering is worth the effort when a consumer's correctness depends on applying one entity's events in sequence. Typical cases:

  • Order and workflow services, where created → paid → shipped → refunded transitions are only valid in that order.
  • Account balances and ledgers, where debits and credits must be applied in sequence or summed without gaps or duplicates.
  • Read models and projections built from change data capture or an outbox, where a stale update must never overwrite a newer one.
  • Inventory and reservation systems, where a release processed before its reservation leaves stock counts wrong.

Within those teams, this matters most to the engineers who design topics and choose keys, and to the service owners who write consumers. The guarantees described here only hold when both sides cooperate. The producer has to pick the right key and retry safely, and the consumer has to preserve per-key order and check versions.

You can usually skip strict ordering for streams of independent facts, such as metrics, logs, clickstream events and notifications. Each record there stands alone. Null or random keys that spread load evenly are the better choice for those streams. Teams whose events can be made commutative or state-carrying should also consider the alternatives in the section on avoiding strict ordering before taking on the cost.

How does it work end to end?

Ordering in Kafka is not one feature. It is a chain of steps, and each step must preserve what the previous one established. For a single order whose status changes:

  1. The source of truth assigns a version. The service that owns the order updates its row and increments a per-entity version column in the same database transaction. It also writes an outbox row carrying that version.
  2. A relay publishes with the entity ID as the key. The outbox relay sends the event to one topic for that aggregate, using the order ID as the record key.
  3. The idempotent producer keeps send order through retries. With enable.idempotence=true, the broker uses the producer ID and per-partition sequence numbers to reject out-of-sequence writes, so a retried batch cannot land after a later one.
  4. The partitioner maps the key to one partition. The default partitioner hashes the key modulo the partition count, so every event for that order goes to the same partition while the count is unchanged.
  5. The broker appends and assigns offsets. Within that partition, offsets reflect append order.
  6. The consumer group assigns each partition to one consumer. Only one member of the group reads a given partition at a time, and it receives records in offset order.
  7. The consumer preserves per-key order internally. It either processes a partition on one thread, or shards records by key hash onto single-threaded workers, so two events for the same order never run concurrently.
  8. The consumer applies each event conditionally. A write that checks the stored version discards duplicates and stale events, and it detects gaps that need a gap policy.
  9. The consumer commits offsets safely. It commits each partition only up to its lowest unfinished offset, accepting that anything past that point may be redelivered and filtered out by step 8.

Steps 3 to 6 are Kafka's part. Steps 1, 2, 7, 8 and 9 are yours. Most ordering bugs come from one link in the chain missing, such as a null key, a thread pool without key affinity, or a consumer with no version check. The sections that follow cover each link in detail.

How do you choose a partition key for ordered events?

Start from the invariant, not the infrastructure. Ask which state transitions would corrupt data if applied in the wrong order. "Refund must follow capture" is an invariant on one order. "Balance must reflect debits in sequence" is an invariant on one account. The entity that owns the invariant is your key: order ID, account ID, or more generally the aggregate ID.

The common wrong defaults all miss this boundary:

  • Tenant ID orders far more than you need. One large tenant pins its whole load to one partition, and therefore to one consumer thread.
  • Event type splits a single entity's history across partitions, so OrderCancelled can be processed before OrderCreated.
  • Random or null keys spread records across partitions with no per-entity affinity, so you lose ordering entirely.
Key choice Ordering scope Load distribution Typical use
Tenant ID All events for a tenant Skewed; large tenants create hot partitions Rarely correct; only for genuinely tenant-wide invariants
Aggregate ID (order, account) Per entity Even when entity count is high Default for state-transition events
Event type Per type, across entities Very skewed Avoid for ordered data
Null / random None Even Independent facts such as metrics and logs

Coarse keys give you a stronger guarantee, but they cap throughput at what one partition and one consumer can handle. Fine-grained keys scale, but they only order events within each entity.

Ordering also applies only within a topic. If an aggregate's events must be applied in sequence, publish all of them to one topic under the same key. Splitting by event type into separate topics forces consumers to merge the streams and reconstruct the order themselves.

What causes out-of-order events in producers and consumers?

On the producer side, the classic cause is retries combined with pipelining. If a producer has several batches in flight to the same partition and an earlier batch fails transiently, the retry can land after a later batch that succeeded, so the log order no longer matches send order. Setting enable.idempotence=true fixes this. The broker tracks a producer ID and per-partition sequence numbers, and it rejects out-of-sequence writes. This preserves order across retries as long as max.in.flight.requests.per.connection stays at 5 or lower, with acks=all and retries enabled.

Keys cause the other producer-side problems. A null key lets the partitioner spread records across partitions, which removes any ordering relationship between them. A custom partitioner that routes on something other than the key, such as region or load, has the same effect.

On the consumer side, a single partition's order survives only until you hand records to a thread pool. Two events for the same key can finish in either order. The fix is to shard work by key: hash the key modulo N and send it to one of N single-threaded queues. Each key then has exactly one worker, and different keys still run in parallel.

Parallelism also limits what you can commit. The committed offset means "everything before this is done," so you can only commit up to the lowest unfinished offset per partition. Anything processed past that offset is redelivered after a rebalance or crash, so consumers will see duplicates and replays.

// Producer: idempotence keeps per-partition order across retries
Properties p = new Properties();
p.put("bootstrap.servers", "broker1:9092");
p.put("enable.idempotence", "true");
p.put("acks", "all");                                  // required by idempotence
p.put("retries", Integer.toString(Integer.MAX_VALUE));
p.put("max.in.flight.requests.per.connection", "5");   // must stay <= 5
p.put("key.serializer", StringSerializer.class.getName());
p.put("value.serializer", StringSerializer.class.getName());

// Consumer: N single-threaded workers; the same key always lands on the same worker
int n = 8;
ExecutorService[] workers = new ExecutorService[n];
for (int i = 0; i < n; i++) workers[i] = Executors.newSingleThreadExecutor();

Map<TopicPartition, ConcurrentSkipListSet<Long>> inFlight = new ConcurrentHashMap<>();
Map<TopicPartition, Long> nextOffset = new HashMap<>();

for (ConsumerRecord<String, String> rec : consumer.poll(Duration.ofMillis(500))) {
    TopicPartition tp = new TopicPartition(rec.topic(), rec.partition());
    inFlight.computeIfAbsent(tp, k -> new ConcurrentSkipListSet<>()).add(rec.offset());
    nextOffset.put(tp, rec.offset() + 1);
    int shard = Math.floorMod(Objects.hashCode(rec.key()), n);  // null keys all go to shard 0
    workers[shard].submit(() -> {
        handle(rec);
        inFlight.get(tp).remove(rec.offset());  // mark done
    });
}

// Commit only up to the lowest unfinished offset per partition;
// anything finished beyond it will be redelivered after a crash or rebalance
Map<TopicPartition, OffsetAndMetadata> commits = new HashMap<>();
inFlight.forEach((tp, pending) -> {
    Long lowest = pending.ceiling(Long.MIN_VALUE);       // null if nothing pending
    long safe = (lowest != null) ? lowest : nextOffset.get(tp);
    commits.put(tp, new OffsetAndMetadata(safe));
});
consumer.commitSync(commits);

An idempotent producer configuration, followed by a consumer that shards records by key hash onto single-threaded workers and commits each partition only up to its lowest unfinished offset.

How do event sequence numbers make ordering detectable?

Duplicates and reordering are unavoidable, so the consumer needs a way to recognise them. The partition offset can't do this. It reflects when a record reached the broker, not the order in which the entity's state changed. The fix is to have the source of truth stamp each event with a per-entity, monotonically increasing sequence number.

The most reliable place to assign it is the transaction that changes the state. Suppose an orders table has a version column. The update that ships order 42 sets status = 'shipped' and version = version + 1 in the same statement. The resulting version, say 8, goes into the event payload, typically through an outbox row written in that same transaction. Because the counter and the state change commit together or not at all, version 8 always describes the state that follows version 7. An aggregate version in an event-sourced system gives the same guarantee.

Wall-clock timestamps look tempting but cannot fill this role:

  • Clock skew: two hosts disagree about the current time, so a later write can carry an earlier timestamp.
  • Same-millisecond writes: two updates can share a timestamp and leave no tiebreaker.
  • Multiple writers: each writer's clock is a separate source of truth, and nothing reconciles them.

The cost is that a sequence number needs one authority per entity: a single writer or a transactional counter. If several services emit events about the same customer, none of them owns the counter. You must route all writes through one owner, or give each emitter its own sequence and accept that ordering across those streams is undefined.

-- PostgreSQL. Producer side: bump the version and write the outbox row in ONE transaction,
-- so the counter and the state change commit together or not at all.
BEGIN;

WITH shipped AS (
  UPDATE orders
     SET status  = 'shipped',
         version = version + 1          -- per-entity, monotonically increasing
   WHERE id = 42
     AND status = 'paid'                -- only ship from a valid prior state
  RETURNING id, status, version         -- e.g. version = 8
)
INSERT INTO outbox (aggregate_id, event_type, payload)
SELECT id,                              -- relay uses this as the Kafka partition key
       'OrderShipped',
       jsonb_build_object(
         'order_id', id,
         'status',   status,
         'version',  version            -- carried in the event, not derived from a timestamp
       )
  FROM shipped;

COMMIT;

-- Consumer side: apply an event only if it is newer than what we already hold.
-- A duplicate or late event (version <= stored version) updates zero rows and is skipped.
UPDATE order_view
   SET status  = 'shipped',
       version = 8
 WHERE order_id = 42
   AND version < 8;

The source database increments the order's version in the same transaction that changes its status and writes an outbox event carrying that version, and the consumer uses the version to ignore duplicate or out-of-order events.

How should consumers handle out-of-order and missing events?

The simplest consumer-side defence is to store the last applied version per entity, in the same database as the entity itself, and discard anything older. Make that check atomic with the write. A conditional upsert such as UPDATE accounts SET balance = $1, version = $2 WHERE id = $3 AND version < $2 lets the database reject stale events. Two consumer instances racing after a rebalance cannot both win.

Version-check dropping is wrong when every event carries meaning. Audit logs, ledgers and anything that sums deltas need all events. For these consumers, a gap is a correctness failure.

When a gap appears (you hold version 7 and receive 9), choose a gap policy per use case:

Strategy Correctness Latency cost Suitable consumer
Drop stale by version Final state correct; history lost None State projections, caches
Buffer and wait with timeout Full sequence if gap fills Up to the timeout; memory grows with open gaps Ledgers, workflows
Re-fetch from owning service Current state correct One synchronous call Read models tolerant of skipped steps
Last-writer-wins May apply older data None Low-stakes, overwrite-only data
Park to dead-letter topic Deferred; needs replay Unbounded for that entity Anything unresolvable

Buffering holds later events in memory per key. Many slow keys can exhaust the heap, so cap both the wait time and the number of buffered keys. Parked events should carry the entity key, expected and received versions, topic, partition and offset so they can be replayed deliberately.

Kafka transactions and exactly-once settings cover reads and writes within Kafka. They do not make database writes, emails or API calls exactly-once. Consumers still need idempotent writes.

-- PostgreSQL. Run inside the same transaction as any other writes for this entity.
-- Parameters: $1 = entity id, $2 = new balance, $3 = event version.

-- Pattern 1: state projection. Drop stale or duplicate events.
-- The version check and the write are one atomic statement, so two consumers
-- racing after a rebalance cannot both apply an older version.
INSERT INTO accounts (id, balance, version)
VALUES ($1, $2, $3)
ON CONFLICT (id) DO UPDATE
  SET balance = EXCLUDED.balance,
      version = EXCLUDED.version
  WHERE accounts.version < EXCLUDED.version
RETURNING version;
-- No row returned: the event was stale or a redelivery. Commit the offset and move on.

-- Pattern 2: ledger-style consumer. Every event must apply, in order.
-- Accept only the next version. Reject anything that skips ahead or repeats.
-- $2 here is a delta to add, not an absolute balance.
UPDATE accounts
   SET balance = balance + $2,
       version = $3
 WHERE id = $1
   AND version = $3 - 1
RETURNING version;
-- No row returned: look up the stored version.
--   stored >= $3     -> duplicate, safe to skip
--   stored <  $3 - 1 -> gap: buffer with a timeout, re-fetch, or park the event
--                       with key, expected/received version, topic, partition, offset

Two PostgreSQL conditional writes. The first drops stale events for a state projection. The second accepts only the next version for a ledger, so the consumer can detect a gap and apply its gap policy.

How can this approach save engineering time?

Ordering bugs are expensive mainly because they are intermittent. They appear under retries, rebalances, deploys and replays, which are exactly the conditions that are hard to reproduce. Designing the ordering chain up front moves that work from incident response to design review, where it is cheaper to do.

  • Fewer "impossible state" investigations. When every event carries a per-entity version and consumers apply it conditionally, a stale or duplicate event becomes a skipped write rather than corrupted data. You don't need to reconstruct offsets and timestamps across services to work out how an order became "shipped" before it was "paid".
  • Safe replays and rebalances. Consumers that tolerate redelivery let you reset offsets, rebuild a projection or restart a consumer group without writing one-off deduplication scripts or coordinating a freeze.
  • Diagnosable failures. A version gap is a specific, loggable condition with an expected and received version. Parked events that carry key, topic, partition and offset can be replayed deliberately instead of being hunted down in the logs.
  • Concurrency changes without a redesign. Once consumers shard work by key and commit only up to the lowest unfinished offset, you can raise worker counts without reintroducing reordering.
  • Planned partition changes. Choosing partition headroom early and having a migration path ready avoids an emergency re-keying project when throughput grows.
  • A reusable review checklist. The checklist at the end of this article turns ordering into a short set of questions answered once per stream, not a property rediscovered by each team that consumes it.

The time saved depends on how often your system replays, rebalances or retries, so it varies by team. The direction is consistent, though. Each link in the chain you build deliberately is one fewer class of production incident you have to debug from scratch.

When should you avoid strict per-key ordering, and what are the alternatives?

Strict per-key ordering means a consumer cannot move past an event until it succeeds. A slow downstream call or a poison message stalls that key. Offsets are committed per partition, so it also stalls every unrelated key sharing the partition. Retry and parking topics unblock the partition, but the parked event is then processed after later events for the same key. That is the reordering you were trying to prevent, unless you also park every subsequent event for that key until the first one clears.

Often the cheaper choice is to make order irrelevant:

Approach Why disorder stops mattering Trade-off
Commutative or idempotent events Applying events in any order yields the same result (e.g. set-add, counter increments with dedup) Not every domain operation can be expressed this way
State-transfer events Each event carries full current state plus a version, so consumers keep the highest version Larger payloads; schema changes affect every consumer
Re-fetch on notification The event only says "entity X changed", and the consumer reads current state from the owner Extra load and coupling to the source service
Last-writer-wins A timestamp or version decides conflicts and older writes are discarded Concurrent updates are silently dropped; clock skew matters if using timestamps
One topic per aggregate vs several One topic keeps all event types for a key in order; splitting by type loses cross-topic ordering Several topics are only safe if consumers tolerate interleaving

Ordering guarantees carry ongoing cost: careful keying, limited consumer concurrency, stored versions per entity, and gap-handling logic. Designing for order tolerance from the start costs less than debugging inconsistent state after a replay or rebalance.

How do you change partition counts or re-key a topic safely?

With the default partitioner, a keyed record's partition comes from a hash of the key modulo the partition count. Adding partitions changes that modulus, so many keys start landing on a different partition. Existing records stay where they were written. For a while, an entity's older events may still be unconsumed in partition 3 while its newer events arrive in partition 7. Two consumers can then process them in either order. Kafka also cannot reduce a topic's partition count. Choose the count with headroom up front, and accept that this means more partitions to monitor, rebalance and pay for.

When you must change the partition count or the key, migrate instead of altering the topic in place:

  1. Create a new topic with the target partition count or key.
  2. Point producers at the new topic. You can dual-write for a period, or cut over at a single point.
  3. Let consumers drain the old topic until their lag reaches zero.
  4. Switch consumers to the new topic.

Steps 3 and 4 can overlap, and dual-writing makes overlap certain. The stored per-entity version check protects you during that window. A consumer that already applied version 12 from the new topic will discard a late version 11 from the old one rather than overwrite newer state.

Checklist for any new event stream

  • Ordering boundary: which entity actually needs order?
  • Key choice: does the key match that boundary, and is it free of hot keys?
  • Producer idempotence: are retries safe from duplicates and reordering?
  • Consumer concurrency model: do you run per-partition or per-key workers?
  • Version field: is it monotonic per entity, and assigned by the source of truth?
  • Stale-event policy: do you drop or log late events, or alert on them?
  • Gap policy: do you buffer, refetch or proceed when an event is missing?
  • Repartitioning plan: what is the headroom, and what is the migration path?

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka message ordering, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.