Event Ordering in Microservices: Clock Skew, Kafka Keys, Sequences

Why wall-clock timestamps cannot order events across services

Event ordering means every consumer applies changes to a given entity in the order they actually happened. Inside one process, a counter gives you that for free. Across services, the tempting shortcut is to stamp each event with the host's wall clock and sort by it. That shortcut fails for physical reasons.

NTP keeps clocks close to each other, but never identical. Each host drifts between corrections, and the offset between any two hosts is unknown and keeps changing. When the offset grows too large to correct gradually, NTP can step a clock backward, so a later event on that host carries an earlier timestamp. Even without steps, timestamps from two hosts can only be compared reliably when they differ by more than the combined error of both clocks.

Worked scenario

Illustrative estimate (assumed numbers, not a benchmark). Service A's clock runs 40 ms fast, so skew S = 40 ms. A writes the full account record with status "frozen" at true time T. Service B writes the record with a new credit limit at T + 15 ms, so the gap G = 15 ms. A stamps its event T + 40 and B stamps T + 15. Record-level last-write-wins keeps A's write and silently drops B's credit limit. Misordering is possible whenever G < S.

Now route both events by key account-42 to one partition. Assign versions from the account row, giving 7 for A and 8 for B. The consumer applies 7, then 8, and rejects any version at or below the one it already holds. B's update survives, whatever the clocks say.

Ordering has to be defined explicitly. It cannot be inferred from clocks.

Choosing an ordering scope: per entity, not total order

The first decision is what "in order" means for your system. Ask what goes wrong if a consumer sees two events in the wrong sequence. In almost every case the answer involves a single entity. A refund is applied before the payment it reverses. A device's "offline" status overwrites a newer "online". An account balance projection adds a debit twice. Each failure involves events about one order, one account or one device.

That is the scope worth guaranteeing. Total order, where every event in the system has one agreed position relative to every other, needs a single serialization point. It caps throughput at what that point can handle and turns it into a shared failure domain. Per-entity order can be partitioned. Events for order 42 go through one ordered path, and events for order 43 go through another in parallel. Ordering holds where it matters, and the system still scales horizontally.

This approach is for backend and platform engineers whose consumers apply state changes or project aggregates, meaning anything that folds a stream of events into current state.

Be explicit about what you give up. Per-key schemes say nothing about the relative order of events on different keys, and nothing across topics. If a workflow spans entities, such as reserving stock and then charging a customer, per-entity ordering will not sequence it. That needs one of three things:

• A saga that coordinates the steps.
• Causal metadata that lets a consumer wait for a prerequisite.
• A design that tolerates temporary inconsistency until the state converges.

How it works: Kafka partition keys, outbox and per-entity sequence numbers

The ordering guarantee comes from three parts working together: a per-entity version assigned at write time, an outbox that publishes that version reliably, and a Kafka key that keeps one entity's events in one ordered log.

Copy Order Service
      |  one DB transaction
      v
 +----------------------------+
 | orders: id=42, version=7   |
 | outbox: key=42, version=7, |
 |         payload            |
 +----------------------------+
      |  relay (poll or CDC)
      v
 topic "orders"  key=42 -> hash -> partition 3
 +-------+-------+-------+-------+
 |  P0   |  P1   |  P2   |  P3   |  ... v5, v6, v7
 +-------+-------+-------+-------+
                             |
                             v
               Consumer (sole owner of P3 in its group)
                             |
                             v
               version store: 42 -> last applied = 6

Kafka preserves order only within a partition. If the producer uses the aggregate ID as the message key, the default partitioner hashes every event for order 42 to the same partition. Within a consumer group, a partition is assigned to exactly one consumer at a time, so events for that entity are never processed concurrently. Changing the partition count changes the key-to-partition mapping, so do not resize a keyed topic casually.

The version comes from the database, not from a clock. The service increments version on the aggregate row in the same transaction as the state change. Row locking or optimistic concurrency makes the version strictly monotonic per entity, regardless of which host performed the write.

The outbox removes the dual write. Writing the outbox row in the same transaction means the state change, its version, and the pending event all commit together or not at all. A relay then publishes outbox rows to Kafka. Delivery is at least once, so the consumer compares each event's version with its stored last-applied value before applying it.

Copy+--------------------------------------+
| Order Service                        |
+--------------------------------------+
        |  single DB transaction
        v
+--------------------------------------+
| Database                             |
|  orders: id=42, version=7            |
|  outbox: key=42, version=7, payload  |
+--------------------------------------+
        |  relay (poll or CDC)
        v
+--------------------------------------+
| topic "orders"   key=42 -> hash      |
| +------+ +------+ +------+ +--------+ |
| |  P0  | |  P1  | |  P2  | |  P3    | |
| |      | |      | |      | |v5 v6 v7| |
| +------+ +------+ +------+ +--------+ |
+-------------------------------|------+
                                |
                                v
                 +----------------------------+
                 | Consumer (sole owner of P3 |
                 | within its consumer group) |
                 +----------------------------+
                                |
                                v
                 +----------------------------+
                 | version store              |
                 |  42 -> last applied = 6    |
                 +----------------------------+

The aggregate version and outbox row commit in one transaction, the relay publishes keyed by aggregate ID, and a single consumer per partition applies events in version order.

Implementing it: producer configuration, outbox writes and consumer version checks

Producer settings

Set acks=all so the leader confirms a write only after all in-sync replicas have it. This prevents an acknowledged event from disappearing in a leader failover. Set enable.idempotence=true as well. Without idempotence, a producer with several in-flight requests per connection can reorder writes. If batch A fails transiently, batch B succeeds, and A is then retried, A lands after B in the partition. With idempotence, the broker tracks a producer ID and a per-partition sequence number, so it rejects duplicates and out-of-sequence batches instead of appending them.

Outbox writes

In one transaction, increment the entity's version and read the new value back. For example, use UPDATE orders SET version = version + 1 WHERE id = $1 RETURNING version. Then insert the outbox row with the entity ID, that version and the payload. The row lock taken by the UPDATE serializes concurrent writers to the same entity, so versions never collide or skip.

Consumer version checks

Keep a last_applied_version per entity, and update it in the same transaction as the side effect. Compare each incoming version (v) with the stored version (s):

• v ≤ s: stale or duplicate. Discard it. This is required because the relay can republish rows after a crash, so the outbox still delivers at least once.
• v = s + 1: apply it.
• v > s + 1: a gap. Do not apply it. Buffer it, retry it, or fetch current state, as covered in the next section.

Serialize processing per key

Parallel workers break ordering even when the partition is ordered. If versions 5 and 6 run on different threads, 6 can commit first and be rejected as a gap, or overwrite 5's result. Route records to workers by key hash so that each entity is handled by one worker at a time. Commit offsets only after the work is applied.

Copy-- PostgreSQL. $1 = order id, $2 = event version, $3 = JSON payload.

-- Producer side: bump the version and write the outbox row atomically.
-- The UPDATE row lock serializes concurrent writers to the same order.
BEGIN;
WITH bumped AS (
  UPDATE orders
     SET status = 'shipped', version = version + 1
   WHERE id = $1
  RETURNING id, version
)
INSERT INTO outbox (aggregate_id, version, event_type, payload)
SELECT id, version, 'OrderShipped', $3::jsonb FROM bumped;
COMMIT;
-- A UNIQUE (aggregate_id, version) constraint on outbox catches any version collision.

-- Consumer side: the version check and the side effect share one transaction.
BEGIN;
-- First event for this entity: start from version 0.
INSERT INTO consumer_versions (entity_id, last_applied_version)
VALUES ($1, 0)
ON CONFLICT (entity_id) DO NOTHING;

-- Lock the row and read s. The application compares it with v ($2):
--   v <= s     -> stale or duplicate (relay redelivery): ROLLBACK and ack
--   v >  s + 1 -> gap: ROLLBACK, then buffer, retry or fetch current state
--   v =  s + 1 -> continue below
SELECT last_applied_version FROM consumer_versions
 WHERE entity_id = $1 FOR UPDATE;

UPDATE order_projection SET status = 'shipped' WHERE order_id = $1;

-- Guarded advance: affects 0 rows unless this is exactly the next version.
UPDATE consumer_versions
   SET last_applied_version = $2
 WHERE entity_id = $1 AND last_applied_version = $2 - 1;
COMMIT;

A PostgreSQL sketch that increments the entity version and inserts the outbox row in one transaction, and a consumer transaction that locks the stored version, discards stale or duplicate events, detects gaps, and advances the version together with the side effect.

When order still breaks and how consumers should respond

Even with keyed partitions and serialized handlers, ordering can still fail in three ways.

Partition count changes. The default partitioner picks a partition from a hash of the key modulo the partition count. Adding partitions moves many keys to a new partition, while their older events stay unconsumed in the old one. A consumer can then process event 8 before event 7. Size partitions for expected growth up front. If you must resize, pause producers, let consumers drain to zero lag, then add partitions. Alternatively, cut over to a new topic once the old one is fully consumed.

Gaps. A version check that sees 7 after 5 knows 6 is missing but not why. Event 6 may be delayed by a producer retry, or it may be lost because an outbox row was skipped. Pick a policy per consumer:

Every policy needs a metric for open gaps and a reconciliation job that compares consumer versions with the source.

Hot keys. One entity maps to one partition and one worker, so its throughput is capped at what a single handler can process. Adding consumers does not raise that ceiling. If one key dominates, split the entity into finer ordering scopes, or make its handler cheaper.

Alternatives: logical clocks, hybrid logical clocks and plain timestamps

Logical clocks are the textbook alternative to wall-clock time. Each one solves a broader problem than most event pipelines have.

A Lamport timestamp is a single counter. A service increments it on every event, and on receipt it sets the counter to the maximum of its own value and the incoming value, plus one. If event A happened-before B, A's timestamp is lower. The reverse does not hold: a lower number does not prove causality, and the clock cannot detect concurrent events.

A vector clock keeps one counter per participant, so it can tell "A caused B" apart from "A and B are concurrent." The cost is size. Every message carries an entry for every writer, and that becomes awkward when replicas or services come and go.

A hybrid logical clock (HLC) pairs a physical timestamp with a logical counter. Values stay close to wall time, which makes them readable, and they still respect happens-before. This only holds if clock skew stays within a known bound.

Most business pipelines need none of these clocks. When a single writer owns an entity, its sequence number already defines that entity's order. Logical clocks would also have to travel through every hop, including brokers and external systems you do not control. Even then, they detect conflicts but do not resolve them.

Wall-clock timestamps are still appropriate in three places:

• showing times to users
• approximate analytics
• time-windowed aggregation with an explicit allowed-lateness bound

They should never be the sole tiebreaker between conflicting writes.

When to avoid it, and the impact on cost and engineering time

Per-entity ordering is not free, so adopt it where a lost update would actually hurt. For low-contention data, such as a user's notification preferences, a display name, or a feature flag edited by one admin, timestamp-based last-write-wins is usually good enough. Concurrent writes to the same key are rare, and when skew picks the wrong winner, the user notices and saves again. Choose it deliberately, and document that it can silently drop updates.

The full design carries these costs:

• Sequence assignment. A per-entity counter needs a single writer per entity, or a row lock or compare-and-set on every write. Both couple writers and add contention on hot keys.
• Outbox relay. This is another process to deploy, monitor and scale. Its lag becomes your publish latency.
• Database load. Every event adds an outbox insert, plus relay reads and cleanup.
• Gap handling. Someone must own alerts for stalled consumers, parked events and replays.

Illustrative estimate (assumed numbers, not a benchmark): extra writes/s ≈ events/s × (1 insert + 1 delete). At an assumed 500 events/s, that is about 1,000 extra writes/s, before counting relay polling reads.

The return is engineering time. Lost updates caused by skew depend on timing and rarely reproduce in staging. An explicit ordering scope with version checks turns them into deterministic, logged rejections. Consumers that compare a single version per entity are easier to reason about and can replay a topic idempotently.

Decision checklist

• Can two writers change the same entity within your clock-skew window?
• Is a silently lost update costly or visible to users?
• Do consumers need to replay or rebuild state?
• Can your team operate and monitor a relay?

If the first two answers are no, last-write-wins is acceptable.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to event ordering microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.