Event Ordering in Distributed Systems: Beyond Wall-Clock Time

Why wall-clock timestamps can't order events across services

Every host keeps its own clock, and NTP keeps those clocks only approximately in line. Between synchronizations, oscillators drift at different rates. When the daemon finds a large offset, it may step the clock instead of slewing it gradually, so a correction can move time backward and the same host can issue a later event with an earlier timestamp. Virtual machines add jumps when they are paused or migrated. Containers read their node's clock, so two pods on different nodes disagree by exactly as much as their nodes do. No two services share a single "now."

This becomes a correctness problem when timestamps resolve conflicts. Consider last-write-wins (LWW) keyed on physical time:

Illustrative estimate (assumed numbers, not a benchmark). Service A's host runs 40 ms fast. Service B's host is accurate. A user changes their shipping address through A at true time T, then corrects it through B 15 ms later.

• A stamps T + 40 ms.
• B stamps T + 15 ms.
• LWW keeps A's write, which is the stale one.

The general condition is: a newer write is lost whenever true gap < (offset of earlier writer − offset of later writer). Plug in your own skew bounds. Any burst of updates closer together than your worst-case skew is at risk.

Nothing fails visibly. No error is raised, the store reports success, and the user's correction is silently discarded.

This article's framing is that physical time remains useful for display, for log correlation, and for bounded-staleness checks such as "reject data older than five minutes." It should not decide which write wins unless you have an explicit uncertainty model, meaning known skew bounds that the system waits out or enforces.

Which ordering guarantee does your workflow actually need?

That rule matters because most ordering problems are not about time at all. They are about which events must be seen in which sequence, and by whom. There are three distinct guarantees, and each one costs more than the last.

• Per-entity order. Every event about one key (an order ID, account, or device) is applied in the order it was produced. Events for different keys may interleave freely.
• Causal order. If one event could have influenced another, every observer sees them in that order. A reply never appears before the message it answers. Unrelated events still have no defined order.
• Total order. Every consumer sees every event in one identical global sequence.

Most business workflows need only per-entity order. An order moving from created to paid to shipped depends on its own history. It does not matter whether another customer's payment landed first. Per-entity order comes from routing by key plus a sequence number checked on write. It scales horizontally because different keys never coordinate.

Over-provisioning is the expensive mistake. Forcing total order on per-entity data couples unrelated workflows, so one slow partition or hot key delays everything behind it.

How to implement per-entity order with sequence numbers and version checks

Per-entity ordering is enough for most workflows. The service that owns the entity assigns each change a version number, and the database decides which writer gets each number. A write updates the row only if its version column still holds the value the writer read, then increments it. Because this compare-and-set is atomic, two concurrent writers cannot both emit version 8. One succeeds, and the other gets zero affected rows and must re-read and retry. Write the event in the same transaction, for example through an outbox table, so the stored version and the emitted version cannot diverge.

Copy Producer        Versioned write              Partitioned stream        Consumer
 --------        ---------------              ------------------        --------
 order 42  --->  UPDATE ... SET v = 8   --->  hash(order_id) -> P3 ---> expects v=8
                 WHERE id = 42 AND v = 7      P3: [v6][v7][v8]         applies v=8
                 (0 rows -> retry)                                     v=10? buffer
                                                                       v<=8? drop

Partition the stream by entity key so every event for one entity lands in the same ordered partition. The consumer stores the last applied version per entity and accepts only last + 1. It drops lower versions as duplicates. It buffers higher versions until the gap fills, or rejects them after a timeout.

This design has three trade-offs:

• No cross-entity order. Events for order 42 and its customer can still be applied in any order relative to each other.
• Hot keys. All writes for one busy entity go to a single partition and a single consumer, so that entity's throughput is limited to what one partition and one consumer can handle.
• Re-keying. Changing the partition count or the key scheme on a live stream can put events for one entity on two partitions at once. To avoid this, drain the old layout first, or make the change explicit with a new stream version.

Copy-- Producer side (PostgreSQL): versioned write plus outbox row in one transaction.
-- The writer previously read order 42 at version 7.
BEGIN;

UPDATE orders
   SET status  = 'shipped',
       version = version + 1
 WHERE id = 42
   AND version = 7          -- compare-and-set: matches only if no one else wrote first
RETURNING version;          -- returns 8 on success, no row if we lost the race

-- If no row came back, the application issues ROLLBACK, re-reads and retries.
-- Otherwise it records the event with the same version it just stored.
INSERT INTO outbox (aggregate_id, version, event_type, payload)
VALUES (42, 8, 'OrderShipped', '{"status": "shipped"}');

COMMIT;

-- Backstop: a duplicate (aggregate_id, version) can never be emitted twice.
ALTER TABLE outbox
  ADD CONSTRAINT outbox_entity_version_uniq UNIQUE (aggregate_id, version);

-- Consumer side: apply event v=8 only if the last applied version is exactly 7.
UPDATE order_view
   SET status       = 'shipped',
       last_version = 8
 WHERE order_id     = 42
   AND last_version = 7;

-- 0 rows affected: read last_version to decide what to do with the event.
--   last_version >= 8 -> duplicate or stale event, drop it
--   last_version <  7 -> gap, buffer the event until the missing versions arrive
SELECT last_version FROM order_view WHERE order_id = 42;

PostgreSQL statements that bump an entity's version with a compare-and-set update, write the matching outbox event in the same transaction, and apply events on the consumer side only when they carry the next expected version.

How Lamport clocks, vector clocks, and hybrid logical clocks work

When causality crosses entities or replicas, a per-entity sequence is not enough. Logical clocks track "happened-before" without trusting wall time.

Lamport clocks

Each node keeps one counter. On a local event or send, c = c + 1. On receive with timestamp t, c = max(c, t) + 1. If A caused B, then L(A) < L(B). The converse fails: L(A) < L(B) does not tell you whether A caused B or the two were concurrent. Breaking ties by node ID gives a total order, but that order is arbitrary for concurrent events. A last-writer-wins merge built on it silently discards one of two conflicting updates.

Vector clocks and version vectors

Each node keeps one counter per writer. On a local event, V[self] += 1. On receive, set V[i] = max(V[i], W[i]) for every i, then increment V[self]. A precedes B if every entry of A is ≤ B and at least one is smaller. If neither vector dominates, the updates are concurrent, which is a real conflict to resolve. The vectors grow with the number of writers, so systems prune old entries. A pruned entry can make ordered updates look concurrent. The cost is justified when several replicas accept writes to the same state.

Hybrid logical clocks

An HLC pairs physical time with a logical counter. It takes the maximum of the local clock and any received timestamp, and it bumps the counter when physical time has not advanced. Timestamps stay near wall time and remain monotonic and causally consistent, which suits human-readable logs and TTLs. They still assume bounded skew. One node with a fast clock pushes timestamps forward on every node it talks to.

Copy# Lamport clock: one counter per node.
class LamportClock:
    def __init__(self):
        self.c = 0

    def tick(self):              # local event or send
        self.c += 1
        return self.c

    def receive(self, t):        # t = timestamp carried by the message
        self.c = max(self.c, t) + 1
        return self.c

# Vector clock: one counter per writer, keyed by node ID.
class VectorClock:
    def __init__(self, node_id):
        self.me = node_id
        self.v = {}

    def tick(self):              # local event or send
        self.v[self.me] = self.v.get(self.me, 0) + 1
        return dict(self.v)

    def receive(self, w):        # w = vector carried by the message
        for node, n in w.items():
            self.v[node] = max(self.v.get(node, 0), n)
        return self.tick()

def compare(a, b):
    # Missing entries count as 0.
    keys = a.keys() | b.keys()
    a_le_b = all(a.get(k, 0) <= b.get(k, 0) for k in keys)
    b_le_a = all(b.get(k, 0) <= a.get(k, 0) for k in keys)
    if a_le_b and b_le_a:
        return "equal"
    if a_le_b:
        return "before"          # a happened-before b
    if b_le_a:
        return "after"
    return "concurrent"          # neither dominates: a real conflict to resolve

Minimal Python versions of the Lamport and vector clock update rules, plus the vector comparison that tells ordered updates apart from concurrent ones.

Alternatives and when to avoid each approach

The simplest way to get a total order is to skip clocks entirely: route every event through a single global sequencer that assigns a monotonically increasing number. Every consumer then sees one unambiguous order, and comparing two events is a single integer comparison.

That simplicity costs you in three ways. Throughput is capped by one process. Every write gains a network hop to that process. When it is down, nothing can be ordered. Failover is the subtle risk. If a new leader takes over while the old one is still alive and handing out numbers, you get duplicate sequence numbers. If the new leader cannot learn the last number the old one issued, you get gaps. A fencing mechanism, such as an epoch or term number that storage checks and rejects when stale, is what makes failover safe. A sequencer without fencing does not actually guarantee order.

Clock-based approaches avoid the central bottleneck because each node stamps its own events. In exchange, they give you only a partial order or a bounded approximation of one.

Avoid when

• Vector clocks for single-writer entities. With only one writer, concurrency cannot occur, so a per-entity sequence number gives the same answer with one integer.
• A global sequencer when per-entity order is enough. Global serialization is expensive, and you would be paying for ordering between entities that never interact.
• Physical-timestamp conflict resolution without a bounded uncertainty model. "Latest timestamp wins" silently discards writes whenever skew exceeds the gap between conflicting updates.

Detecting a conflict does not resolve it. When a vector clock reports two updates as concurrent, you still need a rule to combine them. That rule is either application-level merge logic or a CRDT-style data type, and each adds design and testing complexity that you should budget for up front.

Consumer-side defenses for out-of-order and duplicate events

Most brokers and queues promise at-least-once delivery, so the consumer has to absorb duplicates and reordering. Build these defenses into the consumer from the start rather than adding them after an incident.

Idempotency keys. Each event carries a key that stays the same across producer retries, such as the entity ID plus its sequence number. The consumer records processed keys in the same transaction as the state change. If the record and the side effect commit separately, a crash between them causes either a lost update or a double-apply.

Deduplication windows. You can't keep every key forever, so retain them for a bounded window. Size the window from the longest realistic redelivery delay, including broker retention and consumer downtime, not from typical latency. For per-entity streams, storing the last applied version is often enough: anything at or below it is a duplicate.

Gap handling. When version N+2 arrives before N+1, buffer it and wait. Buffering adds latency for that entity and uses memory, and a hot entity with a missing event can grow the buffer without limit. Cap both the wait time and the buffer size. When the timeout fires, re-fetch current state from the source of truth. If that fails, route the entity to a parking queue and alert. Decide ahead of time what happens when a gap never fills. Silently skipping it corrupts state.

Test both paths. The happy path gets exercised constantly. The gap path, with its timeout, re-fetch, and alert, runs rarely and fails quietly. Write explicit tests that inject duplicates, reorder events, and drop events.

The example below checks the expected version, deduplicates by idempotency key, and resolves a gap after a timeout.

Copyimport time

GAP_TIMEOUT_S = 5.0   # longest we hold an entity waiting for a missing version
MAX_BUFFERED = 100    # per-entity cap so one stuck entity can't exhaust memory

class OrderedConsumer:
    def __init__(self, store, source, alerts):
        self.store, self.source, self.alerts = store, source, alerts
        self.pending = {}    # entity_id -> {version: event} waiting on a gap
        self.gap_since = {}  # entity_id -> monotonic time the current gap opened

    def handle(self, event):
        eid, ver = event["entity_id"], event["version"]
        if ver <= self.store.last_version(eid):
            return  # (entity_id, version) is the idempotency key: already applied
        self.pending.setdefault(eid, {})[ver] = event  # re-buffering a dup is harmless
        self._drain(eid)
        if len(self.pending.get(eid, {})) > MAX_BUFFERED:
            self._resolve_gap(eid)  # don't wait for the timer if the buffer is full

    def _drain(self, eid):
        buf, last = self.pending.get(eid, {}), self.store.last_version(eid)
        while last + 1 in buf:
            with self.store.transaction() as tx:  # state and version commit atomically
                tx.apply(buf.pop(last + 1))
                tx.set_last_version(eid, last + 1)
            last += 1
        if buf:
            self.gap_since.setdefault(eid, time.monotonic())  # still waiting on a gap
        else:
            self.pending.pop(eid, None); self.gap_since.pop(eid, None)

    def check_gaps(self):  # call periodically, e.g. from the poll loop
        now = time.monotonic()
        for eid in [e for e, t in self.gap_since.items() if now - t > GAP_TIMEOUT_S]:
            self._resolve_gap(eid)

    def _resolve_gap(self, eid):
        try:
            snap = self.source.fetch_current(eid)  # source of truth, not the stream
        except Exception:
            self.alerts.park(eid, self.pending.pop(eid, {}))  # never skip silently
            self.gap_since.pop(eid, None)
            return
        with self.store.transaction() as tx:
            tx.replace_state(eid, snap.state)
            tx.set_last_version(eid, snap.version)
        buf = self.pending.get(eid, {})
        self.pending[eid] = {v: e for v, e in buf.items() if v > snap.version}
        self.gap_since.pop(eid, None)  # restart the clock for any remaining gap
        self._drain(eid)

A per-entity consumer that drops events at or below the last applied version, buffers out-of-order events up to a size and time limit, and on timeout re-fetches state from the source of truth or parks the entity and alerts.

Operational hygiene and impact on cost and engineering time

Clocks on the hosts themselves

Measure durations such as timeouts, latencies, and lease expiry with a monotonic clock, such as CLOCK_MONOTONIC on Linux, and never by subtracting wall-clock readings. When the time daemon steps the wall clock, a subtraction can produce negative or inflated durations, and leases can expire early or late. Export each host's offset as reported by its NTP or PTP daemon. Alert when the offset exceeds the skew budget your design assumes, and alert separately on clock steps. A step means some timestamps written around that moment are suspect.

Cost and engineering time

Weaker guarantees cost less to build and run. Per-entity sequence numbers need only a version column and a conditional write. Total order needs a single sequencer or consensus, which adds latency and a component to operate. Vector clocks add metadata to every message and require merge logic. Pay for those only where the workflow requires them.

Explicit versions also change how failures show up. A rejected write with a version mismatch is visible in logs and can be reproduced in tests. A silent overwrite is often found weeks later.

Illustrative estimate (assumed numbers, not a benchmark): hours saved = incidents per year × (debug hours without versions − debug hours with versions). Assume 6 incidents, 16 hours and 3 hours: 6 × 13 = 78 engineer-hours per year.

Decision checklist

• Does order matter only within one entity? Use sequence numbers with version checks.
• Do you need to detect concurrent updates? Use vector clocks or application-level merge.
• Do you need timestamps that roughly track real time? Use hybrid logical clocks.
• Do you need a global total order? Use a single sequencer, and accept the cost.

Copyimport sqlite3
import time

class VersionConflict(Exception):
    pass

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE accounts (id TEXT PRIMARY KEY, balance INTEGER, version INTEGER)")
db.execute("INSERT INTO accounts VALUES ('acct-42', 100, 1)")

def update_balance(conn, acct_id, new_balance, expected_version):
    # Conditional write: applies only if no one else wrote since our read.
    cur = conn.execute(
        "UPDATE accounts SET balance = ?, version = version + 1 "
        "WHERE id = ? AND version = ?",
        (new_balance, acct_id, expected_version),
    )
    if cur.rowcount == 0:
        # Report the conflict instead of silently overwriting newer data.
        raise VersionConflict(f"{acct_id}: expected version {expected_version}")

# Measure durations with a monotonic clock, which is not affected by wall-clock steps.
start = time.monotonic()

update_balance(db, "acct-42", 150, expected_version=1)      # succeeds, version -> 2
try:
    update_balance(db, "acct-42", 90, expected_version=1)   # stale writer
except VersionConflict as err:
    print("rejected:", err)                                 # visible, testable failure

elapsed = time.monotonic() - start
print(f"elapsed: {elapsed:.6f}s")
print(db.execute("SELECT balance, version FROM accounts").fetchone())  # (150, 2)

A per-entity version column with a conditional UPDATE rejects a stale write with an explicit error instead of overwriting newer data, and time.monotonic() measures the elapsed time so wall-clock steps cannot distort it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to event ordering distributed systems, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.