Multi-Region Writes in Microservices: CRDTs vs Last-Write-Wins

Classify your data before choosing a conflict strategy

Multi-region writes in microservices means every region accepts writes for the same logical data, then replicates them to its peers asynchronously. This is active-active replication. Teams adopt it for two reasons. A user in Frankfurt should not wait on a round trip to Virginia to save a change. And a regional outage should not stop writes everywhere else. The price is that two regions can accept conflicting updates to the same record before either has seen the other's. You need a rule that resolves those conflicts deterministically on every replica.

That rule depends on what the data means, not on which database stores it. Sort each field into one of three classes before choosing a mechanism:

Commutative data can be merged without losing work, so a CRDT fits. Overwrite-style data tolerates silently dropping a concurrent edit, which is exactly what LWW does. Invariant-bearing data cannot be merged after the fact. Two regions can each sell the last unit, and no merge function can undo the second sale. That data needs one authoritative writer.

If most of your write volume is invariant-bearing, active-active brings you cost and complexity with little benefit. It pays off when the bulk of your writes fall into the first two classes.

How last write wins actually picks a winner

Every replica stores a timestamp with each value. When two versions meet, the higher timestamp wins. If the timestamps are equal, a deterministic tiebreaker such as region ID or node ID decides. The tiebreaker matters because every replica must choose the same winner independently. Without one, replicas could settle on different values and never converge.

The weak point is the timestamp itself. Suppose eu-west's clock runs 300 ms fast. A user writes there, and 100 ms later (in real time) a second write lands in us-east. The us-east write is newer but carries the smaller timestamp, so every replica discards it. Nothing errors and nothing retries. The newer data is simply gone.

Hybrid logical clocks (HLCs) narrow this gap. An HLC never moves backwards, and it advances past any timestamp it has observed. That means a write made after reading another write is always ordered after it. Truly concurrent writes, where neither saw the other, are still ordered by the physical component, so clock skew still picks the winner.

You also need to choose the merge granularity:

• Row-level LWW discards an entire record. A concurrent edit to an unrelated field is lost along with it.
• Field-level LWW merges each attribute independently. This keeps both edits, but it can produce combinations nobody wrote. If one region changes amount while another changes currency, the merged row may pair a euro amount with USD. Any invariant that spans fields breaks without warning.

Treat LWW as a per-field policy for deliberately losing data. Because it never fails visibly, instrument it. Count overwrites where the losing write's timestamp falls within your expected skew window, and log the values that were discarded.

Copyfrom dataclasses import dataclass

SKEW_WINDOW_MS = 500  # assumed worst-case clock skew between regions

@dataclass(frozen=True)
class Versioned:
    value: object
    ts_ms: int   # wall-clock or HLC physical component
    region: str  # deterministic tiebreaker so every replica picks the same winner

def lww(a, b):
    # Higher timestamp wins; equal timestamps fall back to region ID.
    return max(a, b, key=lambda v: (v.ts_ms, v.region))

def merge_fields(local, remote, on_discard):
    # Field-level LWW: each attribute is resolved independently.
    merged = {}
    for field in local.keys() | remote.keys():
        a, b = local.get(field), remote.get(field)
        if a is None or b is None:
            merged[field] = a or b
            continue
        winner = lww(a, b)
        loser = b if winner is a else a
        # A loss inside the skew window may have discarded the truly newer write.
        if loser.value != winner.value and winner.ts_ms - loser.ts_ms <= SKEW_WINDOW_MS:
            on_discard(field, loser, winner)
        merged[field] = winner
    return merged

# eu-west's clock runs 300 ms fast, so its earlier write gets the larger timestamp.
eu = {"amount": Versioned(90, 1_000_300, "eu-west"),
      "currency": Versioned("EUR", 900_000, "eu-west")}
us = {"amount": Versioned(95, 1_000_100, "us-east"),     # newer in real time, still loses
      "currency": Versioned("USD", 1_000_100, "us-east")}

row = merge_fields(eu, us, lambda f, lost, won: print(f"discarded {f}={lost.value!r} from {lost.region}"))
print({f: v.value for f, v in sorted(row.items())})
# -> discarded amount=95 from us-east
# -> {'amount': 90, 'currency': 'USD'}  (a combination neither region wrote)

A field-level LWW merge with a region-ID tiebreaker shows clock skew silently discarding the newer us-east amount and producing an amount/currency pair no user wrote, and it logs discards that fall inside the skew window so the loss is visible.

How CRDTs merge state: counters, registers and sets

CRDTs avoid choosing a winner by making the merge function itself deterministic. In state-based CRDTs, replicas exchange full state and combine it with a merge that is commutative, associative and idempotent. Order, batching and redelivery therefore don't matter, so plain gossip is enough. Operation-based CRDTs ship smaller deltas but require more from the transport. Each operation must be delivered exactly once, because applying an increment twice double-counts. Some types also need causal delivery: an OR-Set remove must not arrive before the add it observed.

The multi-value register does not resolve the conflict. It hands the conflicting values to your application, which then needs a domain rule or a UI that asks the user to pick one.

All of these types guarantee convergence, not correctness. Two regions can each decrement a PN-Counter for stock from 1, and both replicas then converge on −1. An OR-Set of usernames can converge with "alice" claimed by two accounts. Invariants that span replicas still need coordination.

Copy# State-based PN-Counter: two G-Counters (P and N), each keyed by replica ID.
class PNCounter:
    def __init__(self, replica_id):
        self.id = replica_id
        self.p = {}  # increments per replica
        self.n = {}  # decrements per replica

    def inc(self, k=1):
        self.p[self.id] = self.p.get(self.id, 0) + k

    def dec(self, k=1):
        self.n[self.id] = self.n.get(self.id, 0) + k

    def value(self):
        return sum(self.p.values()) - sum(self.n.values())

    def merge(self, other):
        # Element-wise max is commutative, associative and idempotent,
        # so merge order and redelivery do not change the result.
        for mine, theirs in ((self.p, other.p), (self.n, other.n)):
            for replica, count in theirs.items():
                mine[replica] = max(mine.get(replica, 0), count)

us, eu = PNCounter("us-east"), PNCounter("eu-west")
us.inc(1)          # one unit of stock
eu.merge(us)       # both regions now see stock = 1

# Network partition: each region checks stock locally and sells the last unit.
if us.value() > 0: us.dec()
if eu.value() > 0: eu.dec()

# Partition heals; gossip in both directions, including a duplicate delivery.
us.merge(eu); eu.merge(us); us.merge(eu)

print(us.value(), eu.value())  # -1 -1: converged, but the stock invariant is broken

A state-based PN-Counter in which two partitioned regions each sell the last unit of stock: after merging, both replicas converge on -1, which shows that a CRDT guarantees convergence but not correctness.

Concurrency detection and delete handling

Before any merge logic runs, a replica has to decide whether two versions actually conflict. A version vector makes that decision. Each region keeps a counter per writer and increments its own entry on every local write. If every entry in version A is less than or equal to the matching entry in B, A happened before B, and B replaces it with no conflict. Only when each vector has an entry greater than the other's are the writes concurrent, and only those go through LWW or a CRDT merge. Skipping this check is how LWW silently drops a write that was causally later but carried a lagging timestamp.

Deletes need the same causal treatment. If one region removes an item that another region still holds, a naive merge treats the surviving copy as new data and resurrects it. A tombstone records the delete along with its causal context, so the merge knows the item's absence is intentional.

A tombstone can be purged only once every replica has seen the delete. Otherwise a lagging replica reintroduces the item. Strictly, this means tracking each replica's acknowledged vector and collecting only below the minimum. Many systems approximate it with a fixed retention window. If a region stays partitioned past that window, it rejoins holding items whose tombstones are already gone, and deleted data comes back. Treat such a region as stale and rebuild it from a snapshot instead of merging it.

Metadata grows with topology and churn. Vectors gain an entry per writer, OR-Sets carry tags per add, and tombstones accumulate per delete. Key vectors by region rather than by pod, or their size will track your autoscaler.

Copy# Version vectors are keyed by region, not pod, so their size tracks topology rather than autoscaling.
REGIONS = ["us-east", "eu-west", "ap-south"]

def compare(a: dict, b: dict) -> str:
    keys = a.keys() | b.keys()
    a_le_b = all(a.get(k, 0) <= b.get(k, 0) for k in keys)
    b_le_a = all(b.get(k, 0) <= a.get(k, 0) for k in keys)
    if a_le_b and b_le_a:
        return "equal"
    if a_le_b:
        return "before"      # b causally supersedes a: replace, no conflict
    if b_le_a:
        return "after"       # a causally supersedes b: keep a, no conflict
    return "concurrent"      # only this case goes to LWW or a CRDT merge

def gc_floor(acked: dict) -> dict:
    # Pointwise minimum of the vectors each replica has acknowledged.
    # A replica that is partitioned keeps reporting its old vector and holds the floor down.
    return {r: min(acked[rep].get(r, 0) for rep in REGIONS) for r in REGIONS}

def can_purge(tombstone_ctx: dict, acked: dict) -> bool:
    # Safe only if every replica has seen the delete; otherwise a lagging copy resurrects the item.
    floor = gc_floor(acked)
    return all(floor.get(r, 0) >= n for r, n in tombstone_ctx.items())

# Sequential vs concurrent writes
print(compare({"us-east": 3, "eu-west": 1}, {"us-east": 3, "eu-west": 2}))  # before
print(compare({"us-east": 4, "eu-west": 1}, {"us-east": 3, "eu-west": 2}))  # concurrent

# Delete issued in eu-west at its 7th write; ap-south has not yet acknowledged it
tombstone = {"eu-west": 7}
acked = {
    "us-east":  {"us-east": 12, "eu-west": 7, "ap-south": 4},
    "eu-west":  {"us-east": 11, "eu-west": 9, "ap-south": 4},
    "ap-south": {"us-east": 10, "eu-west": 5, "ap-south": 4},
}
print(can_purge(tombstone, acked))  # False: purging now would let ap-south bring the item back

A minimal Python sketch that classifies two version vectors as sequential or concurrent, and purges a tombstone only once the pointwise minimum of every replica's acknowledged vector covers the delete.

Implementing active-active service boundaries with home regions

Pure active-active is rarely the right default. Most services hold a mix of data: some of it converges safely under merge, and some of it carries invariants, such as "balance never goes negative" or "a seat is sold once," that no merge function can enforce after the fact. Assign every entity or tenant a home region, and route writes to invariant-bearing aggregates only there. Convergent data, such as presence, counters, carts and preferences, can still be written in every region.

The routing lookup needs to be cheap and stable. Store the home-region mapping as data, cache it at the edge or in the service mesh, and change it rarely. A write that lands in the wrong region should be forwarded, not applied locally.

Failover of a home region is a correctness event, not just an availability event. If you promote a new home while the old one still accepts writes, you have two authorities for the same invariant. Fence the old region before promotion, for example with a lease or an epoch number that writes must carry, and accept that unreplicated writes may be lost or need reconciliation.

The cost of this design is latency. A user in a distant region pays a round trip to the home region on every invariant-bearing write. Re-homing tenants close to their users reduces that penalty.

Side effects need their own guard. A merge can change state after an email, webhook or card charge has already gone out. Consumers should therefore deduplicate on idempotency keys derived from the operation, not the row, and should treat the first observed state as provisional. When a later merge invalidates an action, issue a compensating action such as a refund or a correction notice.

Copytype Region = "us-east" | "eu-west" | "ap-south";
const LOCAL_REGION: Region = "eu-west";

// Edge/mesh cache of the tenant -> home mapping. Source of truth is a replicated table;
// epoch is bumped on every re-home or failover so a fenced old home can be detected.
const homeCache = new Map<string, { region: Region; epoch: number }>([
  ["tenant-4821", { region: "us-east", epoch: 7 }],
]);

interface Command {
  tenantId: string;
  opId: string; // client-generated per operation, reused on retries
  kind: "reserveSeat" | "updatePrefs";
  payload: Record<string, unknown>;
}

declare function mergeLocally(cmd: Command): Promise<void>;          // CRDT merge, any region
declare function forward(to: Region, cmd: Command, epoch: number): Promise<void>;
declare function commitIfEpoch(cmd: Command, epoch: number): Promise<boolean>; // conditional write
declare function enqueueSideEffect(idempotencyKey: string, cmd: Command): Promise<void>;

export async function handleWrite(cmd: Command): Promise<void> {
  // Convergent data is accepted in every region and merged later.
  if (cmd.kind === "updatePrefs") return mergeLocally(cmd);

  const home = homeCache.get(cmd.tenantId);
  if (!home) throw new Error(`no home region for ${cmd.tenantId}`);

  // Invariant-bearing writes are forwarded to the home region, never applied here.
  if (home.region !== LOCAL_REGION) return forward(home.region, cmd, home.epoch);

  // Store rejects the write unless its current epoch matches, so a region that was
  // fenced during failover cannot commit as a second authority.
  const committed = await commitIfEpoch(cmd, home.epoch);
  if (!committed) throw new Error("stale epoch: refresh home mapping and retry");

  // Side effects are keyed by the operation, not the row, so consumers can dedupe retries.
  await enqueueSideEffect(`${cmd.tenantId}:${cmd.opId}`, cmd);
}

A write handler that merges convergent data locally, forwards invariant-bearing writes to the tenant's home region, fences commits with an epoch, and emits side effects under an operation-derived idempotency key.

Testing convergence and monitoring conflicts in production

Compensating actions only work if merges are correct, and unit tests with hand-picked interleavings rarely find the cases that break them. Property-based tests do better. Generate a random batch of operations, deliver it to three or more simulated replicas in shuffled orders, duplicate some messages, and hold others back to model a partition that heals later. Then assert that every replica's state is identical after delivery finishes. If a failure shrinks to two operations and one duplicate, you have a missing idempotency check. You learn that in CI, not from a customer.

Production signals

Measure conflict rate per field or entity type, not one global number. A sudden spike on a single field usually means a new code path writes it from both regions. Replication lag tells you how wide the window for concurrent writes is. Tombstone backlog tells you whether garbage collection is keeping up. A growing backlog inflates storage and slows merges. Clock skew between regions matters wherever last-write-wins compares timestamps. Alert well before skew approaches your typical inter-write interval.

Reproducing anomalies

When a user reports that an edit "came back," a retained replication log plus causal metadata, such as vector clocks or dots, lets you replay that entity's exact operation history into a test harness. You can see which concurrent writes merged and why one won. You can then turn the incident into a regression test. Without that log, engineers spend incident time guessing at interleavings. With it, finding the cause becomes a replay.

When to avoid it, alternatives, and impact on cost and engineering time

Avoid multi-region writes when correctness depends on invariants that span records, such as balances that must not go negative, unique usernames, or inventory that cannot oversell. No merge function can enforce these without coordination, so you end up adding coordination back at a higher cost. Also avoid it if your team is too small to staff on-call for partition failures, or if most writes come from one geography and a single-writer primary with read replicas already meets your latency targets.

The development cost is front-loaded and hard to reverse. Converting a field to a CRDT changes its stored and serialized form. A plain integer, for example, becomes a per-replica map. Old readers cannot parse the new format, and collapsing it back discards causal metadata, so treat the migration as one-way.

Debugging means reasoning about interleavings instead of reading a sequence of events. Version vectors and tombstones increase replication traffic. Partitions, clock skew, and tombstone buildup add failure modes that on-call engineers need runbooks and training for.

The savings come from classification. When each field has a declared strategy, conflicts resolve by rule rather than by an engineer. That removes manual reconciliation scripts and the incident hours spent deciding which write should have survived.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to multi-region writes microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.