What is active-active multi-region, and how do write conflicts actually happen?
In an active-active multi-region setup, every region runs the full set of microservices and accepts writes for the same data. Each region commits locally and then replicates its changes to the other regions asynchronously. Teams accept this complexity for two reasons. Writes land in a nearby region, so users avoid a cross-continent round trip on every write. And if one region goes down, the other regions keep serving reads and writes without a promotion step.
The cost comes from asynchronous replication. Because a region commits before its peers see the change, two regions can each accept a write to the same key during the replication lag window. Neither write is wrong locally. They only conflict when they meet. Partitions make this worse. If the link between regions breaks, or traffic shifts during a region failover, the regions can diverge for minutes instead of milliseconds. They then accumulate many conflicting writes that must be reconciled when the link heals.
Two regions editing one order
| t | Region A | Replication | Region B |
|---|---|---|---|
| 0 | order 42: status=PAID, addr=X |
⇄ in sync | order 42: status=PAID, addr=X |
| 1 | user sets addr=Y, commits |
A → B in flight | |
| 2 | support sets status=CANCELLED, commits |
||
| 3 | A ← B in flight | ||
| 4 | receives CANCELLED |
⇄ both applied | receives addr=Y |
| 5 | CANCELLED, addr=Y? |
CANCELLED, addr=Y? |
At t=5, each region must decide what to do with these writes. Both edits are valid on their own, but together they can produce a bad outcome: if the system keeps both changes, the order ends up cancelled with a freshly changed shipping address. If the regions apply different rules, they disagree permanently. The rest of this article covers how to make that decision deliberately.
User near Region A Support agent near Region B
sets addr=Y (t=1) sets status=CANCELLED (t=2)
| |
v v
+--------------------------+ +--------------------------+
| Region A (writable) | | Region B (writable) |
| +--------------------+ | | +--------------------+ |
| | Order service | | | | Order service | |
| +---------+----------+ | | +---------+----------+ |
| | commit | | | commit |
| v | | v |
| +--------------------+ | | +--------------------+ |
| | Order DB |--- async: addr=Y --->| Order DB | |
| | 42 PAID addr=Y |<-- async: CANCELLED -| 42 CANCELLED addr=X| |
| +--------------------+ | (lag window) | +--------------------+ |
+--------------------------+ +--------------------------+
| |
v v
+--------------------------------------------------------------+
| Merge at t=4..5: two concurrent writes to order 42 |
| Region A: addr=Y + incoming status=CANCELLED |
| Region B: status=CANCELLED + incoming addr=Y |
| Risk: cancelled order with a new address, or permanent |
| divergence if A and B apply different merge rules |
+--------------------------------------------------------------+
Both regions commit an edit to order 42 locally, then each receives the other's edit after the asynchronous replication lag, so both regions must resolve the same conflict with the same rule.
How does it work? The write path from request to convergence
Conflict resolution in an active-active system is a pipeline. Every write goes through the same stages, and each stage has one job.
- Routing. A client request reaches the nearest healthy region, typically through latency-based or geographic DNS, or through a global load balancer. For single-writer data, the service checks whether the local region owns the record and forwards the write to the owner if it does not.
- Local commit with version metadata. The service commits the write to the regional database in a local transaction. With the data, it stores whatever the resolution rule needs later: a version vector, a hybrid logical clock timestamp, an owner epoch, or per-region CRDT state. Any side effects, such as emails or payment calls, are written to an outbox table in the same transaction.
- Change capture and shipping. A replication stream reads committed changes from the database's replication log or a change-data-capture feed and ships them to every peer region. This happens after the commit, which is why the lag window exists.
- Conflict detection on apply. The receiving region compares the incoming version metadata with its local copy. If one version causally follows the other, the newer one is applied. If they are concurrent, the record is in conflict.
- Resolution by data class. The region resolves the conflict with the rule assigned to that entity: CRDT merge for merge-tolerant data, a domain merge function or a review queue for business objects, and append for event logs. Single-writer data should not reach this step, because ownership and fencing prevent concurrent writes in the first place.
- Convergence and side effects. Because every region runs the same deterministic rule on the same inputs, all regions end with the same value. Outbox consumers then run side effects, using idempotency keys so a replayed event does not repeat them.
The hard requirement is determinism in step 5. A rule that depends on local state, on arrival order, or on which region happens to run it will make regions diverge permanently. Each later section of this article covers one part of this pipeline in detail.
Who should use active-active, and when is active-passive the better choice?
Active-active pays off for two kinds of workload. The first is a global user base that needs low write latency. Reads are easy to serve locally with replicas or caches. Writes are the hard part, because a user in Sydney writing to a primary in Virginia pays a cross-ocean round trip on every request. The second is a strict availability target where even a short failover window breaks the budget. If promoting a standby takes minutes and your target allows only a few minutes of downtime a month, a single regional incident can use up the quarter.
Outside those cases, active-passive with fast, rehearsed failover is usually the better trade. Treat the choice as a cost decision, and count recurring engineering time as well as the cloud bill.
| Cost dimension | Active-active | Active-passive with fast failover |
|---|---|---|
| Capacity | Each region sized to absorb a peer's load | Standby can run smaller and scale up on promotion |
| Cross-region data transfer | Replication in every direction, on every write | One-way replication only |
| Failure modes | Split brain, replication lag visible to users, divergent merges | Mainly failover correctness and data-loss window |
| Merge logic | Designed, reviewed and tested per data type | Not needed |
| Ongoing engineering | Every new feature must define its conflict behavior | Failover drills on a schedule |
The last row matters most. Merge logic is not a one-time project. Each schema change and new endpoint reopens the question of what happens when two regions disagree. That cost scales with your team's output, not with your traffic.
When should it be used? A decision checklist
Use active-active multi-region writes when all of the following hold:
- Write latency matters to users in more than one geography. If most writes come from one region, or if users tolerate a cross-region round trip on writes, a single write region with local read replicas is simpler.
- Your availability target cannot absorb a failover. If a rehearsed promotion of a standby fits within your downtime budget, active-passive gives most of the resilience at a fraction of the complexity.
- Most of your data is merge-tolerant or append-only. Carts, preferences, counters, activity feeds and event logs fit well. If nearly everything is a balance, a stock level or a uniqueness constraint, you will route most writes to a home region anyway and gain little.
- The team can own merge logic as an ongoing responsibility. Someone has to define conflict behavior for each new entity, test it under concurrency, and watch conflict metrics in production.
Do not use it when:
- The core data needs strong, globally consistent invariants on every write, such as a ledger that cannot tolerate even temporary overdraft.
- The motivation is only read latency. Read replicas and caches solve that without multi-writer conflicts.
- Regulatory or data-residency rules require each record to live in one jurisdiction. In that case, single-home ownership per record is the model, not true multi-writer replication.
A common middle ground is selective active-active. Run all regions active for merge-tolerant and append-only data, and keep single-writer data homed in one region per record. The classification in the next section is how you decide which entity goes where.
How do you classify data by conflict tolerance?
The way to contain that cost is to stop treating replication as a system-wide setting and decide it per entity. For each table or aggregate, ask one question: if two regions accept a write to this record concurrently, can the results be combined without violating a business rule? The answer puts nearly every entity into one of three classes.
| Class | Examples | Why concurrency hurts or doesn't | Strategy |
|---|---|---|---|
| Single-writer | Account balances, inventory decrements, unique usernames | Two valid local writes can produce an invalid global state, such as overdraft, oversell or duplicate key | Route all writes to one home region per record |
| Merge-tolerant | Shopping carts, counters, presence, user preferences | Concurrent updates can be combined deterministically | CRDTs or field-level merge, writes accepted anywhere |
| Append-only | Domain events, audit logs, clickstream | Records are never updated, so writes cannot collide | Write locally, replicate asynchronously, order on read |
This decision is made once per entity, when the entity is designed, and recorded alongside its schema. Later debates become narrow ("is this new field merge-tolerant?") instead of reopening whether the whole platform should be strongly or eventually consistent. Most entities fall into the two cheaper classes, so the expensive single-writer path stays limited to data that needs it.
The cost is real, though. Engineers must know which model applies to each entity they touch. Operations that span classes are the hard part. Checking out a cart (merge-tolerant) to decrement inventory (single-writer) and emit an order event (append-only) cannot run as one local transaction. That flow needs a saga or an outbox pattern, with compensating actions for partial failure.
How does single-home ownership work, including failover and failback?
For single-writer data, the simplest way to avoid conflicts is to never have two regions accept writes for the same record. Each entity gets a home region, and only that region accepts writes for it. Every other region holds a read replica and forwards writes to the home region.
The home region can be stored as a field on the record, such as home_region on a user account. It can also be derived deterministically, for example by hashing the tenant ID onto a fixed region table. A stored field allows individual records to move. A deterministic mapping needs no lookup, but moving one key means changing the mapping.
The cost is latency for remote users. A user in Sydney whose account is homed in Frankfurt pays an intercontinental round trip on every write. Re-homing fixes this. Freeze writes for the record, wait for the target region's replica to catch up to the last committed write, flip the home field, then unfreeze. The freeze is what keeps both regions from treating themselves as the owner at the same time.
Failover is a consistency decision
When the home region fails, you have two options:
- Fence the old region and block writes until it recovers. This keeps consistency and gives up availability for the affected records.
- Promote another region. Any write the old region acknowledged but had not yet replicated may be lost or may later conflict with new writes.
Fencing has to be enforced, not assumed. Use an epoch or ownership token that storage checks, so a partitioned old region cannot keep committing writes.
When the old region comes back, it must not resume as owner. Diff its unreplicated writes against the new home's state, replay the ones that are safe, send the rest to manual review or compensation, and only then fail back.
import hashlib
LOCAL_REGION = "ap-southeast-2"
REGIONS = ["us-east-1", "eu-central-1", "ap-southeast-2"]
def home_for(tenant_id: str, record: dict) -> str:
# A stored home_region wins, so individual records can be re-homed.
if record.get("home_region"):
return record["home_region"]
# Otherwise derive it deterministically: same answer everywhere, no lookup.
digest = hashlib.sha256(tenant_id.encode()).digest()
return REGIONS[int.from_bytes(digest[:8], "big") % len(REGIONS)]
def handle_write(cur, tenant_id, record, new_data, forward):
home = home_for(tenant_id, record)
if home != LOCAL_REGION:
# Not the owner. Forward the write; remote users pay this round trip.
return forward(home, tenant_id, new_data)
if record.get("frozen"):
# A re-home is in progress, so neither region may commit.
raise RuntimeError("record is being re-homed; retry later")
# Fencing: storage applies the write only if this region still holds
# the epoch it read. A failover bumps owner_epoch, so a stale or
# partitioned old home matches zero rows instead of committing.
cur.execute(
"UPDATE accounts SET data = %s "
"WHERE tenant_id = %s AND home_region = %s AND owner_epoch = %s",
(new_data, tenant_id, LOCAL_REGION, record["owner_epoch"]),
)
if cur.rowcount != 1:
raise PermissionError("fenced: ownership moved to another region")
Write routing that uses a stored home_region or a hash-derived home, forwards writes from non-owners, blocks writes during a re-home, and fences stale owners with an epoch-checked conditional UPDATE.
Why does last-writer-wins lose data, and when are CRDTs the answer?
Last-writer-wins compares timestamps and keeps the higher one. Suppose region A sets a cart's quantity to 3 and region B, unaware of A, sets it to 5. Replication keeps one value and discards the other, and nothing records that the losing update existed. With wall-clock timestamps, the winner is decided partly by clock skew. A node running 200 ms fast wins every race inside that window, so later writes can lose to earlier ones.
Hybrid logical clocks keep timestamps close to physical time while guaranteeing that causally related events are ordered correctly. They still impose an arbitrary order on concurrent writes. Version vectors go further: they detect that two writes are concurrent, so the system can merge them or flag them instead of silently dropping one. LWW is acceptable only when losing a concurrent write is harmless, such as presence status, a last-seen timestamp, or a cache entry that will be recomputed.
CRDTs avoid the choice of a winner by defining a merge that is commutative, associative and idempotent. Those three properties mean replicas converge no matter what order updates arrive in, how they are batched, or whether they are redelivered.
| Type | Merge rule | Cost |
|---|---|---|
| G-Counter | per-region max, then sum | one entry per region |
| PN-Counter | two G-Counters (increments, decrements) | twice the entries |
| OR-Set | add wins over concurrent remove | unique tags plus tombstones |
| LWW-Register | highest timestamp | silent loss, as described above |
Metadata grows over time. Tombstones in an OR-Set can only be garbage-collected after every replica has acknowledged them. Convergence is also not correctness. Two regions can each decrement stock from 1, and the PN-Counter will merge correctly to −1. CRDTs cannot enforce invariants of that kind.
class PNCounter:
def __init__(self, region):
self.region = region
self.p = {} # increments per region
self.n = {} # decrements per region
def incr(self, amount=1):
# Each region only ever writes its own entry.
self.p[self.region] = self.p.get(self.region, 0) + amount
def decr(self, amount=1):
self.n[self.region] = self.n.get(self.region, 0) + amount
def value(self):
return sum(self.p.values()) - sum(self.n.values())
def merge(self, other):
# Per-region max is commutative, associative and idempotent,
# so delivery order, batching and redelivery do not matter.
for mine, theirs in ((self.p, other.p), (self.n, other.n)):
for region, count in theirs.items():
mine[region] = max(mine.get(region, 0), count)
a, b = PNCounter("eu-west"), PNCounter("us-east")
a.incr(1)
b.merge(a)
a.decr(1)
b.decr(1)
a.merge(b); b.merge(a)
a.merge(b) # redelivery is harmless (idempotent)
print(a.value(), b.value()) # -1 -1: converged, but invariant violated
A PN-Counter whose per-region max merge lets two regions converge on the same value regardless of delivery order or redelivery, while still allowing stock to fall to -1.
How do you implement application-level conflict resolution and conflict-minimizing operations?
When a CRDT can't express the rule, the application has to resolve the conflict itself. Start with detection. Attach a version vector to each record, with one counter per region. If neither vector dominates the other, the writes were concurrent. Keep both versions as siblings instead of silently picking one.
Merge, then escalate
A domain merge function turns siblings into one value. For a cart, take the union of line items and the maximum quantity per SKU. Treat removals as tombstones so a deleted item doesn't come back. An order merge is harder: if one region shipped while another cancelled, no function can choose correctly. Write the conflict to a queue for a human, or start a compensating workflow such as a refund or a recall request.
Merge code is rarely exercised, because real concurrent writes are uncommon. When it is wrong, it fails quietly. The bad merged value replicates to every region and looks legitimate. Test it with deliberately generated concurrent histories, and log every merge it performs.
Design conflicts out
- Commutative deltas. Send "add 3" instead of "set to 7". Deltas compose in any order. Read-modify-write loses one of two concurrent updates.
- Global idempotency keys. Include the region in the key so retries after failover deduplicate instead of colliding.
- Region-scoped IDs. Embed a region prefix or use per-region ranges so two regions can never mint the same ID.
- Outbox plus idempotent consumers. Write side effects to an outbox in the same transaction as the data change. Consumers record processed event IDs, so emails, payments and webhooks fire exactly once even when events are replayed after replication or failover.
import logging
log = logging.getLogger("cart-merge")
def compare(a, b):
# Version vectors: {region: counter}. A missing region counts as 0.
regions = set(a) | set(b)
a_ge = all(a.get(r, 0) >= b.get(r, 0) for r in regions)
b_ge = all(b.get(r, 0) >= a.get(r, 0) for r in regions)
if a_ge and b_ge:
return "equal"
if a_ge:
return "a_newer"
if b_ge:
return "b_newer"
return "concurrent" # neither dominates: keep both as siblings
def merge_carts(siblings):
# Union of line items, max quantity per SKU. Removals are tombstones,
# so a SKU deleted in any sibling stays deleted after the merge.
# Simplification: a removed SKU can't be re-added under the same tombstone.
items, removed = {}, set()
for s in siblings:
removed |= s["removed"]
for sku, qty in s["items"].items():
items[sku] = max(items.get(sku, 0), qty)
for sku in removed:
items.pop(sku, None)
return {"items": items, "removed": removed}
def resolve(local, remote, region):
order = compare(local["vv"], remote["vv"])
if order in ("equal", "a_newer"):
return local
if order == "b_newer":
return remote
merged = merge_carts([local["cart"], remote["cart"]])
vv = {r: max(local["vv"].get(r, 0), remote["vv"].get(r, 0))
for r in set(local["vv"]) | set(remote["vv"])}
vv[region] = vv.get(region, 0) + 1 # the merge is a new write by this region
# Merges are rare and fail silently, so record every one for auditing.
log.warning("merged concurrent cart versions %s + %s -> %s",
local["vv"], remote["vv"], vv)
return {"vv": vv, "cart": merged}
Detecting concurrent writes with version vectors and merging cart siblings with a union, max-quantity and tombstone rule, logging each merge so the rarely run path can be audited.
What are the alternatives for enforcing uniqueness and invariants across regions, and how do you test it all?
Some invariants cannot be repaired after the fact. Two users holding the same username, or inventory sold below zero, is a correctness failure that merge logic cannot fix. You have three main options.
Escrow (reservation) splits a global limit into per-region allowances. If you have 1,000 units of stock, each of three regions might hold roughly 333 and sell against its local share with no cross-region coordination. The cost is that a region can exhaust its share and reject a sale while the other regions still hold unsold stock. The system as a whole could have accepted that write. A background rebalancer moves unused quota toward busy regions, but quota held in a partitioned region stays stranded until the partition heals.
Pre-allocated ranges apply the same idea to identifiers. Each region draws IDs or ticket numbers from its own block, so two regions never issue the same value.
A global coordinator routes the write to one consensus-backed authority. Use it when the invariant must be checked against the latest global state, as with unique usernames. You pay the cross-region round trip on every such write, and you lose availability for those writes during a partition.
| Approach | Write latency | Partition behavior | Main cost |
|---|---|---|---|
| Escrow | Local | Continues until local quota runs out | Stranded capacity, false rejections |
| Pre-allocated ranges | Local | Continues until local range runs out | Gaps in ID sequences |
| Global coordinator | Cross-region | Unavailable for affected writes | Latency, reduced availability |
Testing and observability
- Fault injection: Inject partitions and clock skew in staging, then assert that the invariants still hold after the regions reconverge.
- Replication lag: Track lag per replication link.
- Conflict rate: Track conflict-resolution counts per entity type. A sudden rise usually means routing or ownership has broken.
- Merge audits: Periodically compare merged state against the per-region operation logs. This is the check that catches writes that were silently discarded.
How can it save engineering time?
Active-active adds engineering work. The practices in this article do not remove that work, but they make it bounded and predictable instead of reactive. The time savings come from a few specific places.
- Choosing active-passive when it is enough. The largest saving is not building multi-writer merge logic you don't need. Running the checklist above before committing to an architecture avoids a recurring per-feature cost with no matching user benefit.
- Deciding conflict behavior once per entity. Recording the conflict class alongside the schema turns an open-ended consistency debate into a short design-review question. New engineers can read the classification instead of rediscovering the rules from production incidents.
- Reusing merge primitives. A small shared library of tested types, such as a PN-Counter, an OR-Set, version-vector comparison, and an epoch-fenced write helper, means feature teams compose existing pieces instead of writing new concurrency code for each service. Concurrency bugs are expensive to find, so writing that code once and testing it thoroughly pays off across every service that uses it.
- Designing conflicts out. Commutative deltas, region-scoped IDs and idempotency keys remove whole categories of conflict. A conflict that cannot happen needs no merge function, no review queue and no incident.
- Outbox and idempotent consumers. Duplicate emails, double charges and repeated webhooks after a failover are slow to investigate and often need manual cleanup. Exactly-once side effects at the consumer remove that cleanup work.
- Finding merge bugs before production. Merge paths run rarely and fail quietly, so a bug can corrupt data in every region before anyone notices. Fault-injection tests and generated concurrent histories in staging find those bugs while the fix is a code change, not a data repair.
- Faster diagnosis. Per-entity conflict rates, per-link replication lag, merge logs and periodic merge audits point an on-call engineer to the affected entity type and region directly, instead of leaving them to reconstruct timelines from scattered logs.
As an illustrative estimate, with all inputs assumed rather than measured: suppose a team ships an assumed 20 new entities or schema changes a year, and without a classification step each one triggers an assumed half-day consistency discussion plus an assumed one-day follow-up when a conflict case is missed. With a recorded classification and a shared merge library, assume the discussion shrinks to an hour and most missed cases are caught by the staging concurrency tests. Under those assumed inputs, the team recovers on the order of several engineer-weeks per year. Your own figures will differ, and the point of the exercise is to count the recurring cost explicitly before choosing the architecture.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to active active multi region microservices, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.