Active-Active Microservices: Data Conflicts and Geo Routing

What active-active microservices are and when you need them

In a multi-region active-active deployment, every region serves live traffic and accepts writes for the same logical dataset. There are two simpler alternatives. Active-passive runs one primary region and keeps a standby warm for failover. Active-active reads with single-region writes serves reads everywhere but forwards every write to one leader region. The distinction that matters is where a write is allowed to commit.

Teams adopt active-active microservices for three reasons:

• Availability: a regional outage does not stop writes.
• Latency: users far from the leader stop paying a cross-ocean round trip on every mutation.
• Capacity: write load is spread across regions instead of concentrated in one.

Most systems should not adopt it. If your availability target tolerates a few minutes of write unavailability during a promotion, active-passive or single-region writes will meet it at far lower complexity. Active-active only pays off when write latency or write availability is a hard requirement.

The thesis of this guide is that geo routing and conflict resolution are not separate concerns. Both answer one question, per entity and per operation: where may this write happen? Routing decides it up front. Conflict resolution handles the cases where routing let two regions write the same thing.

Classify your data before choosing technology

Before choosing a database or replication mode, sort every entity and operation by what happens when two regions write it concurrently. Three buckets cover most systems:

• Immutable or append-only (audit logs, order events, uploaded files): there is nothing to conflict, so any region can write.
• Commutative (view counters, tag sets, likes): concurrent operations can be merged in any order and produce the same result.
• Invariant-bearing (balances, inventory, unique usernames, account status): a merge can break a business rule, so these need a single writer or explicit coordination.

This table drives the rest of the design. It usually shows that only a small share of writes needs heavyweight coordination, which stops teams from routing everything through one expensive global path.

Design writes to avoid conflicts in the first place:

• Generate UUIDs or region-prefixed IDs, because auto-increment sequences collide across regions.
• Require an idempotency key on every mutating API so retries and replays don't apply an operation twice.
• Model changes as events or deltas such as "add 3", not as absolute values like "set 47".
• Update individual fields instead of replacing whole documents, so concurrent edits to different fields both survive.

Copyimport uuid
import psycopg  # psycopg 3

REGION = "eu-west"  # set per deployment

def new_id() -> str:
    # Region-prefixed UUID: unique across regions without a shared auto-increment sequence.
    return f"{REGION}-{uuid.uuid4()}"

def add_views(conn: psycopg.Connection, idempotency_key: str,
              product_id: str, delta: int) -> bool:
    """Apply 'add N' (a delta), never 'set N'. Returns False if this request was already applied."""
    with conn.transaction():
        # Record the client's idempotency key first. A retry or replay hits the
        # primary key on idempotency_key, inserts nothing, and is skipped.
        cur = conn.execute(
            "INSERT INTO applied_ops (idempotency_key, op_id, region, product_id, delta) "
            "VALUES (%s, %s, %s, %s, %s) "
            "ON CONFLICT (idempotency_key) DO NOTHING",
            (idempotency_key, new_id(), REGION, product_id, delta),
        )
        if cur.rowcount == 0:
            return False  # duplicate: same result as the first call, no double count

        # Field-level delta: touches only view_count, so a concurrent edit to
        # other columns of the same row is not overwritten.
        conn.execute(
            "UPDATE product_stats SET view_count = view_count + %s "
            "WHERE product_id = %s",
            (delta, product_id),
        )
    return True

if __name__ == "__main__":
    with psycopg.connect("postgresql://app@localhost/shop", autocommit=True) as conn:
        key = "req-7f3c9a"  # supplied by the client, reused on every retry
        print(add_views(conn, key, "sku-1042", 3))  # True: applied
        print(add_views(conn, key, "sku-1042", 3))  # False: retry ignored

An idempotent, delta-based update handler that records a client-supplied idempotency key, generates region-prefixed IDs, and increments a single field so retries and concurrent edits do not overwrite or double-apply changes.

How home-region ownership and geo routing work together

Homing gives each user, tenant or account one home region. All writes for that entity go to its home region. Reads are served from the nearest replica. Most write conflicts disappear because two regions never accept writes for the same entity at the same time. The conflict-handling techniques above are then needed only for data that is truly shared.

Storing the mapping

The entity-to-region mapping is a small, rarely changing table. Replicate it to every region and cache it in each gateway with a short expiry. Every request depends on it, so treat it as a critical dependency. If a gateway cannot read the mapping, it should fail closed for writes. Guessing the region risks putting writes in the wrong place, which brings back the conflicts homing was meant to remove.

Choosing a routing layer

• DNS latency or geo routing is simple but coarse. Resolvers and clients cache answers beyond the TTL. The resolver's location may not match the user's. Failover is only as fast as caches expire.
• Anycast or global load balancers send traffic to the nearest healthy region quickly. They know nothing about home regions.
• Application-level routing puts that knowledge at the edge. The gateway reads the mapping and serves reads locally. It forwards writes to the home region or redirects the client there.

Stickiness

Read-your-writes breaks when a client writes in one region and its next read lands in another region whose replica is behind. Pin each session to one region, or route the session's reads to the home region for a short window after it writes.

The trade-off is that users far from their home region pay cross-region latency on every write.

Copytype Region = "us-east" | "eu-west" | "ap-south";
type Decision =
  | { action: "serve-local" }
  | { action: "forward"; region: Region }   // proxy to home region (or 307 redirect)
  | { action: "reject"; status: 503 };

const LOCAL: Region = "eu-west";          // region this gateway runs in
const CACHE_TTL_MS = 30_000;              // mapping rarely changes, so a short cache is enough
const RYW_WINDOW_MS = 5_000;              // after a write, read from home for this long
const cache = new Map<string, { region: Region; expires: number }>();

// lookup() queries this region's replica of the entity-to-home-region table.
async function homeOf(id: string, lookup: (id: string) => Promise<Region>): Promise<Region | null> {
  const hit = cache.get(id);
  if (hit && hit.expires > Date.now()) return hit.region;
  try {
    const region = await lookup(id);
    cache.set(id, { region, expires: Date.now() + CACHE_TTL_MS });
    return region;
  } catch {
    return null;                          // mapping unavailable: caller decides how to degrade
  }
}

async function route(method: string, entityId: string, lastWriteMs: number | undefined,
                     lookup: (id: string) => Promise<Region>): Promise<Decision> {
  const home = await homeOf(entityId, lookup);
  if (method !== "GET" && method !== "HEAD") {
    // Fail closed: guessing a region for a write would bring back write conflicts.
    if (!home) return { action: "reject", status: 503 };
    return home === LOCAL ? { action: "serve-local" } : { action: "forward", region: home };
  }
  // Read-your-writes: a session that just wrote reads from home, where its write already landed.
  const recentWrite = lastWriteMs !== undefined && Date.now() - lastWriteMs < RYW_WINDOW_MS;
  if (recentWrite && home && home !== LOCAL) return { action: "forward", region: home };
  return { action: "serve-local" };       // all other reads use the local replica, which may lag
}

A sketch of edge gateway routing logic: it looks up the entity's home region from a cached, replicated mapping, forwards writes there, fails closed when the mapping is unavailable, and sends a session's reads home for a short window after it writes so read-your-writes holds.

Conflict resolution strategies and what each guarantees

Take a cart that holds {book}. Concurrently, a user in EU adds pen while a session in US removes book. Each region accepts its write locally, then replicates.

LWW compares wall-clock timestamps. A region running 200 ms fast wins every race in that window, and no error is ever logged. Version vectors keep one counter per region. If neither vector dominates the other, the writes were concurrent, and you know you must merge rather than overwrite.

CRDTs converge for counters, sets and registers, but only for the types you model. They cannot enforce invariants such as "stock ≥ 0" or "one coupon per order". Their metadata, such as tombstones and per-region tags, also grows until it is compacted.

When an invariant must hold, use synchronous cross-region coordination through consensus or a global lock. Every write then costs at least one inter-region round trip, and the side of a partition that loses quorum stops accepting writes.

Copy# Version-vector comparison plus an add-wins (observed-remove) set merge for the cart scenario.
def compare(a, b):
    regions = a.keys() | b.keys()
    a_ge = all(a.get(r, 0) >= b.get(r, 0) for r in regions)
    b_ge = all(b.get(r, 0) >= a.get(r, 0) for r in regions)
    if a_ge and b_ge: return "equal"
    if a_ge: return "a dominates"   # safe to overwrite b with a
    if b_ge: return "b dominates"
    return "concurrent"             # neither saw the other: must merge

# Each add gets a unique (region, seq) tag; a remove tombstones only the tags it observed.
def merge(x, y):
    adds = {}
    for s in (x, y):
        for item, tags in s["adds"].items():
            adds.setdefault(item, set()).update(tags)
    return {"adds": adds, "removed": x["removed"] | y["removed"]}

def value(s):
    # An item is present if any add-tag survives; an unseen concurrent add wins.
    return {i for i, tags in s["adds"].items() if tags - s["removed"]}

# Base cart {book}, added in EU with tag ("eu", 1).
eu = {"adds": {"book": {("eu", 1)}, "pen": {("eu", 2)}}, "removed": set()}  # EU adds pen
us = {"adds": {"book": {("eu", 1)}}, "removed": {("eu", 1)}}                # US removes book

print(compare({"eu": 2}, {"eu": 1, "us": 1}))  # concurrent
print(value(merge(eu, us)))                     # {'pen'}

# Tombstones in "removed" are never dropped here; real systems must compact them.

A version-vector check detects that the EU and US cart writes are concurrent, and an add-wins set merge resolves them deterministically to {pen}, leaving tombstones that grow until compacted.

Implementing read-your-writes and cross-service consistency

Even when most data tolerates asynchronous replication, users notice when their own write disappears on the next page load. There are three common ways to prevent it.

Session pinning routes every request in a session to one region. It is simple, but failover breaks the guarantee, and it does nothing for writes made from another device.

Write tokens are more precise. After a write, the service returns the commit version, such as a log sequence number or a per-key version. The client sends it back on later reads. A region whose applied version is behind the token waits briefly for replication to catch up. If replication is still behind after that bounded wait, it forwards the read to the home region. The code sketch below shows this check.

Home-region reads after a write are a coarser version of the same idea. For a fixed window after a write, reads go to the home region. The window has to be longer than the replication lag you observe in production, and that lag grows exactly when the system is stressed.

Across services, sagas and outboxes need region awareness. The outbox commits in the same local transaction as the business write. Each saga step runs in the entity's home region, so compensations do not race with a remote copy of the saga. Events from different regions can arrive out of order, and delivery is at least once. Consumers must therefore be idempotent: deduplicate on event ID and discard any event whose version is older than the state already applied. To make that possible, every event should carry its origin region, entity version and causation ID.

Copy// Read path honoring a client-supplied write token (minimum commit version).
interface Replica {
  region: string;
  appliedVersion(): Promise<bigint>; // highest version replicated locally
  read(key: string): Promise<unknown>;
}

interface HomeClient {
  read(region: string, key: string): Promise<unknown>; // forward to home region
}

const MAX_WAIT_MS = 200; // bound the wait so lag never becomes unbounded latency
const POLL_MS = 20;

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

export async function readYourWrites(
  local: Replica,
  home: HomeClient,
  homeRegion: string,
  key: string,
  writeToken?: bigint, // version returned to the client after its write
): Promise<unknown> {
  // No token, or we are the home region: the local copy is good enough.
  if (writeToken === undefined || local.region === homeRegion) {
    return local.read(key);
  }
  // Wait briefly for replication to reach the client's last write.
  const deadline = Date.now() + MAX_WAIT_MS;
  while (Date.now() < deadline) {
    if ((await local.appliedVersion()) >= writeToken) {
      return local.read(key);
    }
    await sleep(POLL_MS);
  }
  // Still behind after the bounded wait: forward instead of serving stale data.
  return home.read(homeRegion, key);
}

A lagging region compares its applied replication version with the client's write token, waits for a bounded interval, and forwards the read to the home region if it is still behind.

Failover, partitions, tenant migration and failback

When a home region goes down, each entity class needs a policy decided in advance, not during the incident. You can reject writes for entities homed there, which keeps a single writer and costs availability. Or you can let a surviving region accept writes and reconcile later. That keeps the service up, but every write accepted outside the home region becomes a backlog item. Some of those items will not merge automatically and need an operator or a business process to resolve.

Tenant migration with a fence token

1. Increment the tenant's epoch in the ownership store.
2. Freeze writes in the old region. Any write carrying a stale epoch is rejected.
3. Wait until replication lag for that tenant reaches zero.
4. Update routing to point at the new region.
5. Unfreeze writes in the new region under the new epoch.

The epoch matters because the old region may keep processing in-flight requests after routing changes. Storage-level fencing rejects them even if application checks are skipped.

Failback

Compare per-entity version vectors or origin-region tags to find writes accepted during the outage. Reconcile each class according to the table above. Hand ownership back by running the same migration procedure. Do not simply flip routing, because that skips the freeze and the lag check.

Testing, observability and the real cost of active-active

What to measure

A sudden rise in conflicts on a single-writer entity class usually means a routing bug, because two regions are writing what only one should. It rarely means a data problem.

Use fault injection to test two cases. First, isolate one region at the network layer and confirm that routing, ownership and reconciliation behave as designed. Second, skew one node's clock and confirm that no timestamp-based resolution silently picks the wrong winner.

The cost

Active-active means duplicated infrastructure in every region. It also means schema changes that must stay compatible while regions run different versions, and harder debugging, because one request can span several regions.

The largest line item is capacity headroom, because each surviving region must absorb a failed region's traffic.

Illustrative estimate (assumed numbers, not a benchmark). Assume a global peak of P = 30,000 req/s spread evenly over N = 3 regions.

• Each region must handle P / (N − 1) = 15,000 req/s.
• Total provisioned capacity is N × P / (N − 1) = 45,000 req/s, which is 50% above peak.
• With N = 2, total provisioned capacity is 100% above peak.
• Uneven traffic across regions raises these figures.

Saving engineering time

• Start with single-region writes or tenant homing.
• Apply multi-master only to the entity classes that need it.
• Keep the classification table as the design record that every new entity must be added to.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to active-active microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.