What is microservice data replication for reference data, and who owns it?

Microservice data replication for reference data means a service keeps its own read-only copy of data that another service owns. Typical examples are product catalogs, currency lists, tax categories, and customer tiers. The consuming service stores the fields it needs in its own database and updates them as the owner publishes changes. When it serves a request, it reads locally and does not call across the network.

Before choosing a technology, decide who owns each dataset. Each dataset should have exactly one system of record, and that system accepts every write. All other copies are projections: they are derived, they are read-only, and nobody edits them or writes them back. If a consumer needs to change a product's price, it calls the catalog service. It does not update its own table.

The main benefit is less runtime coupling. Without a local copy, an order service that synchronously calls the catalog, pricing, and customer services takes on all of their latency, and it fails whenever any of them is down. With a local copy, it keeps working through an upstream outage and serves reads at local-database speed. This suits teams that deploy their services independently and read shared reference data far more often than it changes.

This is different from a cache. A cache holds whatever was fetched last and relies on expiry to keep it acceptably fresh. A replica is kept current by a defined change stream, and you can check how far it lags behind the source.

The common failure is drift in who owns the data. A team starts treating its local copy as authoritative, or adds a "small" local override. Two systems then disagree about the truth, and someone has to work out which one is right.

When should you replicate reference data, and when should you avoid it?

Four questions decide whether a local copy is worth it: how large is the dataset, how often does it change, how stale can a read be, and what does a stale read cause? The last question matters most. A stale read either leads to a wrong decision or only shows an outdated display.

A tax rate table is small and changes a few times a year, so it is cheap to copy. A stale rate, however, produces a wrong invoice. Replicate it, but carry effective dates in the data and publish new rates before they take effect, so consumers already hold the change before it matters. A product description is the opposite case. A stale copy shows yesterday's wording, and nothing downstream breaks.

Eventual consistency is the price of replication. Lag is usually short, but while the broker or a consumer is down, the staleness window grows without bound. Design for the outage, not for the typical lag.

Do not replicate:

  • Very large or fast-changing datasets. You pay to copy and store far more than you read.
  • Data that needs read-your-writes consistency. A user who edits a record expects to see the change on the next page, and a replica cannot guarantee that.
  • Data used by one rare code path. A pipeline that runs constantly to serve an occasional read is not worth maintaining.
Alternative Fits when Trade-off
Synchronous call with short-TTL cache Freshness matters and the owner is reliable Your availability depends on the owner's
Shared read API Many consumers need flexible queries The owner must run and scale it
On-demand fetch Reads are rare Latency on every read

How does event-carried state transfer work?

With event-carried state transfer, the owning service publishes an event every time an entity changes, and the event carries the entity's current state. That can be the whole record or only the fields consumers have agreed they need. Each consumer upserts that state into its own store, keyed by entity id. After that, the consumer answers its own queries from local data and never calls the owner at read time.

A notification event works differently. It says only "product 42 changed," and every consumer then calls the owner to fetch the new version. That callback brings back the problems replication was supposed to remove. Consumers once again depend on the owner being up when they process the event, so the coupling has only moved from request time to event time. A single bulk update also turns into N consumers × M entities requests against the owner, all arriving at the same moment. Carrying the state in the event avoids both problems.

 Owner service
 ┌────────────┐     ┌──────────────┐     Broker topic (key = entity id)
 │  Owner DB  │───▶│ Outbox / CDC │───▶ ┌─────────────────────────┐
 └────────────┘     └──────────────┘     │ P0: id 7 v1, id 7 v2    │
                                         │ P1: id 42 v1, id 9 v1   │
                                         │ P2: id 13 v1, id 42 ... │ ✗ never
                                         └─────────────────────────┘
                                             │        │        │
                                             ▼        ▼        ▼
                                       Pricing   Search   Shipping
                                       projection projection projection

Publish with the entity id as the partition key. The broker then routes every version of an entity to the same partition, and a consumer reads them in the order they were written. Ordering stops there. Events for different entities can land in different partitions, and each consumer receives them in an unspecified interleaving. A consumer that expects product 42 to arrive before product 9 will eventually see them the other way around.

-- Pricing service's local projection of product reference data.
-- It holds only the fields Pricing agreed to consume, not the owner's full schema.
CREATE TABLE IF NOT EXISTS product_projection (
  product_id   bigint       PRIMARY KEY,   -- same id used as the broker partition key
  sku          text         NOT NULL,
  name         text         NOT NULL,
  list_price   numeric(12,2) NOT NULL,
  currency     char(3)      NOT NULL,
  version      bigint       NOT NULL,      -- owner-assigned, increases on every change
  updated_at   timestamptz  NOT NULL DEFAULT now()
);

-- Run once per event. The event carries the current state, so there is
-- no callback to the owner. Parameters come from the event payload.
INSERT INTO product_projection AS p
  (product_id, sku, name, list_price, currency, version)
VALUES
  ($1, $2, $3, $4, $5, $6)
ON CONFLICT (product_id) DO UPDATE
  SET sku        = EXCLUDED.sku,
      name       = EXCLUDED.name,
      list_price = EXCLUDED.list_price,
      currency   = EXCLUDED.currency,
      version    = EXCLUDED.version,
      updated_at = now()
  -- Per-key partitioning keeps one product's versions in order, but
  -- redeliveries and replays still happen. Ignore anything not newer
  -- so a duplicate or stale event cannot overwrite fresher state.
  WHERE p.version < EXCLUDED.version;

-- Reads are served locally; the owner does not need to be up.
SELECT list_price, currency
FROM product_projection
WHERE product_id = 42;

A consumer-side PostgreSQL projection that upserts the state carried in each event, keyed by entity id, and uses a version guard so duplicate or stale deliveries are ignored.

How do you publish events reliably with an outbox or change data capture?

Ordering only matters if the event exists at all, and the most common way to lose one is the dual write. The owning service commits a price change, then calls the broker. If the process crashes between those steps, the database holds the new price and no consumer ever hears about it. Publish first and you get the opposite failure: the event goes out, the transaction rolls back, and every replica now holds a price that never existed. Retries cannot fix this, because no atomic boundary spans both a database and a broker.

Transactional outbox

The owner writes the business row and an event row into an outbox table in the same local transaction, so either both commit or neither does. A separate relay reads unsent rows in insertion order, publishes them, and marks them sent. If the relay crashes after publishing but before marking, it publishes the event again. Delivery is therefore at-least-once, which is why consumers must be idempotent.

Change data capture

CDC tails the owner's transaction log and turns committed changes into events. Rolled-back writes never reach the log, and the application needs no extra write path.

Concern Transactional outbox Change data capture
Atomicity source Local DB transaction Commit log contains only committed changes
Extra operational parts Outbox table, relay process, cleanup of sent rows Log connector, log retention tuning
Event shape Designed explicitly by the owner Mirrors physical columns unless a public event model sits on top
Risk from internal schema refactors Low High: a renamed column can break consumers

A common middle ground is to run CDC against the outbox table itself. The log gives you reliable capture without a polling relay, and the outbox row gives you a deliberate, versioned contract instead of leaking internal tables.

-- PostgreSQL syntax; the pattern ports to any transactional database.
CREATE TABLE outbox (
  id             BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  aggregate_type TEXT        NOT NULL,          -- e.g. 'product'
  aggregate_id   TEXT        NOT NULL,          -- key consumers partition and dedupe on
  event_type     TEXT        NOT NULL,          -- public event name, not a table name
  payload        JSONB       NOT NULL,          -- shape designed by the owner
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  sent_at        TIMESTAMPTZ                    -- NULL until the relay publishes
);
CREATE INDEX outbox_unsent ON outbox (id) WHERE sent_at IS NULL;

-- Business change and event in ONE local transaction: both commit or neither does.
BEGIN;
UPDATE product_price
   SET amount_cents = 1299, currency = 'EUR', updated_at = now()
 WHERE product_id = 'sku-1042';

INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
VALUES ('product', 'sku-1042', 'ProductPriceChanged',
        '{"productId":"sku-1042","amountCents":1299,"currency":"EUR"}');
COMMIT;

-- Relay, step 1: claim a batch of unsent rows in insertion order.
-- SKIP LOCKED lets several relay instances run without claiming the same rows.
BEGIN;
SELECT id, aggregate_id, event_type, payload
  FROM outbox
 WHERE sent_at IS NULL
 ORDER BY id
 LIMIT 100
   FOR UPDATE SKIP LOCKED;
-- ...publish each row to the broker here...
-- Relay, step 2: mark rows sent. A crash before this commit causes a republish (at-least-once).
UPDATE outbox SET sent_at = now() WHERE id IN (101, 102, 103);
COMMIT;

-- Cleanup job: delete sent rows after a retention window so the table stays small.
DELETE FROM outbox WHERE sent_at < now() - INTERVAL '7 days';

A generic outbox table, a transaction that writes the price change and its event atomically, and the relay and cleanup queries that publish rows at-least-once and remove sent rows.

How do you build an idempotent, order-aware consumer?

Whichever capture path you choose, the broker on the other side almost certainly delivers at least once. Duplicates and reordering are normal behavior, not edge cases. A retry after a lost acknowledgment, a partition rebalance, or a replay from an earlier offset will all hand your consumer events it has already seen, or older events after newer ones. A consumer that writes every event it receives will quietly regress data to stale values, and nothing will alert you.

The fix is to make every write conditional. Each event carries an entity version or a monotonically increasing sequence number assigned by the owner. The consumer upserts keyed by entity id and applies the change only when the incoming version is greater than the stored one. In SQL this is an insert with an on-conflict update guarded by a version comparison, so a duplicate or late event matches zero rows and is acknowledged harmlessly. The code example below shows the pattern.

Deletes need the same discipline. Represent them as tombstones or soft-delete flags that carry a version. If you drop rows outright, a delayed update can resurrect a deleted entity because there is no stored version left to compare against. Hard purges and privacy-driven erasure are different. They must physically reach every copy, including tombstones, caches, and backups you control. Each service that replicates personal data widens your compliance surface, which is one more reason to replicate only what you need.

Design the local table for the consumer, not as a mirror of the owner. Keep only the fields the consumer reads, plus the id and version, and put the table in the consumer's own schema. That keeps the copy small, limits exposure, and lets the owner change unrelated fields without breaking you.

-- Consumer-owned copy: only the fields this service reads, plus id and version.
CREATE SCHEMA IF NOT EXISTS billing;

CREATE TABLE IF NOT EXISTS billing.customer_ref (
  customer_id  UUID        PRIMARY KEY,
  display_name TEXT        NOT NULL,
  country_code CHAR(2)     NOT NULL,
  is_deleted   BOOLEAN     NOT NULL DEFAULT FALSE,  -- tombstone, keeps the version
  version      BIGINT      NOT NULL,                -- assigned by the owning service
  updated_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Apply an upsert or delete event. Duplicates and stale events update zero rows.
INSERT INTO billing.customer_ref AS t
  (customer_id, display_name, country_code, is_deleted, version)
VALUES
  ('3f1c9a2e-7b4d-4e1a-9c2f-5d8e6a1b0c42', 'Acme Ltd', 'GB', FALSE, 17)
ON CONFLICT (customer_id) DO UPDATE
SET display_name = EXCLUDED.display_name,
    country_code = EXCLUDED.country_code,
    is_deleted   = EXCLUDED.is_deleted,
    version      = EXCLUDED.version,
    updated_at   = now()
WHERE t.version < EXCLUDED.version;  -- only newer versions win

-- Hard purge (e.g. erasure request): remove the row and its tombstone entirely.
-- Run only on an explicit purge event; caches and backups need their own handling.
DELETE FROM billing.customer_ref
WHERE customer_id = '3f1c9a2e-7b4d-4e1a-9c2f-5d8e6a1b0c42';

A PostgreSQL-style conditional upsert into a consumer-owned table, where stale or duplicate events change nothing, deletes are versioned tombstones, and hard purges are explicit.

How do you bootstrap a new consumer and evolve the event contract?

A new consumer, a schema change on the consumer side, or a corrupted copy all need the same rebuild path: load a full snapshot, then switch to incremental events. There are three common snapshot sources:

Source How it works Main cost
Compacted topic Read from the earliest offset; the broker keeps only the latest record per key Broker storage, plus tombstone handling for deletes
Bulk export endpoint Owner serves a paginated dump of current state Load on the owner's database and pagination consistency
Log replay Re-consume the full event history Retention requirements and long replay times

The overlap rule is simple. Record the stream position before the snapshot starts, load the snapshot, then consume from that recorded position, not from the point where the snapshot finished. Some events will describe changes already in the snapshot. The version check from the previous section discards them, because their version is not newer than the stored row. Starting late can lose updates. Starting early only costs duplicate work, which the version check absorbs.

Rebuilds of large datasets are slow and expensive. Plan for them before you need one. Every additional consumer copy also adds storage and more fan-out traffic from the broker.

Evolve the contract additively. You can add optional fields, but never rename, remove, or retype existing ones. Consumers should ignore fields they don't recognise. The owner then maps its internal model onto a stable public event and can refactor internally without notifying anyone. A breaking change forces every consumer team to deploy in lockstep with the owner, which is the coupling replication was supposed to remove. If a break is unavoidable, publish a new event type alongside the old one and retire the old type only after all consumers have moved.

How do you detect drift, and what does replication cost in engineering time?

Once replicas are live, the risk is silent divergence: a consumer that stopped applying events still serves reads, just stale ones. Export two metrics per consumer. Consumer lag shows how far behind the stream it is, and last-applied version (or timestamp) per entity type shows what it actually holds. Lag can read zero on an idle stream while the replica is still wrong, so you need both.

Lag catches stalls but not lost or misapplied events. For those, run a periodic reconciliation job. It compares row counts or checksums per key range against the owner and re-fetches only the ranges that differ. Throttle it with small ranges, a rate limit, and a read replica where one exists, so the owner never sees it as a load spike. Alert when staleness exceeds the tolerance you set during classification. Do not alert on a generic threshold.

Illustrative estimate (assumed numbers, not a benchmark)

  • Upfront cost = outbox/CDC + consumer + rebuild tooling + reconciliation + contract governance. With assumed values of 3 + 2 + 2 + 1.5 + 1, that is 9.5 engineer-weeks.
  • Weekly savings = (incidents avoided × hours each) + (deploy-coordination hours removed). With assumed values of (0.5 × 16) + 6, that is 14 hours, or about 0.35 engineer-weeks.
  • Break-even = 9.5 / 0.35, or about 27 weeks.

If your savings come out far lower, a synchronous call with a cache may be cheaper.

Readiness checklist

  • Owner and staleness tolerance documented
  • Outbox or CDC in production
  • Idempotent, version-checked consumer
  • Rebuild tested from empty
  • Lag, last-applied version and reconciliation alerts wired up
import hashlib
import time

RANGE_SIZE = 1000          # small ranges keep each owner query cheap
MAX_RANGES_PER_SEC = 5     # rate limit so the owner never sees a load spike

def range_checksum(conn, lo, hi):
    # Hash (id, version) in id order so owner and replica produce identical digests
    cur = conn.cursor()
    cur.execute(
        "SELECT id, version FROM product_ref WHERE id >= %s AND id < %s ORDER BY id",
        (lo, hi),
    )
    h, count = hashlib.sha256(), 0
    for row_id, version in cur:
        h.update(f"{row_id}:{version}\n".encode())
        count += 1
    return count, h.hexdigest()

def reconcile(owner_replica_conn, local_conn, max_id, refetch_range, report_mismatch):
    # owner_replica_conn: a read replica of the owner, not its primary
    mismatched = 0
    for lo in range(0, max_id + 1, RANGE_SIZE):
        hi = lo + RANGE_SIZE
        started = time.monotonic()
        if range_checksum(owner_replica_conn, lo, hi) != range_checksum(local_conn, lo, hi):
            # Repair only the differing range instead of a full rebuild
            refetch_range(lo, hi)
            mismatched += 1
        # Sleep off the remainder of this range's time budget
        elapsed = time.monotonic() - started
        time.sleep(max(0.0, 1.0 / MAX_RANGES_PER_SEC - elapsed))
    # Export as a metric; alert if it stays non-zero across runs
    report_mismatch(mismatched)
    return mismatched

A throttled reconciliation job that compares per-range counts and checksums of (id, version) against the owner's read replica, re-fetches only ranges that differ, and reports the mismatch count as a metric.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice data replication, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.