Microservice Data Duplication: Keeping Denormalized Copies in Sync

What microservice data duplication is and when it is worth it

Microservice data duplication means a service keeps a local, read-only copy of data owned by another service, and updates that copy from the owner's events. It is a deliberate trade: you give up strong consistency (the copy is briefly stale after every change) in exchange for autonomy.

The alternative is a synchronous lookup on every read, which couples you to the owner at runtime. If Orders calls Customers to render each order, Orders' latency includes Customers' latency, and its availability is capped by Customers' availability. When Customers is down or slow, Orders degrades with it.

A local copy pays off when reads need:

• Joins, such as listing orders with customer names, without fanning out one call per row.
• Filtering or sorting on the foreign attribute, such as "orders from gold-tier customers," which no remote API can do efficiently against your tables.
• Availability during upstream outages, so reads keep working when the owner is unreachable.

Teams with high read volume on someone else's data, or strict availability targets, are the usual adopters.

Avoid it when a business rule cannot tolerate stale data. Credit limits and inventory reservations must be checked against the owner at decision time. Also skip it when reads are low-volume and a synchronous call is simpler, or when API composition at the edge already meets your needs.

Ownership rules: one writer, read-only projections

Choosing local copies only works if you are strict about who may change the data. Every piece of business data has exactly one owning service, and only that service writes it. The Customers service owns a customer's name and tier. When Orders stores those fields, its copy is a read-only projection. Orders can index it, reshape it, or drop it and rebuild it from events. It never edits it. If a support agent fixes a misspelled name from the Orders UI, the request goes through the Customers API. The corrected value then comes back to Orders as an event, like every other change.

Teams sometimes try to skip this rule and let both services write and sync with each other. Bidirectional sync breaks down in three ways:

• Conflicting writes. Two services change the same field at about the same time. Each one's write reaches the other after the fact, so neither knows which change should win. Settling it needs vector clocks or last-writer-wins rules that quietly discard data.
• Update loops. A change in A emits an event, B applies it and emits its own event, and A applies that one. Without careful origin tracking, the loop never stops.
• No authoritative answer. When the copies disagree, nothing tells you which one is correct. Auditing, debugging, and repair all depend on having one source of truth.

Single ownership has a cost that users can see: copies lag behind the source. An order placed under a customer's old name may keep showing that name. Often that is the correct behavior, because it records the name in effect when the order was placed. Decide deliberately which projected fields should show current values and which should stay as historical snapshots.

How it works: event notification vs. event-carried state transfer

A thin notification event says only that something changed: CustomerUpdated {id: 42}. Every consumer then calls the Customers API to find out what changed. That reintroduces the runtime coupling duplication was meant to remove. Orders now depends on Customers being up, and every update causes a burst of reads against the owner. Event-carried state transfer avoids this by putting the fields consumers need inside the event, so they can update their projections without calling back.

Snapshots vs. deltas

Prefer a full-entity snapshot per change over a delta like {tier: "gold"}. A consumer can apply a snapshot without knowing prior state. Missed, duplicated or reordered events then cause at most temporary staleness, not corruption, because the next snapshot overwrites everything. Deltas require gap-free, in-order delivery, which is the guarantee that is hardest to obtain.

The cost

Every field in a fat event becomes part of the producer's public contract. Removing one later means finding and migrating every consumer. Snapshots also tend to leak PII to services that have no business holding it. Each of those services then needs access controls and must take part in erasure requests. Include only fields that some consumer has a concrete need for.

Copy[Customers DB] --tx--> [outbox table] --relay/CDC--> [broker topic]
                                                       |       |
                                              [Orders proj] [Billing proj]

Copy-- Orders service: local projection of Customers data, fed by snapshot events.
-- Store only the fields Orders actually uses. No email, phone or address,
-- so this service stays out of most PII access and erasure scope.
CREATE TABLE customer_projection (
  customer_id  bigint PRIMARY KEY,
  display_name text   NOT NULL,
  tier         text   NOT NULL,
  version      bigint NOT NULL  -- owner's per-entity version, bumped on every change
);

-- Apply one CustomerChanged snapshot event.
-- $1..$4 are bound from the event payload, for example:
--   {"id": 42, "displayName": "Ada L.", "tier": "gold", "version": 17}
-- A snapshot overwrites every projected field, so the consumer does not
-- need to know any prior state.
INSERT INTO customer_projection (customer_id, display_name, tier, version)
VALUES ($1, $2, $3, $4)
ON CONFLICT (customer_id) DO UPDATE
SET display_name = EXCLUDED.display_name,
    tier         = EXCLUDED.tier,
    version      = EXCLUDED.version
-- Version guard: a redelivered event (same version) or an older event that
-- arrives late (lower version) updates nothing. A missed event is repaired
-- by the next, newer snapshot.
WHERE customer_projection.version < EXCLUDED.version;

A PostgreSQL upsert that applies full-entity snapshot events to a consumer projection. The version guard makes duplicate and reordered events harmless no-ops, and the projection stores only the fields the consumer needs.

Publishing reliably: the dual-write problem, outbox, and CDC

The Customers service has two things to do when a tier changes: commit the row and publish CustomerTierChanged. These go to two systems with no shared transaction. If the commit succeeds and the publish fails, or the process dies between them, Orders keeps the stale tier indefinitely. Publishing first is no safer. If the commit then rolls back, consumers have applied a change that never happened. Retries do not close the gap, because the crash can happen after the database write and before the retry logic runs.

Transactional outbox

The service writes the event into an outbox table in the same local transaction as the business change. The UPDATE customers ... and the INSERT INTO outbox (id, aggregate_id, event_type, payload, created_at) ... either both commit or both roll back. A separate relay reads unpublished rows, sends them to the broker, and then marks or deletes them. If the relay crashes after sending but before marking a row, it sends that row again on restart. Delivery is therefore at-least-once, which is why consumers must be idempotent.

Log-based CDC

A connector tails the database's replication log and turns committed changes into broker messages. The application does not change at all. The cost is that consumers now see your physical table layout unless you transform or curate the stream into stable events.

Copy-- One-time setup: the outbox lives in the same database as customers.
CREATE TABLE outbox (
  id            VARCHAR(36)  PRIMARY KEY,   -- event id; consumers dedupe on this
  aggregate_id  VARCHAR(36)  NOT NULL,      -- customer id; use as the broker partition key
  event_type    VARCHAR(100) NOT NULL,
  payload       TEXT         NOT NULL,      -- serialized event body, shaped by this service
  created_at    TIMESTAMP    NOT NULL,
  published_at  TIMESTAMP    NULL           -- set by the relay after the broker acks
);

-- Business write and event write share one local transaction:
-- both rows commit together or neither does.
BEGIN;

UPDATE customers
   SET tier = 'gold'
 WHERE id = 'c-1042';

INSERT INTO outbox (id, aggregate_id, event_type, payload, created_at)
VALUES (
  '7f3c9a2e-1b4d-4e8a-9c61-2d5e8f0a7b13',
  'c-1042',
  'CustomerTierChanged',
  '{"customerId":"c-1042","tier":"gold"}',
  CURRENT_TIMESTAMP
);

COMMIT;

-- Relay loop: read a batch of unpublished rows in order, publish them to the broker,
-- then mark only the rows the broker acknowledged. A crash between publish and
-- mark resends those rows, so delivery is at-least-once.
SELECT id, aggregate_id, event_type, payload
  FROM outbox
 WHERE published_at IS NULL
 ORDER BY created_at, id
 FETCH FIRST 100 ROWS ONLY;

UPDATE outbox SET published_at = CURRENT_TIMESTAMP
 WHERE id IN ('7f3c9a2e-1b4d-4e8a-9c61-2d5e8f0a7b13');

-- Periodic cleanup so the table does not grow without bound.
DELETE FROM outbox
 WHERE published_at < CURRENT_TIMESTAMP - INTERVAL '7' DAY;

A generic SQL transactional outbox: the tier update and the CustomerTierChanged event row commit atomically. A relay then publishes unpublished rows and marks them, and a cleanup job prunes old published rows.

Building idempotent, order-tolerant consumers

Whichever publishing path you choose, plan for at-least-once delivery. The relay or connector will occasionally resend an event, and a consumer that crashes before committing its offset will process the same event again. Correctness therefore lives in the consumer.

Order by version, not by clock

Have the owning service stamp every event with a per-entity version that increments on each write, such as a row version column or a log sequence number. The consumer stores that version next to the projected row and applies an update only if the incoming version is higher. A duplicate or a stale event fails that comparison and is skipped. This is last-writer-wins, where "last" means the writer's own sequence, not arrival order.

Timestamps can't do this job. Wall clocks on different hosts drift, NTP can step a clock backward, and two writes in the same millisecond tie. Any of these can let an older update overwrite a newer one without any error.

Partitioning topics by entity key gives per-entity ordering on brokers that support it. That cuts down on out-of-order events, but rebalances and retries can still reorder them, so keep the version check.

Deletes and poison messages

Treat a tombstone event, which carries the key and a final version, as a soft delete. Mark the projected row deleted and keep its version. If you hard-delete the row, a delayed update that arrives later would recreate it. Rows that depend on the deleted entity, such as orders that reference a removed customer, should keep their last known values and get a flag, not a cascade delete.

When a message keeps failing (bad schema, invalid data), retry it a bounded number of times. Then move it to a dead-letter topic with the error and its offset, raise an alert, and let the consumer continue with the next message.

Copy-- PostgreSQL projection of customers owned by another service.
CREATE TABLE customer_projection (
  customer_id  uuid PRIMARY KEY,
  name         text,
  email        text,
  version      bigint  NOT NULL,              -- writer's per-entity version
  deleted      boolean NOT NULL DEFAULT false -- soft delete, row is kept
);

-- Upsert event: apply only if newer. Duplicates and stale events match
-- the conflict but fail the WHERE, so they change nothing (0 rows).
INSERT INTO customer_projection (customer_id, name, email, version)
VALUES ($1, $2, $3, $4)
ON CONFLICT (customer_id) DO UPDATE
  SET name = EXCLUDED.name,
      email = EXCLUDED.email,
      version = EXCLUDED.version
  WHERE customer_projection.version < EXCLUDED.version
    AND NOT customer_projection.deleted;  -- a late update can't revive it

-- Tombstone event: mark deleted and keep the final version. The insert
-- branch covers a tombstone that arrives before any earlier event.
INSERT INTO customer_projection (customer_id, version, deleted)
VALUES ($1, $2, true)
ON CONFLICT (customer_id) DO UPDATE
  SET deleted = true,
      version = EXCLUDED.version
  WHERE customer_projection.version < EXCLUDED.version;

-- Dependent rows keep their last known values and get a flag.
-- Nothing is cascade-deleted.
UPDATE order_projection
   SET customer_missing = true
 WHERE customer_id = $1;

PostgreSQL upserts that use the writer's per-entity version to ignore duplicate and stale events, apply tombstones as soft deletes that a late update cannot undo, and flag dependent orders without deleting them.

Implementation walkthrough: Orders keeping a copy of customer name and tier

Customers owns the data. It publishes CustomerUpdated with customer_id, name, tier, version (a counter it increments on every write), and occurred_at. Orders stores those fields in a local customer_snapshot table along with the version it last applied.

Conditional upsert. The consumer inserts the row, or updates it only when incoming.version > stored.version. Duplicates and out-of-order deliveries then become no-ops at the database level, so you don't need application locks. A per-entity version is safer than a timestamp because clock skew between Customers instances cannot reorder it.

Deletes. CustomerDeleted carries a version too. Keep a tombstone row (deleted = true, version retained) instead of removing the row. If you hard-delete, a delayed older update would recreate the customer.

Backfill. Replaying a compacted or long-retained log gives you exactly the event stream live consumers saw. A snapshot export from Customers is faster when history is gone. In both cases, write each snapshot row with the version it was read at, and use the same conditional upsert. The backfill and live consumer can then run concurrently, and whichever holds the newer version wins.

Schema evolution. Ignore unknown fields and default missing optional ones. That covers additive changes. For breaking changes, use a new type such as CustomerUpdated.v2, and have the owner publish both types until consumers have migrated.

Reconciliation. On a schedule, compare COUNT(*) and a checksum of each row's customer_id and version across both stores. Exclude rows with updated_at inside the last few minutes so in-flight events don't show up as drift. Flag mismatches for targeted re-sync.

Copy-- PostgreSQL, Orders database: local copy of Customers-owned fields.
CREATE TABLE customer_snapshot (
  customer_id TEXT PRIMARY KEY,
  name        TEXT,
  tier        TEXT,
  version     BIGINT NOT NULL,          -- owner's per-entity counter, not a clock
  deleted     BOOLEAN NOT NULL DEFAULT FALSE,
  updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Shared by the live consumer and the backfill job. Parameters are bound per
-- event or per snapshot row. For CustomerDeleted: $2/$3 NULL and $5 = true,
-- which keeps a tombstone that still carries the version.
INSERT INTO customer_snapshot (customer_id, name, tier, version, deleted)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (customer_id) DO UPDATE
SET name       = EXCLUDED.name,
    tier       = EXCLUDED.tier,
    version    = EXCLUDED.version,
    deleted    = EXCLUDED.deleted,
    updated_at = now()
-- Duplicate, stale, or out-of-order versions update zero rows. Because of this,
-- backfill and live events can safely race: the newer version wins.
WHERE customer_snapshot.version < EXCLUDED.version;

-- Reconciliation: run the equivalent query on the Customers side, then compare
-- results per bucket. Rows touched after the cutoff are skipped as in-flight.
-- $1 = cutoff picked once by the job, e.g. job start minus 5 minutes.
-- SUM(version) is a cheap checksum. Mismatched buckets get targeted re-sync.
SELECT MOD(ABS(version), 16) AS bucket,  -- coarse split to narrow down drift
       COUNT(*)              AS row_count,
       SUM(version)          AS version_sum
FROM customer_snapshot
WHERE NOT deleted
  AND updated_at < $1
GROUP BY MOD(ABS(version), 16)
ORDER BY bucket;

A PostgreSQL version-guarded upsert that handles live CustomerUpdated events, tombstone deletes, and concurrent backfill rows, followed by a bucketed count-and-checksum reconciliation query that excludes recently updated rows.

Operating it: drift, debugging, and impact on cost and engineering time

…to a dead-letter queue for review rather than auto-fixing them, so a consumer bug doesn't get silently papered over.

Make staleness measurable. Alert on consumer lag in time (age of the oldest unprocessed event), not offset count. "Customer tier may be up to 90 seconds stale" is a contract you can state. "40,000 messages behind" is not.

Reconciliation costs the owner. Every checksum scan is a read against the Customers service. Run it against a replica or snapshot, rate-limit it, and schedule it off-peak.

Retention decides whether rebuilds are possible. A compacted topic keyed by customer_id lets you rebuild a projection from scratch. A topic with time-based retention only lets you replay recent history, so a full rebuild needs a bulk export from the owner.

Bad events spread. Once a bad event is published, every consumer has stored it. If the field is versioned, a correction event with a higher version fixes it. If not, you need a rebuild.

Illustrative estimate (assumed numbers, not a benchmark). Extra storage = rows × bytes per row × consuming services. Assume 2,000,000 customers, 100 bytes each, and 3 consuming services: 2,000,000 × 100 × 3 = 600 MB. That storage is usually cheap. The engineering time spent on lag alerts, reconciliation, and replay tooling is the cost to budget for.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice data duplication, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.