What is microservice data duplication and why do teams use it?
Microservice data duplication means one service keeps a read-only local copy of data that another service owns. The copy is a projection: the consuming service stores only the fields it needs, shaped for its own queries, and never treats it as the source of truth. If Orders keeps a customers table with ID, name, shipping region and credit status, the Customers service still owns those records. Orders only reads them.
The usual motivation is runtime coupling. Suppose Orders calls Customers synchronously on every request to validate a buyer or render an order summary. Every order request then depends on Customers being up and fast. Its latency adds to yours, its outages become your outages, and its deploys become a risk you carry. With a local projection, Orders answers from its own database. It can also do things a remote API makes awkward, such as joining orders to customer region in a single query or filtering a report by credit status without fanning out thousands of lookups.
The price is consistency. The local copy will lag the owner, sometimes by milliseconds, sometimes longer during an incident. This is a deliberate trade of freshness for autonomy, made with the lag in mind. It is not a cache you hope stays correct. You design for staleness up front and decide which operations can tolerate it.
The pattern pays off most for teams with read-heavy, cross-service lookups on hot paths, and for teams that need to deploy and scale independently of the services whose data they read.
What are the alternatives to duplicating another service's data?
There are three common ways to give one service another service's data, and each pays for its benefits in a different way.
Shared database. Both services read the owner's tables directly. This is the cheapest option to build and gives strongly consistent reads. The cost is coupling: the schema becomes a public contract, so the owner cannot rename a column or split a table without coordinating every reader.
Request-time API call. The consumer asks the owner when it needs the data. Reads are current, and the owner's API hides its schema. The consumer's availability and latency now depend on the owner. A short-lived cache softens both problems, at the price of slightly stale data.
Owner-defined replica via events. The owner publishes changes, and the consumer maintains its own copy. Reads are local and fast, and they keep working when the owner is down. The cost is eventual consistency, plus the machinery to publish, consume, replay and reconcile events.
| Approach | Coupling | Consistency | Availability | Read latency | Ops complexity | Dev cost |
|---|---|---|---|---|---|---|
| Shared database | High (schema) | Strong | Tied to shared DB | Low | Low | Low |
| API call (+ short cache) | Medium (API contract) | Strong (bounded staleness with cache) | Tied to owner | Network round trip | Low–medium | Low |
| Event-fed replica | Low at runtime | Eventual | Independent | Local | High | High |
For low-traffic lookups, or reads that must reflect the latest committed state, such as checking a balance before a debit, a plain synchronous call is usually the right answer. A replica adds a consistency window that you then have to design around.
Data ownership rules: who is allowed to write what?
Every entity has exactly one owner: the service that accepts writes for it, enforces its invariants and emits the events that describe its changes. Everyone else holds a projection. A replica of customer data inside the shipping service is a read-only cache shaped for shipping's queries, not a second customer database. If shipping needs a customer's address corrected, it calls the customer service's API or sends it a command, then waits for the resulting event to update its copy.
The temptation to patch a replica locally is strongest when it looks harmless: fixing a typo, backfilling a missing field, adding a derived flag. Each of these breaks the replica's one guarantee, that it reflects what the owner has said. The next event from the owner will overwrite the local fix, or worse, a partial update will leave a mix of owner data and local edits that nobody can reconcile. Enrichment belongs in separate columns or tables that the consumer clearly owns, keyed by the entity ID, never mixed into the projected fields.
Ownership usually erodes gradually. A consumer adds a "temporary" write path, another team starts reading from it, and within months two services disagree about the same record with no rule for which wins.
Keep a written map of which services hold copies of which entities and fields. This matters most for personal data. Every replica of an email address or date of birth is another store to purge on deletion requests, another place access controls must be enforced, and another system in scope if credentials leak.
How does event-carried state transfer keep a replica current?
The flow has three steps. The owning service commits a change to its own database, publishes an event describing the entity's resulting state, and each consumer applies that event to its local projection. After that, the consumer answers its own queries without calling the owner, which is the reason to replicate in the first place.
| Event shape | Contains | Fits when | Cost |
|---|---|---|---|
| Full snapshot | Every field the consumer needs | Consumers need current state | Larger messages |
| Delta | Only changed fields | Large entities, small edits | Consumers must have prior state and must not miss events |
| Thin notification | ID and change type | Consumers rarely need details | Consumers call back the owner |
Thin events bring back runtime coupling. Every consumer calls the owner's API on each change, so the owner must be available and must absorb load from every consumer. The callback can also return state newer than the event, which makes ordering ambiguous.
A snapshot payload might look like {"customerId": "c-1042", "version": 17, "occurredAt": "2026-03-04T10:15:00Z", "status": "active", "shippingCountry": "DE"}. Each field has a purpose:
- The entity ID keys the upsert.
- The per-entity version lets consumers discard stale or duplicate events.
- The timestamp supports debugging but should not drive ordering, because clocks drift.
- The remaining fields are only what consumers need.
Treat the event as a public API that the producer owns. Evolve it additively by adding optional fields and never renaming or removing them, and make consumers ignore fields they don't recognise. Avoid publishing the whole table row. A "fat" event that mirrors internal columns ties every consumer to the owner's schema, so a routine migration becomes a cross-team breaking change.
-- Consumer-side projection: only the fields this service needs from the owner's event.
CREATE TABLE IF NOT EXISTS customer_projection (
customer_id text PRIMARY KEY, -- entity ID from the event, keys the upsert
version bigint NOT NULL, -- per-entity version from the producer
status text NOT NULL,
shipping_country text,
occurred_at timestamptz NOT NULL, -- kept for debugging, not used for ordering
applied_at timestamptz NOT NULL DEFAULT now()
);
-- Apply one snapshot event, e.g.
-- {"customerId": "c-1042", "version": 17, "occurredAt": "2026-03-04T10:15:00Z",
-- "status": "active", "shippingCountry": "DE"}
-- Parameters are bound by the consumer after parsing; unknown JSON fields are ignored.
INSERT INTO customer_projection
(customer_id, version, status, shipping_country, occurred_at)
VALUES
($1, $2, $3, $4, $5)
ON CONFLICT (customer_id) DO UPDATE
SET version = EXCLUDED.version,
status = EXCLUDED.status,
shipping_country = EXCLUDED.shipping_country,
occurred_at = EXCLUDED.occurred_at,
applied_at = now()
-- Version guard: a redelivered (equal version) or out-of-order (lower version)
-- event matches no row here, so it is a no-op and the newer state is kept.
WHERE customer_projection.version < EXCLUDED.version;
A PostgreSQL upsert that applies a full-snapshot event to a consumer's local projection, keyed by entity ID and using the per-entity version to discard duplicate and stale events instead of trusting the timestamp.
How do you publish events reliably without losing or inventing changes?
Even a well-shaped event is useless if it doesn't reliably match what the owner committed. The owner has to change its database and publish to the broker, and those are two systems with no shared transaction. Any ordering of the two can fail halfway. If you commit first, a crash before publishing loses the event, and the replica never learns about the change. If you publish first, a later rollback leaves consumers holding a change that never happened. Neither error corrects itself. Replicas only move when events arrive, so the drift stays until someone rebuilds them.
Transactional outbox
Write the event into the owner's own database, in the same transaction as the business change. A minimal outbox table holds:
- an event id
- the aggregate id, which doubles as the partition key
- the event type
- the payload
- a created-at timestamp
The business update and the outbox insert commit or roll back together, so an event exists if and only if the change does. A separate relay reads unpublished rows, sends them, and marks them sent. A crash between sending and marking causes a resend, so delivery is at-least-once.
Change data capture
Alternatively, tail the database's replication log. The log contains only committed work, which gives the same guarantee. Capture the outbox table rather than raw business tables, so the event contract stays independent of internal columns.
Ordering alone isn't protection
Brokers typically preserve order within a partition, so keying by aggregate id keeps one entity's events in sequence. Retries, replays and consumer restarts still deliver duplicates and redeliver older events. Consumers must be idempotent and must reject stale versions.
-- Outbox table lives in the owner's own database (PostgreSQL syntax).
CREATE TABLE outbox (
event_id uuid PRIMARY KEY, -- consumers dedupe on this
aggregate_id text NOT NULL, -- also the broker partition key
event_type text NOT NULL, -- e.g. 'CustomerAddressChanged'
payload jsonb NOT NULL, -- public contract, not internal columns
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz -- NULL until the relay has sent it
);
-- Business change and event are written in ONE transaction:
-- both commit or neither does, so no lost or phantom events.
BEGIN;
UPDATE customers
SET street = '12 Example Road', version = version + 1
WHERE id = 'cust-42';
INSERT INTO outbox (event_id, aggregate_id, event_type, payload)
VALUES (
'7f3c2a1e-5b4d-4c8e-9a10-2f6b8d0e1c55',
'cust-42',
'CustomerAddressChanged',
'{"customerId": "cust-42", "street": "12 Example Road", "version": 8}'
);
COMMIT;
-- Relay (separate process): fetch unsent rows in commit-ish order,
-- publish each keyed by aggregate_id, then mark it sent.
-- A crash after publish but before this UPDATE causes a resend (at-least-once).
SELECT event_id, aggregate_id, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100;
UPDATE outbox SET published_at = now() WHERE event_id = '7f3c2a1e-5b4d-4c8e-9a10-2f6b8d0e1c55';
A minimal PostgreSQL outbox table, the same-transaction insert alongside the business update, and the relay's poll-and-mark queries that give at-least-once delivery.
How do you implement a correct consumer? A worked example
The Orders service needs a customer's name for invoices and a shipping tier for pricing. Its projection table, customer_projection, holds only those fields plus bookkeeping: customer_id as the primary key, name, shipping_tier, version, last_updated, and a deleted flag. It does not copy email, address history or marketing preferences. Every extra column is data you must keep correct, protect and migrate, even though no code in Orders reads it.
The version-guarded upsert
The consumer runs one statement per event. It inserts the row, and on a primary-key conflict it updates the row only when the stored version is lower than the incoming one. In PostgreSQL this is INSERT ... ON CONFLICT (customer_id) DO UPDATE ... WHERE customer_projection.version < EXCLUDED.version. The database makes the comparison atomically, so concurrent consumer instances cannot race.
Walking through the events
- Create (v1): No row exists, so the insert succeeds.
- Update (v2): 1 < 2, so the row is updated.
- Late v1 redelivered: 2 is not less than 1. Zero rows change and the consumer acknowledges the message. The event was handled correctly, so this is not an error.
- Delete (v3 tombstone): The consumer sets
deleted = trueandversion = 3. It does not remove the row. If the row were removed, a late v2 would find nothing to compare against and would recreate the customer. Purge tombstoned rows later in a separate job.
Messages that fail to parse or that break schema rules go to a dead-letter queue after a bounded number of retries. Without that limit, one bad message blocks every later message for the same partition.
The consumer has a fixed shape: narrow table, guarded upsert, tombstone, dead-letter queue. Once a team has built and reviewed it, it becomes a template for the next projection. That is cheaper than writing new sync code each time and rediscovering the same ordering bugs.
-- Narrow projection: only the fields Orders reads, plus bookkeeping.
CREATE TABLE customer_projection (
customer_id uuid PRIMARY KEY,
name text, -- nullable so a tombstone can arrive first
shipping_tier text,
version bigint NOT NULL,
last_updated timestamptz NOT NULL DEFAULT now(),
deleted boolean NOT NULL DEFAULT false
);
-- Create or update event. $1=customer_id, $2=name, $3=shipping_tier, $4=version.
-- The WHERE clause makes the write idempotent: a redelivered or older
-- version touches zero rows, and the consumer still acknowledges it.
INSERT INTO customer_projection
(customer_id, name, shipping_tier, version, last_updated, deleted)
VALUES ($1, $2, $3, $4, now(), false)
ON CONFLICT (customer_id) DO UPDATE
SET name = EXCLUDED.name,
shipping_tier = EXCLUDED.shipping_tier,
version = EXCLUDED.version,
last_updated = EXCLUDED.last_updated,
deleted = false
WHERE customer_projection.version < EXCLUDED.version;
-- Delete event (tombstone). $1=customer_id, $2=version.
-- Keep the row so a late, lower version has something to lose against;
-- clear the copied fields because Orders no longer needs them.
INSERT INTO customer_projection
(customer_id, name, shipping_tier, version, last_updated, deleted)
VALUES ($1, NULL, NULL, $2, now(), true)
ON CONFLICT (customer_id) DO UPDATE
SET name = NULL, shipping_tier = NULL,
version = EXCLUDED.version,
last_updated = EXCLUDED.last_updated,
deleted = true
WHERE customer_projection.version < EXCLUDED.version;
A minimal PostgreSQL customer projection with version-guarded upserts for create/update and tombstone events, where stale or duplicate deliveries change zero rows and are safely acknowledged.
When should you avoid microservice data duplication, and how do you live with eventual consistency?
Every replica is stale. Lag is normally small, but it grows without bound while the publisher, broker or consumer is down. Classify each decision by what a stale read costs:
| Decision | Local copy acceptable? | Why |
|---|---|---|
| Display name, avatar | Yes | A wrong value is cosmetic and self-corrects |
| Shipping tier shown in UI | Yes | The owner re-checks at checkout |
| Credit limit approval | No | A stale limit lets money leave |
| Inventory reservation | No | Two services can sell the same unit |
For "No" rows, call the owner synchronously, or let the owner make the decision.
Bootstrapping and rebuilding
A new replica needs history the event stream may no longer hold, because brokers delete events past their retention window. Seed it from a snapshot export, a compacted topic that keeps the latest value per key, or a paginated backfill API, then switch to live events. Use the same path to rebuild after a consumer bug corrupts rows. If you cannot rebuild from scratch, the replica is not trustworthy.
Detecting drift
Run periodic reconciliation jobs that compare keys and versions against the owner and repair mismatches. Alert on consumer lag, dead-letter queue depth and reconciliation drift counts.
Cost versus benefit
You pay for outbox relays, consumers, DLQs, reconciliation jobs and cross-team schema negotiation. In return, you get fewer runtime dependencies and fewer incidents that cascade across team boundaries. Avoid the pattern when:
- reads need the current value
- data changes faster than consumers can apply it
- only one caller needs the data
- no team will own the replica's operations
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to microservice data duplication, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.