Shared Reference Data in Microservices: Replicate or Query on Demand

What shared reference data is and who owns it

Shared reference data is the small set of values that many services read and few people ever change: ISO currency codes, country and region codes, units of measure, product categories and plan tiers. Three properties define it. It changes rarely, often a few times a year. It is read by almost every service. One team governs it, because a wrong value, such as a misspelled region or a retired plan tier, spreads into every place that joins on it.

Those properties set it apart from the two kinds of data it is most often confused with:

Each kind needs a different pattern. Transactional data stays behind its owner's API because it changes too fast to copy safely. Configuration ships with the service that uses it. Reference data sits between the two. It is small and stable enough to copy, but it is shared widely enough that copying creates a governance problem.

The ownership rule follows from that. Exactly one service is the system of record and the only writer. Any replica is a read-only projection, not a co-owned copy. Consumers must not edit it in place or add local rows to it. If a consumer needs an extra category, it asks the owner for one.

This matters to backend and platform engineers whose services join against, filter on or display lookup values that another team owns.

How query on demand and event-based replication work

Both options leave the owner as the only writer. They differ in when the consumer pays for the data: at read time, or ahead of time.

Copy                 +---------------------------+
                 |  Owner service (writer)   |
                 |  reference tables         |
                 +-------------+-------------+
                   sync API    |    events (snapshot / delta)
          +--------------------+---------------------+
          v                                          v
+-------------------+                    +-------------------------+
| Consumer A        |                    | Consumer B              |
| client cache, TTL |                    | local lookup table      |
| miss -> call owner|                    | read-only projection    |
+-------------------+                    | upsert by key + version |
                                         +-------------------------+

Query on demand

The consumer calls the owner's API when it needs a value and keeps the result in a client-side cache with a TTL. The TTL sets how stale a value can be, and it bounds how often the consumer calls the owner. Values stay close to current, and the consumer stores nothing durable. In exchange, each cache miss adds a network hop and makes the consumer depend on the owner being available. The consumer also cannot join reference rows against its own tables in the database.

Event-based replication

The owner publishes changes. It can send a full-state snapshot, a delta per key, or both. Each consumer writes them into a local lookup table that only it reads. This works only if a few mechanics are in place:

• Bootstrap or backfill. A new consumer has no history, so it needs a full snapshot or a replay from the start of the stream before it can trust its table.
• Idempotent upserts. Most brokers deliver at least once. Applying the same event twice must give the same row.
• Per-key ordering. Partition events by key and carry a version on each one. The consumer then discards anything older than the version it already holds.
• Replay. If the projection is lost or corrupted, the consumer rebuilds it from the stream. Nobody needs to edit it by hand.

Copy                  +--------------------------------------+
                  |      Owner service (sole writer)     |
                  |      reference tables, versioned     |
                  +--------------------------------------+
                       ^  |                  |
      request on miss  |  | response         |  publish snapshots
                       |  v                  |  and per-key deltas
+------------------------------+             v
| Consumer A                   |  +-------------------------+
| client-side cache, TTL       |  | Event stream / log      |
| hit: serve cached value      |  | partitioned by key      |
| miss: call owner, fill cache |  | ordered per key         |
| TTL bounds staleness and     |  | retained for replay     |
| call rate to owner           |  +----------+--------------+
| no durable copy, no joins    |             |  consume, upsert
+------------------------------+             v
   QUERY ON DEMAND                +------------------------------------------+
                                  | Consumer B                               |
                                  | read-only projection (local lookup table)|
                                  | bootstrap: snapshot or replay from start |
                                  | upsert by key, skip if version <= stored |
                                  | (idempotent: redelivery is harmless)     |
                                  | joins locally against own tables         |
                                  +------------------------------------------+
                                    EVENT-BASED REPLICATION

The owner is the only writer: Consumer A reads through a TTL cache and calls the owner synchronously on a miss, while Consumer B bootstraps from a snapshot or replay and then keeps a read-only local projection current with idempotent, versioned upserts from a per-key-ordered event stream.

Implementing each option without breaking the ownership boundary

Query on demand

Give every lookup a timeout shorter than your own request budget, so a slow owner degrades your response instead of consuming it. Decide the fallback in advance. Serve a stale cached value, fail the request, or proceed without the field. Each is valid for different data, but the decision belongs in code, not in an incident review. Put a bounded cache in front of the client, with a size limit and a TTL. An unbounded cache is effectively a replica with no invalidation. Most importantly, collect codes before you call. If you are rendering 200 order lines, gather the distinct currency or country codes and make one batch request. Calling per line turns a loop into 200 network round trips.

Replication

Apply events with an idempotent upsert keyed on the reference code. The write should take effect only when the incoming version is greater than the stored one. Duplicates and out-of-order deliveries then become no-ops rather than regressions. Keep the projection table read-only and limited to what the owner publishes. Once you add a consumer-owned column, a replay can wipe it, and you have quietly taken on a share of ownership.

Evolving the contract

Treat the code or ID as the API and the display label as presentation that can change at any time. Never reuse an identifier, because old records still point to it. Retire values with a deprecation flag instead of deleting them, so historical references still resolve. Make schema changes additive only, so consumers that have not been updated can ignore fields they don't recognize.

Copy-- PostgreSQL. Read-only projection of the currency reference data.
-- Holds only columns the owner publishes; no consumer-owned columns.
CREATE TABLE ref_currency (
  code          text PRIMARY KEY,           -- stable ID, never reused
  display_name  text     NOT NULL,          -- presentation only, may change
  minor_units   smallint NOT NULL,
  deprecated    boolean  NOT NULL DEFAULT false, -- retire values, never delete
  version       bigint   NOT NULL           -- owner-assigned, increases per change
);

-- Only the event projector writes; application code can only read.
REVOKE ALL ON ref_currency FROM orders_app;
GRANT SELECT ON ref_currency TO orders_app;
GRANT SELECT, INSERT, UPDATE ON ref_currency TO currency_projector;

-- Apply one CurrencyChanged event. Parameters come from the event payload.
-- Duplicates and out-of-order events match the conflict but fail the
-- version check, so they update nothing.
INSERT INTO ref_currency (code, display_name, minor_units, deprecated, version)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (code) DO UPDATE
  SET display_name = EXCLUDED.display_name,
      minor_units  = EXCLUDED.minor_units,
      deprecated   = EXCLUDED.deprecated,
      version      = EXCLUDED.version
  WHERE ref_currency.version < EXCLUDED.version;

A PostgreSQL read-only projection table for currency reference data, with an idempotent upsert keyed on the code that applies an event only when its version is newer than the stored one, and that marks retired codes with a deprecation flag instead of deleting them.

Snapshotting reference values into business records

Deprecation keeps old codes resolvable, but it does not keep their meaning stable. A currency code still resolves after its rounding rule changes, and a tax category still resolves after its rate changes. If an invoice line stores only tax_category = 'STD' and looks up the rate when it is read, an invoice issued last year will show this year's rate. Its totals will no longer match what was charged.

The fix is to copy the values the business record depended on into the record at write time. Store the code for identity and joins, plus the values that were in effect when the record was created:

Snapshot a value when the record states a fact that was decided at a point in time: prices, tax, rounding, and anything shown on a legal or financial document. Resolve the value at read time when the record should reflect current truth, such as display labels, translated names, or a region's current support contact.

The trade-off is that corrections do not propagate. If the owning service published a wrong rate for two weeks, fixing the rate in the reference data leaves every record written during those two weeks unchanged. Repairing them takes an explicit migration: find the affected rows by code and time window, recompute them, and record why they changed. Often the business rules require issuing compensating documents rather than rewriting the original records.

Copy-- Business record: keeps the code for identity plus the values in effect at write time
CREATE TABLE invoice_line (
  invoice_line_id      BIGINT PRIMARY KEY,
  invoice_id           BIGINT NOT NULL,
  net_amount           NUMERIC(18,4) NOT NULL,
  tax_category_code    TEXT NOT NULL,          -- identity, joins, reporting
  tax_rate_applied     NUMERIC(6,4) NOT NULL,  -- snapshot: rate used for this line
  currency_code        CHAR(3) NOT NULL,       -- identity
  currency_minor_units SMALLINT NOT NULL,      -- snapshot: precision used to round
  tax_amount           NUMERIC(18,4) NOT NULL, -- computed once from the snapshot
  created_at           TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- At write time, copy values from the local reference replicas into the row.
-- Later changes to ref_tax_category or ref_currency do not alter this line.
INSERT INTO invoice_line (
  invoice_line_id, invoice_id, net_amount,
  tax_category_code, tax_rate_applied,
  currency_code, currency_minor_units, tax_amount
)
SELECT
  1001, 501, 12000,
  t.code, t.rate,
  c.code, c.minor_units,
  ROUND(12000 * t.rate, c.minor_units)
FROM ref_tax_category t
JOIN ref_currency c ON c.code = 'JPY'
WHERE t.code = 'STD';

-- Reading back uses the stored values, not a fresh lookup,
-- so last year's invoice still shows the rate that was charged.
SELECT invoice_line_id, tax_category_code, tax_rate_applied, tax_amount
FROM invoice_line
WHERE invoice_id = 501;

A PostgreSQL invoice_line table that stores the tax category and currency codes next to the rate and precision captured at write time, with an insert that copies those values from local reference tables.

Choosing per consumer: a decision framework

That decision belongs to each consumer, not to the platform. A billing service and a search indexer can read the same currency table and still need different strategies. A single "always replicate" or "always call the API" rule fits one of them badly. Ask four questions per consumer:

1. How stale can the data be? Seconds of lag may be harmless for a dropdown and wrong for a credit check.
2. How large is the dataset, and how often does it change? Fifty country codes and a million-row product catalog cost very different amounts to copy.
3. Does the consumer join or filter on it locally? You cannot efficiently filter orders by region in SQL if region lives behind an HTTP call.
4. What happens if the owner is down? Some consumers can degrade. Others stop taking orders.

These options combine. A consumer can replicate a lookup table for filtering and also snapshot the values onto each invoice it issues.

Alternatives and failure modes of each approach

There is a third option for data that almost never changes, such as ISO-style country or currency code lists: ship it with the deployable as a read-only library or static file bundle. Lookups become in-process and have no runtime dependency. This works when updates are rare, a stale copy is harmless for weeks, and the publisher of the list is an external standards body rather than one of your services.

The cost shows up at update time. Every change to the data becomes a coordinated redeploy of every consumer. Teams that skip a release stay on an older version, so two services can disagree about whether a code is valid. That disagreement surfaces as a rejected message or a failed join, not as an obvious error. If a list changes often enough that you are planning redeploys around it, it no longer belongs in a bundle.

Making replication observable

A replica that is never queried for its freshness is assumed fresh, and that assumption fails silently. If a consumer's subscription is dropped, a deserialization error is swallowed, or a deployment leaves the handler disabled, the service keeps answering from its projection. Nothing errors, so nothing pages anyone. The consumer can act on a withdrawn tax rate or a retired product code for days, and the problem usually surfaces as a customer complaint rather than an alert.

Track lag per consumer

Each consumer should record the timestamp or sequence number of the last event it applied, alongside the projection itself. Export two gauges per consumer: the time since the last applied event, and the gap between the owner's latest published position and the consumer's applied position. Measuring only one is not enough. Reference data changes rarely, so "no events applied in six hours" can be normal. "Six hours behind the owner's head" is not.

Reconcile against the owner

Lag metrics only show whether events are arriving. They cannot detect events that were applied incorrectly or skipped. A scheduled reconciliation job compares the projection with the owner's data. The reconciliation query returns two values: a row count, and an order-independent checksum. The checksum is computed by hashing each row's key and relevant columns, then combining the hashes with a commutative operation such as sum or XOR. Because the combination does not depend on row order, the owner and the consumer can compute it independently and compare the results directly.

Alert on silence

Alert when a consumer's applied position stops advancing while the owner's position moves. Alert separately when a reconciliation mismatch persists across consecutive runs. Treat a mismatch as a trigger to rebuild the projection from a snapshot, not as something to patch by hand.

Copy-- Reconciliation query (PostgreSQL). Run the same query against the owner's
-- table and the consumer's projection, then compare the two result rows.
-- Both sides must serialize columns identically, or the checksums will differ
-- even when the data matches.
SELECT
  count(*) AS row_count,
  -- Hash each row's key and relevant columns, take 64 bits of the MD5,
  -- and add them up. Addition is commutative, so row order and scan order
  -- do not affect the result. sum() over bigint returns numeric, so it
  -- cannot overflow.
  coalesce(sum(
    ('x' || substr(md5(
      concat_ws('|',
        rate_code,                           -- business key
        rate_bps::text,                      -- rate in basis points
        valid_from::text,
        coalesce(valid_to::text, '<null>')   -- make NULL explicit; concat_ws would skip it
      )
    ), 1, 16))::bit(64)::bigint
  ), 0) AS checksum
FROM tax_rate_projection;

-- On the owner side, replace tax_rate_projection with the source table
-- (for example, tax_rates). Apply the same filter on both sides if the
-- projection only holds a subset, such as active rows.

A PostgreSQL reconciliation query that returns a row count and an order-independent checksum over a tax-rate projection. The owner and consumer each run it independently and compare the results.

Impact on latency, cost and engineering time

Illustrative estimate (assumed numbers, not a benchmark). Assume 3 reference lookups per request, 4 ms per remote call, an 80% cache hit rate, 200 requests per second, and lookups made one after another.

• Remote calls/s = requests/s × lookups × (1 − hit rate) = 200 × 3 × 0.2 = 120
• Average added latency = lookups × (1 − hit rate) × per-call latency = 3 × 0.2 × 4 ms = 2.4 ms
• Worst case, all three miss = 3 × 4 ms = 12 ms

A local join against a replicated table makes zero network calls for the same work. The 2.4 ms average hides two costs: tail requests that miss repeatedly, and 120 calls per second that depend on the owner being up. Plug in your own numbers. Hit rate usually matters most, because it multiplies everything else.

Replication costs more at the start. That cost grows with every consumer that needs its own projection. Query on demand looks cheaper at the start, but you keep paying for it: every outage at the owner becomes your incident.

Engineering time is saved in three places:

• Choosing the pattern per consumer avoids rewriting a consumer when its latency or availability needs turn out to be different.
• Stable, versioned contracts let the owner change its internals without a coordinated migration across teams.
• Reconciliation catches drift while it affects a few rows. Without it, the drift spreads into business records and becomes a data-correction project.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to shared reference data microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.