What is microservice reference data caching, and when is a local copy acceptable?
Reference data is the slowly changing lookup state that other records point at: country codes, currency lists, product categories, pricing tiers, plan definitions. In a microservice system one service owns each table. Every other service that needs it either asks the owner on each request or keeps a local copy. Microservice reference data caching means choosing the local copy, which is a deliberate trade. You accept some staleness so that your service keeps validating, enriching, and pricing when the owner is slow, deploying, or down.
Whether that trade is acceptable depends on the data, so classify each table before you build anything. Three properties matter:
- Change rate: how often an entry is added, edited, or retired.
- Size: whether a full copy fits comfortably in memory or in a local table.
- Correctness impact: what a stale value actually breaks.
| Dataset | Change rate | Size | Impact of a stale value | Local copy? |
|---|---|---|---|---|
| ISO country codes | Rare | Small | Cosmetic or rare rejection | Yes |
| Pricing tiers | Periodic, scheduled | Small | Wrong charge, refund, audit finding | Only with tight freshness bounds |
| Feature-gating plans | Frequent, often on demand | Small to medium | Paid feature withheld or free access granted | Case by case |
Correctness impact usually decides the question. If a stale entry costs money or breaks a contractual promise, the size and change rate of the table don't change that.
This applies to any team whose service makes decisions using data it does not own. The owning team should hear about each consumer's copy, because how it publishes changes determines whether consumers can stay correct.
Who owns reference data, and what are consumers allowed to do with it?
Every piece of reference data has exactly one writer. If the catalog service owns product categories, only the catalog service creates, changes, or retires them. That is the core of data ownership: one team decides what a value means and when it changes. Every other service consumes it.
A consumer may keep a local copy, called a projection, shaped for its own queries. It can drop columns it doesn't need, denormalize, or index differently. What it may not do is write to that copy. If a shipping service sees a wrong country code in its local table, it does not patch the row. It reports the problem to the owner and waits for the corrected value to arrive through the normal channel.
The rule exists because the projection must be rebuildable. You should be able to delete it and repopulate it entirely from the owner's data, whether from a snapshot, a replayed event stream, or a bulk export. A local edit is lost on rebuild, so it was never real. Worse, while it survives, the consumer disagrees with the owner and with every other consumer. Once a team starts treating its copy as a second source of truth, its copy and the owner's data drift apart, and no one can say which value is correct.
Teams accept these constraints for a concrete payoff. Reading locally removes the synchronous call to the owner. If the catalog service is down or slow, shipping keeps working on the last known values. The two services can now fail independently and be deployed independently.
What are the alternatives: TTL cache-aside, snapshot pulls, or event-carried state transfer?
There are three common ways to get there, and they differ mainly in when data crosses the boundary and what happens when the owner is unavailable.
TTL cache-aside: shipping calls the catalog API on a miss and keeps the result for a fixed TTL. Staleness is bounded by the TTL, but the call path still exists. A cold cache or an expired entry during an outage becomes an error.
Snapshot pull: a scheduled job fetches the whole table and replaces the local copy. Reads never leave the process, and an outage only delays the next refresh. Staleness equals the pull interval. The cost grows with table size, not with change rate.
Event-carried state transfer: the owner publishes a change event containing the full entity state, and consumers apply it to a local projection. Updates arrive within seconds, and reads have no runtime dependency on the owner. You also take on a broker, schema evolution, replay, and ordering concerns.
| Criterion | TTL cache-aside | Snapshot pull | Event-carried state transfer |
|---|---|---|---|
| Freshness | Bounded by TTL | Bounded by interval | Near real time |
| Runtime coupling | On every miss | Only at refresh | None on read path |
| Operational surface | Cache config | Scheduler, bulk endpoint | Broker, schemas, replay |
| Owner outage | Misses fail | Serves last snapshot | Serves last state |
| Build cost | Low | Low to moderate | High |
For small tables that rarely change, such as country codes or carrier lists, TTL caching is often good enough. Building an event pipeline for them is unjustified complexity that wastes engineering time.
How does event-carried state transfer work for reference data?
The flow starts inside the owning service. When it changes a row in its reference table, it writes the new state and an outbox record in the same database transaction. A relay publishes that record to a log keyed by entity ID. Without the shared transaction, a crash between the database commit and the publish would silently drop the change.
A compacted topic keeps the latest record per key, so the log is both the live stream and the snapshot. A new consumer reads from offset zero, builds its projection, and continues tailing without switching modes. If your broker doesn't compact, the owner must publish periodic snapshots and record the log position each one corresponds to. Consumers then resume from that position.
A reconciliation job sits beside the stream. On a schedule, it compares the consumer's projection against the owner's read API, or compares checksums per key range, and repairs any drift. It exists because projection bugs and mishandled poison messages happen even when delivery is correct.
Each event should carry four things:
- Full state, not a diff, so any single event is enough to rebuild the row.
- A stable entity key, such as the owner's ID and never a display code that can be renamed.
- A per-entity version that increases on every change, so consumers can discard stale or duplicate events.
- Tombstones for deletions, so a removed key disappears from every projection.
An update might look like {"key":"SKU-4471","version":18,"state":{"name":"Steel bracket","taxClass":"standard","active":true}}, and its deletion {"key":"SKU-4471","version":19,"deleted":true,"state":null}. Brokers that compact on null values discard the whole payload, including the version. In that case, carry the version in a message header so ordering checks still work.
import json
import sqlite3
conn = sqlite3.connect("projection.db")
conn.execute("""CREATE TABLE IF NOT EXISTS sku_projection (
key TEXT PRIMARY KEY, -- stable owner ID, never a display code
version INTEGER NOT NULL, -- highest per-entity version applied
deleted INTEGER NOT NULL, -- 1 = tombstoned; readers filter on deleted = 0
state TEXT -- full-state JSON, NULL when deleted
)""")
def apply_event(key: str, value: bytes | None, headers: dict[str, bytes]) -> None:
"""Apply one event idempotently. Use the same path for replay from offset zero and live tailing."""
if value is None:
# Null-value tombstone: the payload is gone, so read the version from a header.
version, deleted, state = int(headers["entity-version"]), 1, None
else:
event = json.loads(value)
version = event["version"]
deleted = 1 if event.get("deleted") else 0
# Full state replaces the row. There is no merge with the previous value.
state = None if deleted else json.dumps(event["state"])
# Write only when the incoming version is newer. Stale or duplicate events do nothing.
# A delete leaves a marker row, so a late older update cannot bring the key back.
conn.execute(
"""INSERT INTO sku_projection (key, version, deleted, state)
VALUES (?, ?, ?, ?)
ON CONFLICT(key) DO UPDATE SET
version = excluded.version,
deleted = excluded.deleted,
state = excluded.state
WHERE excluded.version > sku_projection.version""",
(key, version, deleted, state),
)
conn.commit()
apply_event("SKU-4471", b'{"key":"SKU-4471","version":18,"state":{"name":"Steel bracket","taxClass":"standard","active":true}}', {})
apply_event("SKU-4471", None, {"entity-version": b"19"}) # tombstone
apply_event("SKU-4471", b'{"key":"SKU-4471","version":18,"state":{"name":"Steel bracket","taxClass":"standard","active":true}}', {}) # redelivered, ignored
A consumer-side projection handler that applies full-state events keyed by entity ID. It discards stale or duplicate versions with a guarded upsert and handles both explicit and null-value tombstones, reading the version from a message header when the payload has been dropped.
How do you implement an idempotent, order-tolerant consumer projection?
The apply rule is a single comparison: write the incoming record only if its version is strictly greater than the stored one. A redelivered event carries the same version and changes nothing. A late event carries a lower version and is ignored. This only works if tombstones are stored as rows, with a deleted flag and their version, rather than removed. If the consumer hard-deletes the row, a delayed older upsert finds no row to compare against and brings the key back. Purge tombstones later, once the delay you are willing to tolerate has passed.
In SQL, use an upsert whose update branch includes a WHERE stored.version < incoming.version guard. The consumer loop reads a message, applies the conditional upsert, and commits the offset after the write succeeds. With that order, a crash causes redelivery, which the guard absorbs, and never a skipped message.
To bootstrap a new or rebuilt consumer, record the stream position first, then load a snapshot or replay the log, then start consuming from the recorded position. Events that overlap with the snapshot are rejected by the version check, so the overlap is harmless. Starting from a position taken after the snapshot leaves a gap.
That gap is usually silent. A missed delete leaves no error. The stale key sits in the projection until someone looks it up weeks later. Run a periodic reconciliation against a full snapshot and tombstone any local key the owner no longer has.
-- PostgreSQL. Tombstones are rows (deleted = true), not removed rows.
CREATE TABLE currency_projection (
code text PRIMARY KEY,
name text,
deleted boolean NOT NULL DEFAULT false,
version bigint NOT NULL, -- owner-assigned, increases per key
applied_at timestamptz NOT NULL DEFAULT now() -- local apply time, used by reconciliation
);
-- Apply one event (upsert or tombstone). Params: $1 code, $2 name, $3 deleted, $4 version.
-- A redelivery (same version) or late event (lower version) matches the conflict
-- but fails the WHERE guard, so the row is left untouched.
INSERT INTO currency_projection AS stored (code, name, deleted, version)
VALUES ($1, $2, $3, $4)
ON CONFLICT (code) DO UPDATE
SET name = EXCLUDED.name,
deleted = EXCLUDED.deleted,
version = EXCLUDED.version,
applied_at = now()
WHERE stored.version < EXCLUDED.version;
-- Consumer loop: read message -> run this statement -> commit offset only after it succeeds.
-- Reconciliation: snapshot_keys holds every key the owner returned in a full snapshot
-- requested at $1 (snapshot start time). Tombstone local keys the owner no longer has.
-- Rows applied after the snapshot started are skipped: they may be newer than the snapshot.
UPDATE currency_projection AS p
SET deleted = true, applied_at = now()
WHERE NOT p.deleted
AND p.applied_at < $1
AND NOT EXISTS (SELECT 1 FROM snapshot_keys s WHERE s.code = p.code);
-- Purge tombstones once they are older than the reordering delay you tolerate.
DELETE FROM currency_projection
WHERE deleted AND applied_at < now() - interval '7 days';
A PostgreSQL projection table where a version-guarded ON CONFLICT upsert absorbs redelivered and out-of-order events, tombstones are kept as rows, and a reconciliation pass against a full snapshot catches deletions the stream lost.
How stale is too stale, and what happens when a key is missing?
"Stale" only means something once you can put a number on it. Have the projection record the owner's version or event timestamp for the last change it applied, and expose that value as a metric. Lag is then the current time minus that timestamp, or the broker's consumer-group offset lag. One caveat: a quiet topic also produces a growing timestamp gap, so have the owner publish periodic heartbeat events if you alert on time. Set the threshold per use case, not per service. Display labels can usually be hours behind without harm. Validating an order against an active tax category may tolerate only seconds. When lag passes a use case's threshold, that use case should degrade or refuse, and the rest can carry on.
A missing key is the sharper problem. Suppose a category was created three seconds ago and the event hasn't arrived yet. You have three options:
| Option | Behaviour | Consequence |
|---|---|---|
| Reject | Return a validation error | Simple and honest. Users hit false failures right after a create. |
| Defer and retry | Park the request or message and recheck after a delay | Correct once the event lands. Needs a retry budget and a dead-letter path for keys that never appear. |
| Synchronous fallback | Call the owner and use its answer | Hides the gap. Brings back the runtime dependency you removed. |
The fallback is the risky one. During normal operation misses are rare, so the dependency looks harmless. A projection rebuild or a burst of new keys can turn misses into most of your traffic. If the owner is slow or down at that moment, your request threads block on it, and its outage becomes yours. If you keep the fallback, give it a tight timeout, a circuit breaker and a rate limit. Once the breaker opens, reject the request.
When should you avoid caching another service's reference data, and what does it cost?
Cases where a local copy is the wrong tool
Skip the copy when a decision must reflect the latest committed state. Examples include approving a credit limit, accepting an order at a price the owner just changed, or enforcing a revoked permission. Eventual consistency guarantees a window in which you validate or charge against a value the owner has already replaced, and lag monitoring only measures that window. It does not close it. Call the owner synchronously for these decisions.
Also skip it for data that is "reference" in name only. Per-customer settings and prices updated many times a day are large, fast-changing, or both. Replicating them turns every consumer into a partial, lagging copy of the owner's database.
The ongoing cost
Each consumer pays for duplicated storage, a tested rebuild procedure, lag alerting, and memory and disk sizing. Those costs multiply with replicas.
Illustrative estimate (assumed numbers, not a benchmark): footprint = rows × bytes per row × consumer replicas. With 2,000,000 rows at 500 bytes across 12 replicas, that is 2,000,000 × 500 × 12 = 12 GB before indexes.
Payloads become a contract
Once consumers project events, the event shape is public. Internal refactors in the owner now ripple outward. Keep changes additive by introducing new optional fields. Make consumers tolerant readers that ignore unknown fields and default missing ones.
To rename or remove a field, stage the change:
- Emit both the old and new fields.
- Move consumers to the new field.
- Rebuild any projections that stored the old one.
- Confirm that no consumer still reads the old field.
- Stop emitting it.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to microservice reference data caching, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.