What is microservice cache invalidation, and why are local caches harder than a shared cache?
Microservice cache invalidation is the work of making sure cached copies of data stop being served after the source of truth changes. With a shared cache, such as a single Redis cluster, there is one copy: a writer deletes the key and every reader sees the miss. A local cache breaks that model. If a service runs twelve instances, each holding an in-process map or LRU, you have twelve independent copies. There is no central place to delete from, and at any moment two instances can return different values for the same key.
Teams accept this because local caches remove a network hop entirely. A lookup in process memory typically takes microseconds rather than the sub-millisecond-to-millisecond round trip of a remote cache. It also takes read load off the database and the shared cache, which matters for hot keys such as feature configuration, pricing tables, or tenant metadata that every request touches.
The approach this guide takes is that event-driven invalidation is never exactly consistent. Events are delayed, reordered, duplicated, or missed when an instance restarts or a consumer falls behind. Treat it as a bounded-staleness system instead:
- A TTL on every entry, as a backstop when an event is lost.
- Versioned events, so an instance can reject an older update.
- Explicit decisions about which reads may be stale and for how long.
This fits read-heavy services where seconds of staleness are acceptable. It does not fit reads that must reflect a write immediately, such as balances or inventory checks during checkout.
Who should use event-driven local cache invalidation?
This pattern suits teams whose services meet most of these conditions:
- Reads far outnumber writes for the cached entities. Configuration, catalog data, permissions, tenant settings, and reference tables are typical. Each cached value is read many times between changes.
- Remote cache or database latency shows up in request latency. If a request makes several lookups to a shared cache or database, removing those hops matters. If a request makes one lookup, a shared cache is usually simpler.
- The business can state a staleness tolerance. Someone can say "a price change may take up to N seconds to appear everywhere." Without that number you cannot set TTLs or alert thresholds.
- A message broker and a way to emit events from committed writes already exist, or the team is willing to run them. That means an outbox relay or a CDC connector, plus a topic that supports fan-out to every instance.
- The team can own the operational pieces. That includes consumer lag monitoring, per-instance metrics, and the reconnect and replay behaviour described below.
It is a poor fit for small fleets where a shared cache adds negligible latency, for write-heavy entities, and for data that needs read-your-own-writes across instances. The next section covers those cases and the alternatives.
When should you avoid local caches, and what are the alternatives?
Three conditions make a local cache a poor choice.
- Write-heavy data. If entries are invalidated about as often as they are read, most cached values are evicted before anyone reuses them.
- Strict read-your-own-writes. If a client writes to one instance and reads from another, that second instance may still hold the old value until the event arrives.
- High instance counts. Every write has to reach every instance, so invalidation traffic grows with the size of the fleet.
Illustrative estimate, not a measured result. Assume 200 writes/s, 50 instances and 300-byte events. Each write is delivered 50 times, which gives 10,000 deliveries/s, or about 3 MB/s of invalidation traffic before broker overhead. If you double either the write rate or the instance count, the traffic doubles. Broadcast cost exceeds the cache savings when a given instance reads a key less often than that key is invalidated. In that case, every delivery costs work but no hit pays for it.
| Option | Consistency | Read latency | Operational cost | Failure mode |
|---|---|---|---|---|
| Local cache + events | Eventual, per instance | In-process | Broker, consumers, fan-out | Lost or delayed event leaves one instance stale |
| Shared cache | One copy, so all instances agree | One network hop | Cache cluster to run | Outage shifts full load to the database |
| Read-through to database | Strong, using the primary | Query cost | Database capacity | Database overload |
| Short-TTL-only | Stale up to the TTL | In-process on hit | Minimal | Bounded staleness, expiry-driven load spikes |
A shared cache keeps a single copy of each value, so the per-instance fan-out disappears. For balances and inventory, read from the database.
How does event-driven local cache invalidation work, end to end?
At a high level the mechanism has five parts. Each one is covered in detail in the sections that follow.
- Emit an event tied to the commit. When the owning service changes a row, an invalidation event is produced as part of that commit. You can write it to an outbox table in the same transaction, or let change data capture read it from the replication log. This ensures that committed writes always produce an event and rolled-back writes never do.
- Publish to a fan-out topic. A relay or CDC connector publishes the event to a broker topic. Every instance that caches the entity receives its own copy, rather than one instance receiving it as a work item.
- Apply the event locally. Each instance either deletes the matching key or applies the new value, but only if the event's version is newer than what it holds. This makes duplicates and reordered events harmless.
- Guard the fill path. Cache fills that started before an invalidation are rejected using versions or short-lived tombstones. Concurrent misses are coalesced so one load runs per key per instance.
- Bound everything with a TTL and measure it. Every entry expires within the business's staleness tolerance, so a lost event causes bounded staleness rather than permanent staleness. Per-instance metrics show how far each cache actually trails the source of truth.
The only durable, synchronous step is the database commit. Everything after it is asynchronous, so the system guarantees eventual invalidation within a bound, not immediate consistency.
Where should invalidation events come from: dual write, transactional outbox, or change data capture?
If the service commits a row and then publishes an invalidation as a separate step, two things can go wrong. First, the process can crash, or the broker can reject the publish, after the commit. The data has changed but no event goes out, so every instance keeps serving the old value until its TTL expires. Second, if you publish before committing, the transaction can roll back after the event is sent. Instances then evict or overwrite entries for a write that never happened. Neither failure raises an error at the point where it matters. Dual writes therefore drop invalidations silently, and the only symptom is a stale cache that someone reports days later.
A transactional outbox fixes this by writing the event into an outbox table in the same transaction as the business row. A separate relay polls or tails that table and publishes each event. Because of that indirection, delivery is at-least-once. Change data capture takes a different route: it reads the database's replication log, so every committed change produces an event and rolled-back changes produce none. The application doesn't need to change.
| Approach | Tied to commit? | Operational cost | Development cost |
|---|---|---|---|
| Dual write | No | None beyond the broker | Lowest |
| Transactional outbox | Yes | Relay process, outbox cleanup | Outbox writes in every write path |
| Change data capture | Yes | Connector, change-event schema handling, monitoring log retention so the connector never falls behind it | Low in services, higher on the platform |
The outbox lets you design the event payload yourself. With CDC, consumers depend on your table schema unless you add a transformation step.
-- PostgreSQL: the outbox table lives in the same database as the business data.
CREATE TABLE outbox (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
aggregate_id text NOT NULL, -- cache key the consumers should evict
event_type text NOT NULL,
payload jsonb NOT NULL, -- payload you design, not your table schema
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz -- NULL until the relay has sent it
);
-- Write path: the business row and the invalidation event commit or roll back together.
BEGIN;
UPDATE products SET price_cents = 1999 WHERE id = 'sku-1042';
INSERT INTO outbox (aggregate_id, event_type, payload)
VALUES ('product:sku-1042', 'product.updated',
jsonb_build_object('id', 'sku-1042', 'version', 17));
COMMIT;
-- Relay: claim a batch without blocking other relay instances.
-- SKIP LOCKED lets two relays run without picking the same rows.
BEGIN;
SELECT id, aggregate_id, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;
-- ...publish each row to the broker here, then mark it sent.
-- A crash between publish and this UPDATE resends the batch: at-least-once delivery.
UPDATE outbox SET published_at = now()
WHERE id = ANY ('{}'::uuid[]); -- replace with the ids just published
COMMIT;
-- Cleanup job: keep the outbox small so the relay query stays cheap.
DELETE FROM outbox
WHERE published_at < now() - interval '1 day';
A PostgreSQL transactional outbox in which the invalidation event commits atomically with the business row, and a relay claims, publishes and cleans up events with at-least-once delivery.
How does event-driven invalidation work across every instance?
The full path has four hops. A write commits in the owning service's database, together with an outbox row or as a change the CDC connector will read. A relay or connector publishes an invalidation event to a topic. Every running instance of every service that caches that entity consumes the event. Each instance then evicts or updates the matching key in its in-process cache. The commit is the only durable step. Everything after it is asynchronous, so the guarantee is "eventually invalidated," never "immediately."
Fan-out, not work distribution
A competing-consumer queue hands each message to exactly one consumer. That suits jobs but breaks invalidation: one pod evicts the key, and the other nineteen keep serving the old value until their TTL expires. Every instance needs its own copy of the stream. You can use a broadcast or pub/sub subscription, or a log-based topic where each instance reads under its own consumer identity. If you give each instance its own identity, delete those identities when instances die. Otherwise abandoned subscriptions accumulate and backlog builds up on the broker.
Joining mid-stream
A new, restarted or reconnected instance has a gap: events published while it wasn't listening. You have two safe options:
- Clear on (re)connect. This is simple and correct, but it empties the cache exactly when the instance is cold.
- Resume from a known position. Persist the last applied offset, or warm from a snapshot tagged with one, then replay from that point. This only works if broker retention covers the gap. Fall back to clearing when it doesn't.
Either way, subscribe before you start serving reads from the cache.
Mass clears are the risk to plan for. A rolling deploy or a broker blip can make every instance clear at once and send all of their misses to the origin together. Add jitter to reconnects, stagger restarts, and prefer resuming from a known position over clearing where retention allows.
+------------------------------+
| Owning service database |
| commit: row + outbox row | <- only durable step
+--------------+---------------+
| relay / CDC connector (async from here)
v
+------------------------------+
| Broker topic: invalidations |
+--------------+---------------+
| fan-out: each instance has its own copy
+-------------------+-------------------+
v v v
+--------------+ +--------------+ +--------------+
| Instance A | | Instance B | | Instance N |
| local cache | | local cache | | local cache |
| evict key | | evict key | | evict key |
+--------------+ +--------------+ +--------------+
Competing-consumer queue (wrong): only A evicts; B..N serve stale until TTL.
New / restarted / reconnected instance: subscribe, then before serving:
+-> offset still in retention? -- yes --> replay from saved offset
| -- no --> clear local cache (cold)
Risk: mass clears (rolling deploy, broker blip) => cache-miss storm on DB.
A commit writes an outbox row or CDC change, a relay publishes it to a topic, and every instance receives its own copy to evict its local key; joining instances replay from a saved offset or clear.
Should invalidation events delete entries or carry new values, and how do you handle ordering and duplicates?
Delete-on-event is the safer default. The handler removes the key, and the next read loads the current row from the source of truth. Because the reload reads whatever is committed now, a late or duplicated delete costs only an extra miss and cannot install wrong data. The cost appears on hot keys: every instance misses at once and reloads together, which the next section addresses.
Update-in-place events carry the new value, so instances can skip the database read. The risk is ordering. With at-least-once delivery and possible reordering, an older event can arrive after a newer one. A handler that blindly writes the payload then replaces fresh data with stale data, and that entry stays wrong until its TTL expires or another write occurs.
If you carry values, also carry a monotonic version that the source of truth assigns. Good choices are a row version column incremented in the same transaction, or the log sequence number of the commit. Store that version alongside the cached value. On each event, compare versions:
- If the event's version is greater than the cached one, apply it.
- If it is equal or lower, drop it.
- If the key is absent, either insert the value or ignore the event and let the next read load it.
This rule also makes the handler idempotent: replaying the same event a second time is a no-op.
Do not order events by wall-clock timestamps. When several writers or hosts generate them, clock skew and timestamp precision can make a later write look earlier, and the version check will then keep the wrong value.
import threading
from dataclasses import dataclass
@dataclass(frozen=True)
class Entry:
value: dict
version: int # row version or commit LSN assigned by the database, never a wall-clock time
class VersionedCache:
def __init__(self):
self._lock = threading.Lock()
self._entries: dict[str, Entry] = {}
def on_update_event(self, key: str, value: dict, version: int) -> bool:
"""Apply a value-carrying event. Safe under duplicates and reordering."""
with self._lock:
current = self._entries.get(key)
if current is None:
# Not cached here: ignore it and let the next read load the row.
# Inserting would let a late, stale event fill an empty slot.
return False
if version <= current.version:
return False # duplicate or older event: drop it (idempotent)
self._entries[key] = Entry(value, version)
return True
def on_delete_event(self, key: str) -> None:
with self._lock:
self._entries.pop(key, None) # repeat deletes are harmless
def get(self, key: str, fetch_row) -> dict:
with self._lock:
if (hit := self._entries.get(key)) is not None:
return hit.value
value, version = fetch_row(key) # returns (row, row_version) from the DB
with self._lock:
current = self._entries.get(key)
# An event may have landed while we read; keep whichever is newer.
if current is None or version > current.version:
self._entries[key] = Entry(value, version)
return self._entries[key].value
A local cache that stores a database-assigned version with each entry, applies value-carrying events only when their version is newer, and treats duplicate, older, and delete events as idempotent no-ops or misses.
How do you prevent the read-repopulate race and reload stampedes?
The race, step by step
- Instance A misses on
product:42and reads version 7 from the database. - A writer commits version 8 and publishes an invalidation.
- Instance A processes the invalidation and deletes
product:42, which is already absent. - Instance A's slow fill finishes and caches version 7.
Instance A now serves stale data until the next event or the TTL expiry. The invalidation arrived on time but had nothing to delete.
Rejecting stale fills
Version checks on fill. Read the row's version with the value. On fill, write only if the version is newer than any version the cache has recorded for that key, including versions recorded by an invalidation.
Invalidation markers. If a version is not available, replace the deleted entry with a tombstone that lives a few seconds. A fill that started before the tombstone's timestamp is discarded. Set the tombstone's lifetime longer than your slowest realistic load.
TTL as the safety net
Set the TTL from the business's staleness tolerance, such as "prices may lag by 60 seconds." Do not pick a TTL for the hit ratio you want. TTLs catch lost events and missed races. Add jitter of roughly ±10% so entries filled together do not expire together. Use single-flight coalescing on misses so that one load per key runs per instance and other callers wait on it.
Serving stale data while revalidating keeps latency flat. It extends the worst-case staleness by one reload duration past the TTL, so include that in your staleness budget.
Read-your-own-writes
The writer can invalidate its own local cache synchronously before it acknowledges the write, so it sees its own change without waiting for the event. Users routed to other instances can still read old data. For flows that need read-your-own-writes guarantees, bypass the cache and read from the source for a short window after the write, for example by flagging the session.
import random, threading, time
from concurrent.futures import Future
class LocalCache:
def __init__(self, ttl_seconds):
self._ttl = ttl_seconds # from business staleness tolerance
self._lock = threading.Lock()
self._entries = {} # key -> (version, value, expires_at)
self._seen = {} # key -> highest version known; prune in production
self._inflight = {} # key -> Future for single-flight loads
def invalidate(self, key, version):
# Record the event's version even when there is nothing to delete.
with self._lock:
self._seen[key] = max(self._seen.get(key, -1), version)
self._entries.pop(key, None)
def _fill(self, key, version, value):
with self._lock:
if version < self._seen.get(key, -1):
return # slow fill lost the race: v7 arrived after the v8 invalidation
self._seen[key] = version
jitter = random.uniform(0.9, 1.1) # +/-10% so co-filled keys expire apart
self._entries[key] = (version, value, time.monotonic() + self._ttl * jitter)
def get(self, key, load): # load(key) -> (version, value), read together
with self._lock:
entry = self._entries.get(key)
if entry and entry[2] > time.monotonic():
return entry[1]
fut = self._inflight.get(key)
leader = fut is None
if leader:
fut = self._inflight[key] = Future()
if not leader:
return fut.result() # wait on the one load already running
try:
version, value = load(key)
self._fill(key, version, value)
fut.set_result(value)
return value
except BaseException as exc:
fut.set_exception(exc)
raise
finally:
with self._lock:
self._inflight.pop(key, None)
A per-instance cache that records the version from each invalidation so a slower fill carrying an older version is discarded, adds ±10% jitter to the TTL, and coalesces concurrent misses into one load per key.
How should you design cache keys and invalidation granularity for lists and derived data?
Single-entity keys such as order:{id} are easy because the event names the key: an OrderUpdated event for order 42 maps to exactly one entry. Aggregate entries break that mapping. A cached result for "open orders for customer 7, newest first" depends on many rows. An event about order 42 does not say which lists contain it. A status change can also move an order into a list it was never part of, so tracking current members is not enough.
You have three practical options:
| Approach | Precision | Build and upkeep | Thundering-herd risk | Good fit |
|---|---|---|---|---|
| Dependency tracking | High: evicts only affected entries | High: a reverse index from entities to aggregate keys, updated whenever a query changes | Low | A few hot, expensive aggregates with stable queries |
| Generation counter per namespace | Coarse: orders:customer:7:g{n} becomes unreachable on bump |
Low: one counter per namespace and a key convention | High if the namespace is hot and broad | Most list and query caches |
| Short TTL only | None: staleness is bounded by the TTL | Minimal | Low, because expiries are spread out | Data where minutes of staleness is acceptable |
Dependency tracking is the costliest to maintain. Every new filter or sort order needs matching index logic, and a missing dependency causes silent staleness rather than an error. Generation counters avoid that code, but a bump invalidates every entry in the namespace on every instance at once. Keep namespaces narrow, scoped to one customer rather than "all orders", and put the single-flight loading from the previous section in front of the reload. Combine either approach with a TTL as a backstop for lost events.
How do you measure real staleness and debug stale cache bugs?
A TTL caps staleness on paper. Measurement tells you what users actually see. Tag every metric below with an instance ID, because fleet-wide averages hide the one pod that is serving old data.
| Metric | What it reveals | Signal to watch |
|---|---|---|
| Hit rate | Cache effectiveness | A sudden rise can mean invalidations stopped arriving |
| Events received / applied / ignored as stale | Handler health and ordering | Received without applied points to a handler bug. A high ignored count points to reordering or replays |
| Consumer lag per instance | How far each cache trails the stream | One instance falling behind while the others keep up |
| Commit-to-invalidation time | End-to-end staleness window | Stamp commit time in the event and subtract it at apply time |
These metrics describe the pipeline, not the data. To measure actual staleness, run a sampler on each instance. It picks random cached keys, reads the same keys from the source of truth, and compares versions or content hashes. Skip mismatches younger than your normal propagation time so in-flight updates don't count as errors. Record the mismatch rate and the age of each stale entry.
Per-instance visibility changes how incidents go. A report of "the price is sometimes wrong" usually traces to a single instance whose consumer crashed or lost its partition assignment. If logs record the key, the event version, and the instance, you can find that instance with a dashboard query instead of spending hours trying to reproduce the bug.
Expect lag spikes during broker outages and deploys, when consumers rebalance and new instances start cold. Alert on lag that stays above your TTL budget, not on brief spikes. During a deploy, annotate the alert rather than suppressing it, so lag that doesn't recover still pages someone. If an instance can't catch up, have it flush its cache or shorten its TTL so stale reads stay bounded.
How can event-driven invalidation save engineering time?
The time savings come from removing recurring manual work and making stale-data bugs diagnosable. They do not come from the cache itself. They appear only if the pieces above are built once and shared.
- Fewer silent stale-data bugs to chase. Tying events to the commit with an outbox or CDC removes the dual-write failure mode. In that failure mode an invalidation is dropped with no error, and the bug surfaces days later as an intermittent user report. Those reports are among the most expensive to investigate, because the evidence is gone by the time someone looks.
- No manual cache flushes or "flush on deploy" workarounds. Without reliable invalidation, teams often fall back to restarting pods, adding admin endpoints to clear caches, or shortening TTLs whenever someone complains. Event-driven invalidation with a TTL backstop makes those interventions unnecessary for normal writes.
- One shared implementation instead of per-service variants. The versioned apply logic, stale-fill rejection, single-flight loading, TTL jitter, and reconnect handling can live in one internal library. Each new service then adds a key mapping and an event subscription rather than re-solving ordering and race conditions. Bugs fixed in the library are fixed everywhere.
- Faster incident triage. Per-instance lag, applied/ignored counters, and a staleness sampler turn "the data is sometimes wrong" into a query that names the instance and the key. This replaces attempts to reproduce a timing-dependent bug locally.
- Clear decisions up front. Agreeing on staleness tolerances per entity, and choosing where the pattern does not apply, avoids later rework. Typical rework includes retrofitting read-your-own-writes onto a cache that cannot provide it.
Illustrative estimate, not a measured result. Assume an organisation with eight services that each cache data locally. Assume each service, without shared tooling, has two stale-data investigations a quarter, at an assumed one engineer-day each. That adds up to 16 engineer-days a quarter spent on investigation. If a shared library and per-instance dashboards cut each investigation to an assumed two hours, the quarterly cost falls to about 4 engineer-days. That saving has to be weighed against the assumed upfront cost of building the library, relay or CDC pipeline, and dashboards, plus ongoing maintenance. Replace every assumed input with your own incident history before using this to justify the work.
The trade-off runs the other way for a small fleet with few cached entities. There, a shared cache or short TTLs alone usually cost less engineering time than building and operating an event pipeline.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to microservice cache invalidation, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.