Distributed Cache Invalidation Across Microservices with Change Events

What is distributed cache invalidation, and why do naive approaches leave a stale cache?

Distributed cache invalidation is the work of keeping copies of data consistent when the service that owns the data is not the one caching it. An orders service might cache customer records owned by the customer service, and a pricing service might cache product data from the catalog. When the owner changes a row, every downstream cache has to evict or refresh its copy.

Short TTLs alone trade correctness for load. A 5-second TTL still serves stale data for up to 5 seconds, and it multiplies read traffic on the owning service. For data that changes rarely but must be correct soon after it does, such as permissions or prices, teams want an explicit signal at the moment of change.

Most stale cache bugs come from two failure modes.

The dual write. The owner commits to its database and then publishes an invalidation or deletes the cache key. These are two separate systems with no shared transaction. If the process crashes, the broker is unreachable, or the delete times out after the commit, the cache keeps the old value until its TTL expires. If there is no TTL, it keeps the old value indefinitely.

The read-repopulate race. Deleting the key does not guarantee it stays deleted:

1. Reader A misses the cache and reads version 1 from the database.
2. Writer commits version 2 and deletes the key.
3. Reader A writes version 1 into the cache.

The cache now holds stale data, and no future invalidation is coming.

Both failures share a root cause. The invalidation signal was neither guaranteed to arrive nor guaranteed to be ordered against reads. Fixing them is a delivery-guarantee problem, not a caching problem.

Is Redis Pub/Sub enough for cache invalidation?

Redis Pub/Sub is fire-and-forget. When a publisher sends a message, Redis pushes it to whichever subscribers are connected at that moment and then discards it. Nothing is written to memory or disk for later delivery, so delivery is at-most-once. A subscriber that is restarting, failing over, or briefly partitioned misses every invalidation published during that gap, and it has no way to discover what it missed. Redis can also disconnect a subscriber that reads too slowly once its output buffer exceeds the configured limit, which drops messages silently under load.

That makes Pub/Sub a reasonable fast path for evicting in-process caches, because every connected instance receives every message. It is unsafe as the only invalidation path. A missed message leaves a stale entry that persists until its TTL expires.

Consumer groups split messages across their members. To invalidate in-process caches, each service instance therefore needs its own group or must read without a group, so that every instance sees every event.

How does change-event-driven invalidation work end to end?

The reliable pipeline has five stages, and each one inherits its guarantee from the stage before it:

Service write + outbox row (one DB transaction) → outbox relay or CDC reader (database log) → durable, replayable log → consumer group per service → shared Redis tier → broadcast channel → in-process caches on every instance

The important step is the first one. The business row and the outbox row commit in the same transaction, so an event exists only if the write committed. A rollback leaves nothing to publish. With CDC, you get the same guarantee from a different source. The reader tails the database's commit log, which contains only committed changes, in commit order.

The relay publishes to a log that keeps messages after delivery. That retention is what separates it from fire-and-forget pub/sub. Each event should carry the entity key and a monotonic version, such as a commit sequence or row version. Consumers can then discard events older than what the cache already reflects.

Each owning service runs a consumer group that deletes or rewrites the affected Redis keys. The group commits its offset only after the Redis operation succeeds. If a consumer crashes or is down for an hour, it restarts from the last committed offset and replays everything it missed. Some events will be reprocessed, which is why the consumer must be idempotent. Deleting a key twice is harmless.

After Redis is updated, the consumer emits a lightweight eviction message on a broadcast channel. Every running instance drops its local copy. Losing that message is tolerable because the in-process cache is short-lived and bounded by a TTL. The authoritative state lives upstream in the log.

Copy+-------------------------------------------------------------------+
| Service: UPDATE business row + INSERT outbox row, ONE DB txn      |
+-------------------------------------------------------------------+
    | COMMIT (rollback => no outbox row, so no event)
    v
+-------------------------------------------------------------------+
| Outbox relay polls outbox  -or-  CDC reader tails DB commit log   |
+-------------------------------------------------------------------+
    | publish {entity key, monotonic version}  in commit order
    v
+-------------------------------------------------------------------+
| Durable, replayable log (events retained after delivery)          |
+-------------------------------------------------------------------+
    | resume from last committed offset  |
    v                                   v
+-------------------------------+   +-------------------------------+
| Consumer group: service A     |   | Consumer group: service B     |
| skip if version <= cached     |   | skip if version <= cached     |
| DEL/SET Redis key (idempotent)|   | DEL/SET Redis key (idempotent)|
| commit offset after Redis OK  |   | commit offset after Redis OK  |
+-------------------------------+   +-------------------------------+
    v                                   v
+-------------------------------------------------------------------+
| Shared Redis tier                                                 |
+-------------------------------------------------------------------+
    | then: lightweight eviction msg (best effort; loss tolerable)
    v
+-- Broadcast channel: every instance drops its local copy ---------+
| [Instance 1]  [Instance 2]  ...  [Instance N]  in-proc + TTL      |
+-------------------------------------------------------------------+

Each stage inherits its guarantee from the one before: events exist only for committed writes, the retained log lets consumer groups replay from their last committed offset, and a best-effort broadcast clears TTL-bounded in-process caches.

How do you implement the outbox, event payload, and idempotent consumer?

Write the entity and the event atomically

In one database transaction, update the entity row, increment its version column, and insert an outbox row that carries the same version. Either both commit or neither does, so an event can never describe a write that rolled back.

Relay to a partitioned stream

A relay polls outbox rows that have not been published yet. It hashes each entity key to one of N streams and appends the event with XADD, then marks the row as published. If the relay crashes between those two steps, the event is appended twice. The consumer must therefore tolerate duplicates. Per-key order holds only if each partition stream is read by a single consumer at a time.

Payload

Each event carries four fields: entity_type, key, version (or a commit sequence number), and op, which is either invalidate or update. Send the full value only when it is small and most consumers need it. Otherwise send just the key, so events stay small and never carry data that has already been superseded.

Idempotent consumer

The consumer reads with XREADGROUP and applies each event through a Lua script that compares the incoming version with the cached one. It writes only if the incoming version is newer, and then calls XACK. Duplicate and late events become no-ops.

Choose set-on-event when the cached value must never fall back to the database. Choose delete-on-event when the cached value is costly to compute. Deleting creates a race: a concurrent reader can repopulate the cache with stale data right after the delete. There are three ways to close it:

• Leave a versioned tombstone instead of removing the key.
• Delete again after a short delay.
• Use lease tokens that the invalidation revokes, so a reader holding a revoked token cannot write.

Copyimport redis

# Version compare-and-set, run atomically inside Redis.
# The cache entry is a hash: v = last applied version, val = cached value.
# 'invalidate' keeps v as a tombstone, so a stale reader or late event
# carrying an older version cannot write the value back.
CAS_LUA = """
local cur = tonumber(redis.call('HGET', KEYS[1], 'v') or '0')
local inc = tonumber(ARGV[1])
if inc <= cur then return 0 end  -- duplicate or out-of-order: no-op
if ARGV[2] == 'update' then
  redis.call('HSET', KEYS[1], 'v', ARGV[1], 'val', ARGV[3])
else
  redis.call('HSET', KEYS[1], 'v', ARGV[1])
  redis.call('HDEL', KEYS[1], 'val')
end
return 1
"""

r = redis.Redis(host="cache.internal", port=6379, decode_responses=True)
apply_event = r.register_script(CAS_LUA)

# One consumer per partition stream preserves per-key order.
STREAM, GROUP, CONSUMER = "cache-inval:3", "cache-invalidator", "worker-3"

while True:
    resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: ">"}, count=100, block=5000) or []
    for _stream, entries in resp:
        for msg_id, f in entries:
            cache_key = f"{f['entity_type']}:{f['key']}"
            apply_event(keys=[cache_key],
                        args=[f["version"], f["op"], f.get("value", "")])
            # Ack only after the CAS ran; a crash here means redelivery,
            # which the version check turns into a no-op.
            r.xack(STREAM, GROUP, msg_id)

An idempotent Redis Streams consumer reads one partition with XREADGROUP, applies each event through a Lua version compare-and-set that leaves a versioned tombstone on invalidate, and calls XACK only after the script has run.

Outbox vs change data capture vs dual writes: which should you choose?

Whichever race fix you choose, the event still has to be produced reliably. There are three common ways to do it, and they trade correctness against effort.

CDC's weak spot is silence. Rename a column or split a table, and consumers may keep receiving well-formed events that no longer mean what they expect. Nothing fails loudly. The outbox avoids this because the payload is a deliberate contract, but every service that writes the data must be changed to emit it.

Payload shape matters too. Full-value events save a reload, but they bind consumers to the producer's schema. They also broadcast fields, such as PII, to every subscriber, including services that should never see them. Key-only events leak nothing and survive schema changes. The cost is that a hot key can trigger a thundering herd of simultaneous reloads, so pair them with request coalescing or jittered refetch.

Recommendation:

• Small teams with low-sensitivity data: start with dual writes plus a short TTL.
• Teams that own their write paths: adopt an outbox with key-only events.
• Platform teams that need to cover legacy writers: reserve CDC for this case, and only with schema-change review in place.

When should you avoid event-driven cache invalidation, and what does it cost?

Skip the extra machinery when a single service owns both the data and its cache. That service can delete the key in the same code path, right after the transaction commits, with no log or relay involved. Skip it when consumers tolerate minutes of staleness or the data changes a few times a day, because a TTL handles both cases. Skip it if your team cannot operate a relay or CDC connector and a durable log during off-hours. A stalled relay fails silently and leaves every downstream cache stale.

The cost covers the log cluster, the relay or connector, pruning of the outbox table, consumer-lag alerts, and a replay runbook. Each of these adds to the on-call rotation.

The approach has hard limits. It remains eventually consistent. A user who writes through service A and then reads through service B can see old data until B's consumer catches up, so you get no read-your-writes guarantee across services. When you need that guarantee, read from the owning service.

Bulk updates, migrations, and backfills can emit millions of events. Each consumer then refetches from the source, which spikes database load. To prevent this, run backfills in rate-limited batches, or emit one coarse event that bumps a namespace version instead of one event per row.

Illustrative estimate (assumed numbers, not a benchmark)

Break-even months = build hours ÷ (monthly incident hours − monthly ops hours)

• Assume a 120 h build and 4 h/month of operations.
• At 1 incident/month × 6 h, break-even is 120 ÷ (6 − 4) = 60 months. Don't build it.
• At 3 incidents/month × 8 h, break-even is 120 ÷ (24 − 4) = 6 months. Build it.

How do you bound staleness with TTLs and detect it in production?

Even when invalidation events work, keep a TTL on every entry. Events fail silently: a consumer crashes mid-batch, a deploy drops a subscription, or a bug skips a key. The TTL caps how long any of those failures can serve wrong data.

Choosing a TTL per data type

Start from how long the business can tolerate stale data, not from cache performance. A shorter TTL means more misses and more load on the backend. A longer TTL raises the hit rate but makes the worst case of a missed invalidation last longer.

In-process caches need extra care. Redis Pub/Sub has no delivery guarantee, so a subscriber that disconnects misses every message published while it was gone. Give these caches short TTLs, or flush and resync them whenever the subscriber reconnects.

Signals that make staleness visible

• Consumer lag: how far invalidation consumers are behind the event stream. Growing lag means growing staleness.
• Invalidation event rate: if the rate drops to zero while writes continue, the pipeline is broken.
• Hit ratio shifts: a sudden rise can mean invalidations have stopped arriving.
• Sampled verification: periodically compare random cache entries against the source of truth and count mismatches.

Without these signals, a report like "customers sometimes see old data" can take days of log reading to explain. With them, the same problem shows up as a specific alert, such as "lag exceeded threshold on the orders consumer," that an on-call engineer can act on in minutes.

Copyimport json
import time
import redis
from prometheus_client import Counter

r = redis.Redis(host="cache.internal", port=6379, decode_responses=True)

SAMPLED = Counter("cache_verify_sampled_total", "Cache entries checked", ["data_type"])
MISMATCH = Counter("cache_verify_mismatch_total", "Cache entries that differ from source", ["data_type"])

def load_product_from_db(product_id):
    # Placeholder: read the authoritative row from the primary database.
    raise NotImplementedError

def verify_sample(sample_size=100, grace_seconds=2):
    for _ in range(sample_size):
        key = r.randomkey()  # random key gives an unbiased sample of the keyspace
        if key is None:
            return  # empty cache
        if not key.startswith("product:"):
            continue
        cached = r.get(key)
        if cached is None:
            continue  # expired or evicted between calls
        product_id = key.split(":", 1)[1]
        SAMPLED.labels("product").inc()
        if json.loads(cached) == load_product_from_db(product_id):
            continue
        # A write may be in flight and its invalidation not yet applied.
        # Wait briefly and re-check so only persistent staleness is counted.
        time.sleep(grace_seconds)
        cached = r.get(key)
        if cached is not None and json.loads(cached) != load_product_from_db(product_id):
            MISMATCH.labels("product").inc()

if __name__ == "__main__":
    while True:
        verify_sample()
        time.sleep(60)  # alert on mismatch rate / sampled rate rising above zero

A periodic sampler that compares random Redis entries against the source of truth, re-checks after a short grace period to skip in-flight writes, and exports mismatch counts so silent staleness becomes an alertable metric.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to distributed cache invalidation, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.