What is distributed cache invalidation, and why does TTL-only expiry break down?
Distributed cache invalidation is the problem of removing or refreshing cached copies of data when the source of truth changes, and those copies live in caches owned by several different services. In a monolith, the code that writes a row can delete the cache key in the same process. In a microservice system, the pricing service owns prices, but the cart, search, and recommendations services each keep their own copies. None of them see the write happen.
TTL-only expiry deals with this by accepting staleness up to the TTL. That holds for a single cache. It stops holding once caches overlap. If cart caches prices for 5 minutes and search caches them for 10, a user can see two different prices for up to 10 minutes. If search builds its index from the cart's cached view, the TTLs add up and the worst case reaches 15 minutes.
To work out what each service can tolerate, ask two questions. What does one stale read cost, and how often does the key change? Illustrative estimate: assume a popular SKU's price changes on average every 2 minutes and the cart caches it for 5 minutes. With random writes and a refresh at each TTL expiry, roughly 60% of cached reads return an outdated price. Lowering the TTL until that number is acceptable usually removes most of the cache's benefit.
The thesis of this article is that an invalidation event is an unreliable hint. TTLs, versioning, and a durable change source are what keep the system correct when an event is lost, delayed, or reordered. The article is written for teams whose services cache data owned by another service.
Who should use event-driven eviction?
Event-driven eviction fits teams that meet most of these conditions:
- Several services cache data they do not own. The writer cannot delete keys in other services' caches directly, and those services cannot see the write.
- Stale reads have a visible cost. Prices, inventory, permissions, account status, and similar values that users or downstream systems act on immediately.
- TTL tuning has run out of room. The TTL that would keep staleness acceptable is short enough to erase most of the cache's hit rate, or it causes load spikes on expensive queries.
- The owning service can emit reliable change events. It can add an outbox table, or the team can run CDC against the database's replication log.
- The source has a monotonic version per entity, such as a version column or commit timestamp. Without one, the ordering and write-back protections described below are much weaker.
- Someone will operate the pipeline. Consumer lag, pending entries, stream trimming, and reconnect behavior all need monitoring and an owner.
If you have a single service that both writes and caches the data, write-through or delete-on-write in the same process is simpler. If the data rarely changes or minutes of staleness are harmless, TTL-only expiry is usually enough. The alternatives are compared in detail near the end of the article.
How does it work? Event-driven eviction across service caches
The flow has one writer per entity. Only the owning service changes the row. Everything else holds a copy that it may discard.
Owner DB → outbox table or CDC stream → event channel → subscribers, which evict from the shared Redis cache and from each service instance's local in-process cache.
The owner commits the write, and the change source emits an invalidation event keyed by entity type and ID. Subscribers delete the matching entries; they do not overwrite them. The next read misses and loads from the source of truth. Delete is the better default for three reasons:
- A delete is idempotent.
- It cannot apply a stale value when events arrive out of order.
- Consumers don't need to know how the owner serializes the entity.
An update-in-place event carries a full copy of the data. Each consumer then has to apply it correctly, in order.
Eviction is asynchronous, so the writing service itself sees stale reads unless it updates or bypasses its own cache on the write path. The event alone does not give it read-your-writes.
In short, the mechanism has four parts, each covered in a section below: a change source tied to committed writes (outbox or CDC), a delivery channel with known guarantees (Pub/Sub or Streams), idempotent version-aware eviction in every consumer, and a guard against slow readers writing old values back. TTLs remain on every entry as the backstop when any of these fails.
The event payload
Keep the payload small:
| Field | Purpose |
|---|---|
entity_type |
Namespaces the cache key, e.g. order |
entity_id |
The key to evict |
version |
Monotonic row version or commit timestamp from the source |
The version lets a handler record the highest version it has seen per key. It can then drop duplicates and late arrivals without side effects. The example that follows shows a sample payload and an eviction handler.
Treat this schema as a public contract. Add fields only in backward-compatible ways, and never repurpose existing fields. A consumer that misreads the key silently stops evicting.
import json
import redis
SAMPLE_EVENT = '{"entity_type": "order", "entity_id": "ord_8f3a21", "version": 1042}'
shared = redis.Redis(host="cache.internal", port=6379)
local_cache = {} # this instance's in-process cache: key -> value
seen_versions = {} # highest version evicted per key (bound this in production, e.g. LRU)
def cache_key(entity_type, entity_id):
# Must match the key format every reader uses, or eviction silently misses.
return f"{entity_type}:{entity_id}"
def handle_invalidation(raw):
event = json.loads(raw)
# Read only the fields we know; ignore any new ones (backward compatibility).
key = cache_key(event["entity_type"], event["entity_id"])
version = int(event["version"])
# Duplicate or late event: a version at least this new was already handled.
if version <= seen_versions.get(key, -1):
return
# Delete, never overwrite: the next read misses and reloads from the owner DB.
local_cache.pop(key, None)
shared.delete(key) # idempotent, so it is safe for every instance to run this
# Record the version only after eviction succeeds, so a failed delete is retried.
seen_versions[key] = version
handle_invalidation(SAMPLE_EVENT)
handle_invalidation(SAMPLE_EVENT) # redelivery is dropped without side effects
This example shows a sample invalidation payload and a subscriber handler that uses the version to drop duplicate and out-of-order events, then deletes the key from both the local in-process cache and shared Redis.
Is Redis Pub/Sub reliable enough for cache invalidation?
Redis Pub/Sub delivers at most once. PUBLISH invalidations '{"key":"product:42"}' pushes the message to whichever clients are subscribed at that moment and returns the number of clients that received it. Nothing is stored. A subscriber that is restarting, partitioned, or dropped for exceeding its output buffer limit never sees the message. When it reconnects with SUBSCRIBE invalidations, it resumes listening with no record of what it missed. Its local cache keeps serving the old value until something else removes it.
Redis Streams with consumer groups behave like a durable log. The producer appends with XADD invalidations * key product:42. Each service reads through its own group with XREADGROUP GROUP pricing-svc worker-1 COUNT 100 STREAMS invalidations >. Redis tracks delivered-but-unacknowledged entries in a pending list until the consumer calls XACK. After a crash, the consumer re-reads its pending entries and then continues from the group's position.
| Property | Pub/Sub | Streams + consumer groups |
|---|---|---|
| Delivery | At most once | At least once |
| Offline subscriber | Messages lost | Resumes from stored position |
| Replay | None | Yes, until trimmed |
| Operational cost | Minimal | Trimming, pending-entry recovery, lag monitoring |
Durability has costs. You must cap stream length or memory grows. You must reclaim entries left pending by dead consumers. You must alert on consumer lag, because a backlog during a write spike can delay evictions longer than a TTL would have.
Pub/Sub is acceptable when a short TTL is the real correctness bound and events only shorten typical staleness. Under that design, a lost message costs at most one TTL of stale reads. If that window is acceptable for the data, the extra components a stream requires are not worth running.
redis-cli SUBSCRIBE invalidations
redis-cli PUBLISH invalidations '{"key":"product:42"}'
redis-cli XGROUP CREATE invalidations pricing-svc '$' MKSTREAM
redis-cli XADD invalidations MAXLEN '~' 100000 '*' key product:42
redis-cli XREADGROUP GROUP pricing-svc worker-1 COUNT 100 STREAMS invalidations 0
redis-cli XREADGROUP GROUP pricing-svc worker-1 COUNT 100 BLOCK 5000 STREAMS invalidations '>'
redis-cli XACK invalidations pricing-svc 1700000000000-0
redis-cli XPENDING invalidations pricing-svc
redis-cli XAUTOCLAIM invalidations pricing-svc worker-2 60000 0-0 COUNT 100
Contrasts Pub/Sub's PUBLISH/SUBSCRIBE, which only reaches connected clients, with a Streams consumer group that trims, re-reads pending entries, acknowledges with XACK, and reclaims work from dead consumers.
How do you keep invalidation events tied to committed writes?
A more common source of lost invalidations sits upstream of the broker. A service that commits to the database and then publishes an event is doing two separate writes. The process can crash, the broker can reject the publish, or a network timeout can occur between the two. When that happens the row has changed, no event exists, and every cache holding that key serves the old value until its TTL expires. Retries inside the request handler don't solve this, because a crash erases the retry along with everything else.
Transactional outbox
The fix is to make the event part of the commit. The service inserts a row into an outbox table in the same transaction as the business write. That table typically has an id, aggregate type, aggregate key, payload, created timestamp, and a nullable published timestamp. If the transaction rolls back, the event disappears with it. A separate relay polls unpublished rows in id order, publishes them, and marks them done. Delivery is at-least-once, which is acceptable because evicting a key twice is harmless.
Change data capture
CDC tails the database's replication log, such as the PostgreSQL WAL or the MySQL binlog. It emits a change for every committed row, so application code never has to remember to write an event.
| Property | Outbox + relay | CDC from log |
|---|---|---|
| Operational cost | Extra table, relay process, row cleanup | Connector cluster, replication slot, log retention tuning |
| Ordering | Per outbox id; total if one relay | Commit order from the log |
| Added lag | Polling interval plus publish | Log read latency, usually lower |
| Schema coupling | Explicit event payload | Raw row changes, tied to table layout |
-- PostgreSQL. Outbox rows commit or roll back together with the business write.
CREATE TABLE outbox (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
aggregate_type TEXT NOT NULL, -- e.g. 'product'
aggregate_key TEXT NOT NULL, -- cache key suffix to evict
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
published_at TIMESTAMPTZ -- NULL until the relay publishes it
);
-- Keeps the relay's poll cheap: only unpublished rows are indexed.
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;
-- Business write and invalidation event in one transaction.
BEGIN;
UPDATE products SET price_cents = 1999 WHERE id = 42;
INSERT INTO outbox (aggregate_type, aggregate_key, payload)
VALUES ('product', '42', '{"op": "evict", "key": "product:42"}');
COMMIT;
-- Relay, step 1: claim a batch in id order. Run it in its own transaction.
-- With a single relay this preserves outbox id order. SKIP LOCKED lets
-- several relays run, but then ordering across relays is no longer total.
BEGIN;
SELECT id, aggregate_key, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;
-- Relay publishes each row to the broker here.
-- Step 2: mark only the ids that the broker acknowledged. A crash before
-- this UPDATE means a re-publish, which is harmless for evictions.
UPDATE outbox SET published_at = now() WHERE id = ANY('{101,102,103}'::bigint[]);
COMMIT;
A PostgreSQL outbox table, a business update that commits its eviction event in the same transaction, and the relay's claim-publish-mark loop that gives at-least-once delivery.
How do you prevent stale data from being written back after eviction?
Ordering the events correctly does not stop a slow reader from undoing the eviction. The race takes four steps:
- Reader A misses the cache and reads version 7 from the database.
- A writer commits version 8 and publishes the invalidation.
- The consumer deletes the key, which is already empty.
- Reader A finishes and writes version 7 into the cache.
The cache now holds stale data, and no further event will arrive to remove it. The entry stays wrong until its TTL expires.
| Mitigation | What it guarantees | What it does not |
|---|---|---|
| Versioned entry with compare-and-set (a Lua script that sets only if the incoming version is newer) | An older version can never overwrite a newer one | Correctness when the source has no reliable monotonic version. Legacy tables and third-party APIs often lack one. |
| Short tombstone after eviction | Write-backs are rejected for the tombstone window | Protection from readers slower than the window. Misses are also forced for its duration. |
| Delayed double delete | A second delete clears most late write-backs | Any hard bound. A reader slower than the delay still wins. |
Compare-and-set is the only option here that is correct regardless of timing. The other two make the race less likely but do not remove it.
Eviction also creates a load problem. When a hot key is deleted, every concurrent request misses at once and queries the database. Single-flight coalescing lets one request per key load the value while the others wait for its result. Run it per process, or across the fleet with a short lock key. Pair it with compare-and-set so that the single loader cannot reintroduce stale data either.
import redis
r = redis.Redis(host="cache.internal", port=6379)
TTL = 300 # seconds; also bounds how long the version marker survives
WRITE_IF_NEWER = r.register_script("""
local cur = tonumber(redis.call('HGET', KEYS[1], 'v'))
if cur and cur >= tonumber(ARGV[1]) then return 0 end
redis.call('HSET', KEYS[1], 'v', ARGV[1], 'd', ARGV[2])
redis.call('EXPIRE', KEYS[1], tonumber(ARGV[3]))
return 1
""")
INVALIDATE = r.register_script("""
local cur = tonumber(redis.call('HGET', KEYS[1], 'v'))
if cur and cur >= tonumber(ARGV[1]) then return 0 end
redis.call('HDEL', KEYS[1], 'd')
redis.call('HSET', KEYS[1], 'v', ARGV[1])
redis.call('EXPIRE', KEYS[1], tonumber(ARGV[2]))
return 1
""")
def on_invalidation(key, new_version):
INVALIDATE(keys=[key], args=[new_version, TTL])
def read_through(key, load_from_db):
data = r.hget(key, "d")
if data is not None:
return data
# load_from_db must return the row's monotonic version (e.g. a version column).
version, value = load_from_db()
# Returns 0 if a newer version already landed; the stale value is not cached.
WRITE_IF_NEWER(keys=[key], args=[version, value, TTL])
return value
A Redis Lua compare-and-set where invalidation leaves the new version behind as a marker instead of deleting the key, so a slow reader's older version is rejected on write-back; once the marker's TTL expires the protection lapses, the same limit a tombstone has.
How do you invalidate lists, search results, and other aggregate keys?
A single entity change rarely maps to a single key. Updating one product's price can invalidate its detail key, its category listing, a "top sellers" page, and every cached search result that included it. The writer usually cannot enumerate those keys, and making it try couples the writer to every reader's caching choices.
Tag sets keep invalidation precise. When a reader caches a derived value, it also adds that key to a set per entity it depends on, such as tag:product:42. On change, the invalidator reads the set, deletes its members, and deletes the set. The cost is fan-out proportional to set size, plus dangling members when entries expire on their own.
Generation counters trade precision for simplicity. Embed a version in the key (catalog:g17:category:9) and increment it on any relevant write. Old keys become unreachable in one O(1) operation, but they stay in memory until TTL or the eviction policy removes them. Always set a TTL anyway, and accept that one write empties the whole namespace.
For unbounded key spaces like free-text search, tracking dependencies costs more than it saves. A short TTL and a few seconds of stale cache are usually the right answer.
In-process caches in front of the shared cache need their own subscription, because deleting from Redis does not touch them. Redis Pub/Sub has no replay or sequence numbers, so a subscriber cannot tell what it missed. Any reconnect must flush the entire local cache.
To limit broadcast traffic, batch evictions over a short window into one message, or publish coarse namespace or generation bumps instead of individual keys.
When should you avoid event-driven eviction, and what does it cost to run?
Event-driven eviction is a pipeline you have to run, so it should solve a real staleness problem. Skip it for data that rarely changes, for data where minutes of staleness are acceptable, or when no one on the team can operate a durable event pipeline.
What are the alternatives?
Each alternative below trades staleness, load, or coupling differently. Pick per data type, not per system; many teams run TTL-only for reference data and event-driven eviction only for a handful of high-cost entities.
| Approach | Fits when | Avoid when |
|---|---|---|
| TTL-only | Reference data, config, or catalog fields where a bounded window of staleness is fine | Users act on the value immediately, such as prices, permissions, or inventory |
| Write-through / update-on-write | A single service owns both the write and the cache | Many services cache derived views of the data |
| No cache / read replicas | The source can absorb the read load, and replica lag is acceptable or already measured | Read cost or latency is the real bottleneck |
| Read-through with short TTL | Hot keys where seconds of staleness are tolerable | A short TTL would multiply misses on expensive queries |
Two further options sit between these and full event-driven eviction. A consumer can ask the owning service for the entity's current version on each read and serve its cached copy only if the versions match; this removes staleness at the cost of one cheap round trip per read. Or a consumer can stop caching the owner's data and call the owner directly, moving the caching responsibility to the one service that sees every write.
How can it save engineering time?
The costs are ongoing. You maintain versioned event contracts. You operate outbox or CDC infrastructure. Every consumer needs reconnect handling, idempotent eviction, and updates whenever the schema changes. In return, you get fewer stale-data incidents and fewer late-night manual cache flushes. That saving only appears if those incidents are actually happening today.
Where the time savings come from, when they are real:
- Less debugging of "wrong value" reports. Stale-cache bugs are slow to diagnose because the database looks correct and the symptom disappears once the TTL expires. A versioned pipeline with lag metrics turns "can't reproduce" into a specific consumer, key, and version.
- No cross-team TTL negotiation. Without events, every consumer has to guess an acceptable TTL for data it does not own and revisit it whenever the owner's write patterns change. With events, the TTL becomes a backstop rather than a correctness setting that has to be tuned.
- Writers stay decoupled from readers. The owner publishes one event per change. New consumers subscribe without the owner changing its code, which avoids the coordination cost of adding cache-delete calls for every new reader.
- Fewer manual flushes and runbook steps. Operators stop flushing caches by hand after data fixes, migrations, or backfills, because those writes go through the outbox or CDC like any other change.
- One shared consumer library. Idempotent, version-aware eviction and reconnect handling are written once and reused, instead of each team building its own partial version.
Illustrative estimate: assume a team handles 4 stale-data incidents a month, each taking 3 engineer-hours to triage and resolve, and that the pipeline needs 6 engineer-hours a month to operate. The saving would be roughly 12 hours against 6 hours of upkeep. With 1 incident a month at the same assumed cost, the pipeline costs more than it saves. Plug in your own incident history before deciding.
Measure staleness instead of assuming it is solved:
- Invalidation lag: the time from commit to eviction, tracked as a distribution with its tail visible, not only the mean.
- Events published vs. consumed: compare counts per consumer. A persistent gap points to a broken or slow subscriber.
- Reconnect counts: each reconnect triggers a full local flush, so frequent reconnects show up as both a load problem and a correctness problem.
- Sampled comparisons: periodically read random cached keys, compare them to the source of record, and report the mismatch rate. This is the only metric that shows what users actually see.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to distributed cache invalidation, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.