Cache Invalidation in Microservices with CDC and Change Events

Why application-level dual writes leave stale cache data

Event-driven cache invalidation means the cache is updated or purged in response to a record of a committed database change, rather than by the code path that made the change. Teams adopt it because the usual alternative is the application-level dual write: commit to the database, then delete or overwrite the cache entry. That pattern has three common failure modes.

A crash between the two writes. The transaction commits. Then the process is killed, the cache call times out, or a deploy drains the pod. The cache keeps the old value until its TTL expires, and nothing records that an invalidation was missed.

Concurrent writers finishing in different orders. Writer A commits version 1 and writer B commits version 2. Their cache updates are separate network calls with no ordering guarantee, so B's update can land first and A's can land second. The cache then holds version 1 even though the database holds version 2.

The read-after-delete race. A reader misses the cache and loads version 1 from the database. A writer then commits version 2 and deletes the key. The slow reader finally writes version 1 back into the cache. The delete ran, but the cache is still stale.

The common cause is that the cache write is a second, independent operation that can fail, reorder, or be overtaken. The principle that follows is that the database commit should be the single trigger for invalidation. This matters most to teams that run a shared cache across several services or many replicas. In that setup, every writer is a separate source of these races, and no single codebase can coordinate them.

How it works: outbox and log-based CDC as the source of invalidation events

The fix is to stop treating the writer as the source of invalidation and derive invalidation from the committed data itself. The flow has four stages. A transaction commits. A capture process sees that commit, either by tailing the database's replication log or by reading rows from an outbox table written in the same transaction. The capture process publishes an event to a broker topic partitioned by the entity's primary key. Invalidation consumers read the topic and delete or refresh the affected cache entries. Partitioning by key matters because it keeps every change to one row in a single ordered stream. Consumers then never see the update for a row arrive before the insert that created it.

The outbox fits better when consumers need stable, meaningful events and you own the write paths. CDC fits better when many legacy services write to the same tables and changing all of them isn't realistic. With CDC, plan a translation layer to keep raw column names from leaking into cache logic.

Copy+-----------------+        +-----------------+
| Service A write |        | Service B write |
+--------+--------+        +--------+--------+
         v one transaction          v one transaction
+- Database ---------------------------------------------+
| +-------------+   +--------------+   +--------------+  |
| | entity rows |   | outbox table |   | repl. log    |  |
| +-------------+   +------+-------+   +------+-------+  |
+--------------------------+------------------+----------+
                           v poll rows        v tail log
                  +--------+---------+  +-----+------------+
                  | Outbox relay     |  | CDC connector    |
                  | domain events    |  | row before/after |
                  +--------+---------+  +-----+------------+
                           +--------+---------+
                                    v publish, key = primary key
          +-------------------------+--------------------------+
          | Broker topic, partitioned by key  [p0] [p1] [p2]   |
          +-------------------------+--------------------------+
                                    v ordered per key
                  +-----------------+------------------+
                  | Invalidation consumers             |
                  | CDC path: translate columns first  |
                  +-----------------+------------------+
                                    v delete or refresh entry
                  +-----------------+------------------+
                  | Cache                              |
                  +------------------------------------+

Committed writes reach the cache through an outbox relay or a log-tailing CDC connector, a broker topic partitioned by primary key, and invalidation consumers.

Implementing a consumer that is safe under late, duplicate and out-of-order events

Each event leaves the consumer two choices: delete the cached key, or write the new value into it.

Delete plus cache-aside reload is the safer default because the database stays the only source of values. Use refresh only for keys that are too expensive to miss.

Partition the event stream by entity primary key. All changes to one row then land on one partition and arrive in commit order. That ordering holds only within a partition, and rebalances and retries still redeliver events, so the consumer must be correct without it.

For refresh, store the row's commit position next to the cached value. This can be the log sequence number or an outbox sequence column. Apply the write atomically, as a scripted or compare-and-set operation: read the stored version, write only if the incoming version is greater, and otherwise drop the event. Cache-aside reloads should carry the version too. Without it, a reader that fetched the row before the delete can repopulate the key with stale data.

Deleting a missing key succeeds, so the delete handler needs no deduplication table. Commit the stream offset only after the cache operation completes. A crash then causes redelivery, never loss. Replay after an offset reset is also safe: deletes repeat harmlessly, and the version check rejects old writes.

Copyimport json
import redis  # redis-py client

r = redis.Redis(host="cache.internal", port=6379)

# Atomic compare-and-set: write only if the incoming version is newer.
# Lua numbers are doubles, so the version must stay below 2^53.
CAS_WRITE = r.register_script("""
local cur = redis.call('HGET', KEYS[1], 'v')
if cur and tonumber(cur) >= tonumber(ARGV[1]) then
  return 0  -- stale or duplicate event: drop it
end
redis.call('HSET', KEYS[1], 'v', ARGV[1], 'data', ARGV[2])
redis.call('EXPIRE', KEYS[1], ARGV[3])
return 1
""")

def refresh(key, version, row, ttl=3600):
    # Used by the CDC consumer and by cache-aside reloads alike,
    # so a reader holding an old row cannot overwrite a newer one.
    return CAS_WRITE(keys=[key], args=[version, json.dumps(row), ttl])

def invalidate(key):
    # DEL on a missing key returns 0 without error: duplicates
    # and replays need no deduplication table.
    r.delete(key)

def handle(event, hot_keys):
    key = f"order:{event['pk']}"
    if event["op"] == "delete" or key not in hot_keys:
        invalidate(key)  # safe default: next read reloads from the DB
    else:
        refresh(key, event["commit_seq"], event["row"])

def consume(stream, hot_keys):
    for event in stream.poll():
        handle(event, hot_keys)
        # Commit only after the cache operation succeeds:
        # a crash causes redelivery, never loss.
        stream.commit(event)

A Redis Lua script performs a version-checked conditional write that rejects stale or duplicate refreshes, while the delete handler stays idempotent and the consumer commits the stream offset only after the cache operation completes.

Mapping row changes to cache keys, including derived and query caches

A safe consumer is only useful if it knows which keys to touch. That mapping is often the hardest part, because the consumer must work out every cache entry that depends on a row it has never seen before.

Direct entity keys are easy. A change to orders row 42 maps to order:42. Derived keys are harder. A customer's lifetime-spend total depends on every order row for that customer. The consumer therefore needs the foreign key in the event payload, not just the primary key. List and query caches are the worst case. A cached "open orders, page 3, sorted by date" result can change when a row enters, leaves or moves within the result set. You cannot infer that from the changed row alone. Cross-service read models add another layer: the dependency crosses a team boundary, and nobody owns the full graph.

A common split is precise eviction for entity and derived keys, and version bumps for query caches.

Do not build key mapping on raw CDC rows. If the owning service renames a column or splits a table, every downstream mapper breaks. Have the owner translate rows into a stable event contract, such as OrderStatusChanged with explicit IDs. The owner can then change its schema without breaking the consumers' key logic.

Copyimport redis

r = redis.Redis(host="cache.internal", port=6379)

# Consumes the owner's stable contract (OrderStatusChanged), not raw CDC rows.
# Example event:
# {"type": "OrderStatusChanged", "order_id": 42, "customer_id": 7,
#  "old_status": "open", "new_status": "shipped"}

def keys_for(event: dict) -> list[str]:
    order_id = event["order_id"]
    customer_id = event["customer_id"]  # FK must be in the payload for derived keys
    return [
        f"order:{order_id}",                     # direct entity key
        f"customer:{customer_id}:lifetime_spend",  # derived / aggregate key
        f"customer:{customer_id}:order_count",
    ]

def handle(event: dict) -> None:
    if event["type"] != "OrderStatusChanged":
        return
    # Precise eviction for entity and derived keys.
    r.delete(*keys_for(event))
    # Query caches: the row may have entered or left "open orders" pages,
    # which we cannot compute from one row. Bump the namespace generation
    # so every cached page under the old generation is simply never read again.
    if "open" in (event["old_status"], event["new_status"]):
        r.incr("gen:orders:open_list")

def open_orders_page_key(page: int, sort: str) -> str:
    # Readers embed the current generation in the key name.
    gen = int(r.get("gen:orders:open_list") or 0)
    return f"orders:open:v{gen}:sort={sort}:page={page}"

# Old-generation pages should carry a TTL when written so they expire
# on their own instead of accumulating after a bump.

A consumer that maps a stable OrderStatusChanged event to precise evictions for entity and derived keys, and uses a generation-counter bump for the open-orders query cache.

TTLs, monitoring and bounding the staleness window

Event-driven invalidation should be the primary path, but every entry still needs a TTL. When a connector stalls or a consumer crashes, no events arrive and nothing gets evicted. Entries written just before the stall would then stay wrong indefinitely. The TTL is the only mechanism that keeps working when the pipeline does not. Size it from the business side: how long can a customer see an old price or order status during an outage? Don't size it from hit-rate targets. A shorter TTL costs more source reads. A longer one widens the worst-case outage exposure.

Monitor the pipeline as a staleness system:

• Consumer lag, expressed in seconds rather than offsets. It approximates how far behind the cache is.
• Invalidation failure and retry counts. A rising retry rate often comes before visible lag.
• Connector health, including task state and the age of the last captured change.
• Sampled reconciliation. Periodically read a random set of keys from both the cache and the source of truth, and record the mismatch rate. This catches mapping bugs that produce no errors.

Worked example (illustrative estimate)

All values below are assumed inputs, not measurements.

With a 5-minute TTL, normal staleness is about 10 seconds. A stalled pipeline degrades staleness to 5 minutes rather than leaving it unbounded. Alert when lag approaches the TTL. At that point the TTL, not invalidation, is what keeps data fresh.

When to avoid it and what to use instead

If a five-minute TTL already meets your product's staleness tolerance and the cache serves a handful of services, a CDC pipeline adds a connector, a broker topic and a consumer to operate for little gain. The opposite case also rules it out. Workloads that need strict read-your-writes, such as a user editing their own profile or a checkout total, should read from the primary. Asynchronous invalidation always leaves a window between commit and eviction, and no amount of tuning closes it.

The pipeline also creates hazards of its own:

• Invalidation storms. A bulk update, backfill or schema migration emits one event per row. That can evict most of the cache at once and send the full read load to the database. Throttle these jobs, or pause invalidation and let TTLs handle the change.
• Thundering herds. When a hot key is evicted, every concurrent miss queries the database unless you coalesce requests so that a single caller rebuilds the entry while the others wait.
• Silent staleness. A stuck connector, a poison message blocking a partition, or a lost replication position produces no errors. Invalidations simply stop arriving.

Impact on development cost and engineering time

The pipeline is not free. Someone has to run the connectors, keep the broker healthy, rotate credentials for a replication user that can read every table it captures, and handle schema changes so a renamed column does not silently stop invalidations. Connector and broker upgrades need testing against your database version, and that work recurs. Budget it as an ongoing on-call responsibility for a platform owner, not a one-time project.

The savings are less visible but compound. Invalidation calls scattered through write paths disappear, so new endpoints cannot forget them. Stale-data bugs, which are expensive because they depend on timing and rarely reproduce, become pipeline problems you can see in lag and error metrics. Each additional service reuses the same consumer pattern instead of building its own.

To compare the two for your system:

Net time saved per quarter = (services × invalidation-related engineering hours per service) + (stale-data incidents avoided × hours per incident) − pipeline operating hours

Fill this in with your own incident history. If you have no records of stale-data incidents, that is itself a signal that you can wait.

Adopt now if you have several services sharing cached entities, a team that already operates a broker, and recurring stale-data incidents.

Defer if you have one team, one database and few cached write paths.

Start with an outbox for a single service if one high-churn entity causes most of the pain. It gives you ordered events without committing to log-based CDC everywhere.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cache invalidation microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.