Microservices Caching Invalidation: TTLs, Events and Stale Data

What a cross-service cache really promises: staleness as a producer-owned contract

An inter-service cache is a copy of another team's data that your service keeps so it doesn't have to call the owning service on every request. Consumers add one for three reasons. Reads get faster because a local lookup replaces a network hop. The producer's load drops because repeated reads never reach it. And the consumer stays available because it can keep serving when the producer is slow or down.

The cost is that every cached value may be out of date. The useful question is not whether the data is stale but how stale it is allowed to be, and that depends on what the field is used for.

Most teams leave this choice to each consumer, so each one picks a TTL by guesswork. The producer is in a better position to decide. It knows how often each field changes and what a wrong value costs downstream. It should publish a maximum acceptable staleness for each field class alongside its API contract and version that guarantee like any other part of the interface. Consumers then build their caches to meet a stated limit rather than an assumption.

This guide is for teams that read another team's data at enough volume that calling the producer on every request is not an option.

How TTLs and invalidation events work together

The model is simple: the TTL sets the worst-case staleness bound, and events only shrink the typical case. If every event is lost, a consumer still converges once the TTL expires. So derive the TTL from the producer's staleness budget. If the contract says "prices may be up to 60 seconds stale", the TTL is at most 60 seconds, whatever the load. Raising it to protect the origin breaks the contract.

Illustrative estimate

Assumed inputs: 20 consumer instances, each holding 2,000 hot keys that are read continuously. Each instance refetches each key once per TTL.

If the origin can't sustain the rate your budget requires, fix capacity or the contract. Don't quietly raise the TTL.

Why events stay best-effort

A durable broker keeps messages, but delivery to the cache can still fail or arrive late:

• A consumer is down or lagging.
• A consumer is partitioned from the broker.
• A consumer is mid-deploy.
• The producer commits its write and then crashes before publishing.

The transactional outbox closes that last gap. The producer writes the row and an outbox record in one transaction, and a relay publishes from the outbox. Delivery becomes at-least-once, so invalidation handlers must be idempotent. The outbox does nothing about consumer-side failures. That is what the TTL is for.

CopyProducer DB ──(same txn)── Outbox table
                              │ relay
                              ▼
                           Broker ──(may lag/drop)──► Consumer
                                                        │ invalidate
                                                        ▼
                                                  Cache entry
                                                        │
                                       TTL expiry ──────┘ (fallback bound)

Copy+---------------- Producer service -----------------+
|                                                   |
|  +--------------+  same txn  +----------------+   |
|  | Producer DB  |<---------->|  Outbox table  |   |
|  | (data row)   |   commit   | (event record) |   |
|  +--------------+  together  +-------+--------+   |
|                                      |            |
+--------------------------------------+------------+
                                       | relay publishes
                                       | (at-least-once)
                                       v
                               +----------------+
                               | Durable broker |
                               +-------+--------+
                                       |  may lag or never arrive:
                                       |  consumer down, lagging,
                                       |  partitioned, mid-deploy
                                       v
+---------------- Consumer service ----+-----------------------+
|                                      v                       |
|  +---------------------+     +-------+----------------+    |
|  | Local cache         |<----| Invalidation handler   |    |
|  | entry expires at TTL| del | (idempotent)           |    |
|  +----------+----------+     +------------------------+    |
|             | miss or expired                              |
+-------------+----------------------------------------------+
              v
              refetch from producer origin (TTL fallback)
              worst-case staleness <= TTL, even if all events are lost

The outbox makes the data write and its invalidation event commit atomically, the broker delivers events at-least-once to an idempotent handler, and because consumer-side delivery can still lag or fail, the cache TTL remains the fallback that bounds worst-case staleness.

Implementing invalidation correctly: payloads, ordering and the cache-aside race

Thin vs. fat events

Thin events push load back to the producer. A popular key can trigger a refetch burst from every consumer. Fat events avoid that burst, but consumers now rely on the producer's schema.

Ordering and versions

Brokers redeliver messages and may reorder them across partitions. If version 7 arrives after version 8, a naive handler puts stale data back into the cache. The fix is for the producer to attach a version that only increases per entity, such as a row version or a counter updated in the same transaction. Consumers then reject any write whose version is lower than the one already cached.

The cache-aside race

A reader misses the cache, reads version 7 from the producer, and pauses. The update event for version 8 arrives and deletes the key. The reader resumes and writes version 7 back. Mitigations:

1. Version-checked writes. Use an atomic compare that writes only if the incoming version is higher than the stored version or the tombstone version.
2. Tombstones. On an event, write a short-lived marker carrying version 8 instead of deleting the key, so late version 7 writes fail the check.
3. Event invalidation plus a short TTL. This bounds the damage when the first two mitigations miss a case.

The handler follows these steps: parse the key and version, perform an atomic compare-and-set of the tombstone, and acknowledge only after the write succeeds.

These interleavings rarely show up in normal tests. Use fault injection, such as delayed readers, duplicate events and reordered events, to exercise them.

Copyimport json
import redis

r = redis.Redis(host="cache.internal", port=6379)

# Each cache entry is a hash: v = version, d = payload ("" marks a tombstone).
# Write only if the incoming version is newer, or if it equals a tombstone's
# version (a refetch of the state the event announced).
VERSIONED_SET = r.register_script("""
local cur = redis.call('HMGET', KEYS[1], 'v', 'd')
local inc = tonumber(ARGV[1])
if cur[1] then
  local cv = tonumber(cur[1])
  if inc < cv or (inc == cv and cur[2] ~= '') then return 0 end
end
redis.call('HSET', KEYS[1], 'v', ARGV[1], 'd', ARGV[2])
redis.call('PEXPIRE', KEYS[1], ARGV[3])
return 1
""")

TTL_MS = 60_000        # short TTL bounds damage from any missed case
TOMBSTONE_MS = 5_000   # outlives a slow in-flight cache-aside read

def cache_fill(key, version, entity):
    # Cache-aside reader: a late write of v7 fails against a v8 tombstone.
    return VERSIONED_SET(keys=[key], args=[version, json.dumps(entity), TTL_MS])

def on_entity_changed(event):
    # Thin event: {"key": ..., "version": N}. Tombstone instead of DEL.
    key, version = event["key"], int(event["version"])
    VERSIONED_SET(keys=[key], args=[version, "", TOMBSTONE_MS])
    # Stale or redelivered events (lower version) are rejected by the script.

def cache_get(key):
    v, d = r.hmget(key, "v", "d")
    if v is None or d == b"":
        return None    # miss or tombstone: caller refetches with version
    return json.loads(d)

Illustrative Redis Lua compare-and-set that rejects lower-version writes, with an event handler that writes a short-lived tombstone instead of deleting so a paused cache-aside reader cannot restore stale data; the race and redelivery paths still need fault-injection tests.

Layering mechanisms and preventing stampedes

No single mechanism is enough on its own, so stack them and give each one a distinct job. The TTL is the safety net. It sets the worst-case bound when an event is lost, a consumer is down, or a bug skips a publish. Invalidation events handle fast convergence in the common case. For high-value reads, add conditional revalidation: send the cached ETag or version number to the producer, and it either confirms the entry is current or returns a fresh copy. This costs one round trip but no full payload, and it gives you near-read-through freshness without discarding the cache.

Some decisions should not read from a cross-service cache at all. These include authorization checks, balance or payment state, inventory reservations, and anything where acting on stale data is irreversible or exploitable. Query the owning service directly. A revoked permission that stays live for one TTL is a security incident, not a latency win.

Preventing stampedes

When a mass invalidation lands, or many entries written at the same time expire together, every caller misses at once and the producer absorbs the full load. Three defenses address this:

• TTL jitter. Add a random offset to each TTL, for example ±10% of the base value, so expirations spread out instead of clustering.
• Per-key single-flight. Coalesce concurrent misses so that only one request per key fetches from the producer. The other callers wait on that in-flight result.
• Stale-while-revalidate. Serve the expired value while one refresh runs in the background. Do this only while the entry's age stays inside the producer's published staleness budget. Past that point, block on the refresh or fail.

Copyconst BASE_TTL_MS = 300_000; // 5 minutes
const JITTER = 0.1;          // +/-10% of base

// Uniform in [base*(1-j), base*(1+j)] so entries written together expire apart.
function jitteredTtl(baseMs: number, jitter = JITTER): number {
  const offset = (Math.random() * 2 - 1) * jitter * baseMs;
  return Math.round(baseMs + offset);
}

type Entry<T> = { value: T; expiresAt: number };
const cache = new Map<string, Entry<unknown>>();
// One pending producer call per key. In-process only: each instance
// still makes its own call, so N instances means at most N calls per key.
const inflight = new Map<string, Promise<unknown>>();

async function getOrLoad<T>(key: string, load: () => Promise<T>): Promise<T> {
  const hit = cache.get(key) as Entry<T> | undefined;
  if (hit && hit.expiresAt > Date.now()) return hit.value;

  // Single-flight: concurrent misses for this key wait on the same promise.
  const pending = inflight.get(key) as Promise<T> | undefined;
  if (pending) return pending;

  const p = (async () => {
    try {
      const value = await load();
      cache.set(key, { value, expiresAt: Date.now() + jitteredTtl(BASE_TTL_MS) });
      return value;
    } finally {
      // Always clear, even on failure, so the next caller retries
      // instead of reusing a rejected promise.
      inflight.delete(key);
    }
  })();
  inflight.set(key, p);
  return p;
}

An in-process cache that applies a +/-10% jittered TTL to each entry and coalesces concurrent misses for the same key into one producer call.

Key and schema design across service boundaries

A cache key identifies more than an entity. It also identifies the shape of the bytes stored under it. If the key is customer:4821 and the customer service ships a deploy that renames address to addresses and turns it into a list, consumers keep reading entries in the old shape until they expire. A consumer on the new client library also starts failing on entries it can no longer parse. To prevent this, put the producer's contract version in the key. When the contract changes, the version bump moves everyone to a fresh keyspace. Old entries are never read again and age out on their TTL. Base the version on the schema or contract version, not the deploy SHA. Otherwise every unrelated release cold-starts every consumer.

Also namespace keys by the consuming service. A shared cache between teams creates coupling that no API contract records:

• Key formats. One team's naming change breaks another team's lookups.
• Eviction pressure. A batch job filling memory evicts a latency-sensitive service's hot set.
• Flushes. A flush issued during one team's incident wipes everyone's cache.

Per-service namespaces, or separate instances, remove that coupling. The cost is memory. If three consumers each cache the same 2 GB of customer records, they hold roughly 6 GB instead of 2 GB (illustrative estimate). That trade is usually worth it for hot, small datasets. It is worth less for large, rarely shared ones.

A workable layout is {consumer}:{producer}:{entity}:v{contract}:{id}. For example, billing:customers:customer:v3:4821 sits beside shipping:customers:customer:v3:4821. Each consumer can then be scanned, flushed or measured independently.

Copyimport json
import redis

# Contract version comes from the producer's published schema, not the deploy SHA,
# so unrelated releases do not cold-start consumers.
CUSTOMER_CONTRACT_VERSION = 3
CONSUMER = "billing"  # each consuming service owns its own namespace

r = redis.Redis(host="cache.internal.example", port=6379)


def customer_key(customer_id: int) -> str:
    # {consumer}:{producer}:{entity}:v{contract}:{id}
    return f"{CONSUMER}:customers:customer:v{CUSTOMER_CONTRACT_VERSION}:{customer_id}"


def get_customer(customer_id: int, fetch_from_service) -> dict:
    key = customer_key(customer_id)
    cached = r.get(key)
    if cached is not None:
        # Safe to parse: anything under this key was written with contract v3.
        return json.loads(cached)
    customer = fetch_from_service(customer_id)
    # Entries under v2 keys are never read again; they age out on their TTL.
    r.set(key, json.dumps(customer), ex=300)
    return customer


def count_own_keys() -> int:
    # Scoped to this consumer only; shipping's keys are never touched.
    # SCAN is incremental, unlike KEYS, so it does not block the server.
    return sum(1 for _ in r.scan_iter(match=f"{CONSUMER}:customers:*", count=500))


def purge_own_keys() -> None:
    # A targeted purge instead of FLUSHALL, which would wipe every team's cache.
    for key in r.scan_iter(match=f"{CONSUMER}:customers:*", count=500):
        r.delete(key)

A Python cache helper using redis-py that builds keys as {consumer}:{producer}:{entity}:v{contract}:{id}, so a contract version bump moves reads to a fresh keyspace and each consumer can count or purge only its own keys.

Making staleness observable

If invalidation is a contract, you need evidence that it is being honoured. Four signals cover most failure modes:

• Entry age at read time. Record how old each cached value is when it is served, as a histogram tagged by key prefix. If the producer promises 30 seconds of staleness and the p99 age sits at four minutes, events are not arriving or not being applied.
• Consumer lag. Measure the gap between when an event was published and when the consumer processed it. Lag that exceeds the TTL means the TTL is the only thing bounding staleness.
• Version-rejected writes. Count writes discarded because a newer version was already cached. A steady trickle is normal reordering. A spike points to a replay or a misbehaving producer.
• Tombstone hits. Count reads that land on a deletion marker. These are reads the race guard actually prevented, which shows whether tombstones earn their cost.

None of these signals proves the cached data is correct. For that, run a reconciliation job that samples a small slice of cached keys on a schedule, fetches the same entities from the source of truth, and compares them by version or content hash. Report the divergence rate per consumer and per entity type, and keep the mismatched keys so someone can trace them.

This changes how stale-data bugs reach you. "Customer address looked wrong yesterday" becomes "shipping's customer cache diverged 2% between 14:00 and 14:20 while consumer lag peaked." That report names a service, a time window and a likely cause, which saves the hours otherwise spent proving the cache was even involved.

When to avoid event-driven invalidation, alternatives and cost impact

That payoff only exists when events are worth running. Two cases argue against them.

For low-traffic or highly tolerant data, such as country lists, plan metadata or a profile edited monthly, a plain TTL is cheaper and more honest. The staleness bound is the TTL, stated once, and no broker sits in the failure path. For reads that must be current, such as a balance check before a debit or stock at reservation time, no cache meets the requirement. Read the source of truth.

Illustrative estimate (assumed numbers, not a benchmark): producer reads/s under TTL only = hot keys ÷ TTL seconds. Assume 10,000 keys and a 60 s TTL. That gives 10,000 ÷ 60 ≈ 167 reads/s. If the producer handles that comfortably, events add no load benefit.

Events carry real costs:

• A broker to operate.
• An outbox table and its relay.
• Consumer handlers in every service that caches.
• Version metadata on each entry.
• Tests for duplicates, reordering and lag.

The staleness contract and its metrics recover part of that cost. Incidents become a dashboard query rather than a cross-team meeting.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservices caching invalidation, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.