Monolith Caching Strategy: In-Process vs Shared Cache Trade-Offs

What changes when a monolith runs as N replicas

A monolith running as a single process has one cache. A write goes through that process, so the process can evict or update the entry it just changed, and every later read sees the new value. Put the same monolith behind a load balancer as N replicas and you have N independent in-process caches. None of them knows what the others hold, and an update handled by one replica does nothing to the copies held by the rest.

Consider a user who renames a project. Replica A handles the write, updates the database and evicts its own cached entry. The next request goes to replica B, which still holds the old name and serves it until its TTL expires or an invalidation message arrives. The user refreshes and, depending on where that request lands, the old name may come back. Nothing has failed here. Each replica is behaving correctly with the state it has. The problem only appears when you compare replicas.

This applies to any team that has scaled a monolith horizontally without changing its caching code, which is the usual path.

This article takes a default position. Start with a small in-process cache with a short TTL, and use it only for hot data that can tolerate brief staleness. Add a shared cache only when you can name the specific problem it solves:

• Cross-instance consistency: replicas must agree on a value.
• Cold-start load: new instances hit the database too hard while their caches warm up.
• Memory duplication: N copies of a large working set cost too much.

Cost model: memory access versus network round trip

Each tier exists because a lookup costs very different amounts in each place. An in-process hit is a memory access that returns an object reference, with no serialization. A shared-cache hit adds a network round trip, serializing on write and deserializing on read, and a dependency that can time out or fail.

Illustrative estimate (assumed numbers, not a benchmark). Assume 10 replicas at 200 req/s each (2,000 req/s total), a 20 ms database query, an in-process hit near 0 ms with an 80% hit ratio, and a shared hit of 1 ms with a 95% hit ratio.

• DB load = total rate × (1 − hit ratio): 400 q/s in-process versus 100 q/s shared.
• Mean latency = h × hit cost + (1 − h) × (lookup + DB cost): about 4.0 ms in-process versus 0.95 + 0.05 × 21 ≈ 2.0 ms shared.

With these inputs the shared cache gives lower mean latency, because its higher hit ratio outweighs the extra round trip on each hit. If you change the hit ratios, the result can reverse.

There are also hidden costs:

• Heap pressure: in-process entries add GC work, and that cost is multiplied by the replica count.
• Serialization coupling: during a rolling deploy, old and new class shapes read the same shared entries.
• Masked slow queries: a slow query stays hidden until a cold start or a flush exposes it.

An in-process cache has no extra infrastructure to run. A shared cache adds provisioning, monitoring, failover and capacity planning, and that operational work takes ongoing engineering time.

How an in-process Caffeine cache works and how to configure it

Caffeine stores entries on the JVM heap, so every bound you set is a bound on heap usage and GC pressure.

Size-bounded eviction

maximumSize caps the entry count. If entries vary widely in size, use maximumWeight with a weigher instead. When the cap is reached, Caffeine evicts using a frequency-aware policy (Window TinyLFU) rather than plain LRU, so a one-off scan of cold keys does not flush the hot set. Without a bound, a cache keyed on user input is a memory leak.

Expiry modes

expireAfterWrite limits how old a value can be, no matter how often it is read. This puts a hard ceiling on staleness, which matters when replicas cannot see each other's writes. expireAfterAccess removes entries that nobody reads, but a hot entry can live indefinitely. That makes it a memory-reclamation tool, not a freshness guarantee. For data that changes, prefer write-based expiry.

Refresh and loading

refreshAfterWrite marks an entry eligible for reload after the interval. The next read returns the current value immediately and triggers an asynchronous reload, so callers never wait on the backing store for a hot key. Refresh only fires on access, so pair it with a longer expireAfterWrite to evict entries that are never read again.

Building with a CacheLoader produces a LoadingCache. Concurrent get calls for the same missing key wait on a single load instead of each querying the database. This coalescing works within one process only. N replicas can still issue N loads.

recordStats exposes hit rate, load time and eviction counts through stats(). Without these numbers you cannot tell whether the cache is helping.

Copyimport com.github.benmanes.caffeine.cache.Caffeine;
import com.github.benmanes.caffeine.cache.LoadingCache;
import com.github.benmanes.caffeine.cache.stats.CacheStats;
import java.util.concurrent.TimeUnit;

public class ProductCache {
    private final LoadingCache<Long, Product> cache;

    public ProductCache(ProductRepository repository) {
        this.cache = Caffeine.newBuilder()
            // Bound heap use: evict once 10,000 entries are cached (W-TinyLFU policy)
            .maximumSize(10_000)
            // Hard staleness ceiling: no value older than 10 minutes is ever served
            .expireAfterWrite(10, TimeUnit.MINUTES)
            // After 1 minute, the next read returns the old value and reloads in the background
            .refreshAfterWrite(1, TimeUnit.MINUTES)
            // Collect hit rate, load time and eviction counts for stats()
            .recordStats()
            // The loader makes this a LoadingCache: concurrent misses on one key share one load
            .build(id -> repository.findById(id));
    }

    public Product get(long id) {
        // On a miss this blocks on the single in-flight load for this key
        return cache.get(id);
    }

    public void logStats() {
        CacheStats s = cache.stats();
        System.out.printf("hitRate=%.3f evictions=%d avgLoadMs=%.1f%n",
            s.hitRate(),
            s.evictionCount(),
            s.averageLoadPenalty() / 1_000_000.0); // averageLoadPenalty is in nanoseconds
    }
}

A bounded Caffeine LoadingCache. It caps entry count, enforces a 10-minute staleness ceiling with expireAfterWrite, serves the stale value while refreshAfterWrite reloads it in the background, and exposes hit rate, eviction and load-time stats.

Cache invalidation across replicas: TTLs, pub/sub and versioned keys

Each replica holds its own copy, so a write on one instance leaves the others serving the old value until something removes it. There are three common mechanisms, and each guarantees something different.

A TTL alone is simple and cannot fail silently, but it only puts an upper limit on how stale a read can be. Pub/sub broadcast closes most of that window. However, an instance that is restarting or reconnecting can miss an event and keep the old entry indefinitely, so keep a TTL as a backstop. Versioned keys, such as product:42:v7, make old entries unreachable instead of deleting them. That moves the problem to fetching the current version, which usually means a read against the database or a shared store on every request.

Ordering matters. If you evict before the transaction commits, a concurrent read can reload the old row and cache it again. Publish the invalidation only after commit, using an after-commit hook or an outbox table.

The listener in the accompanying example subscribes to an invalidation channel and calls invalidate(key) on the local Caffeine cache. The cache keeps expireAfterWrite configured, so a missed message costs one TTL of staleness rather than unbounded staleness.

Copyimport com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
import java.time.Duration;

// Transport-agnostic: wire this to whatever pub/sub client you use.
interface InvalidationBus {
    void subscribe(String channel, java.util.function.Consumer<String> onKey);
    void onReconnect(Runnable callback);
}

public final class LocalProductCache {
    // TTL is the safety net: a missed event costs at most 5 minutes of staleness.
    private final Cache<String, Product> cache = Caffeine.newBuilder()
            .maximumSize(10_000)
            .expireAfterWrite(Duration.ofMinutes(5))
            .build();

    public LocalProductCache(InvalidationBus bus) {
        // Writers publish the key only after their transaction commits,
        // so a reload triggered by this eviction sees the new row.
        bus.subscribe("cache-invalidation", key -> cache.invalidate(key));

        // Events sent while disconnected are lost; drop everything rather
        // than serve entries whose invalidation we may have missed.
        bus.onReconnect(cache::invalidateAll);
    }

    public Product get(String key, java.util.function.Function<String, Product> loader) {
        return cache.get(key, loader);
    }
}

record Product(long id, String name, long priceCents) {}

A Caffeine-backed local cache that evicts keys on post-commit invalidation messages, clears itself after a reconnect, and keeps expireAfterWrite as a backstop for missed events.

When to add a shared cache, and when to avoid caching entirely

Start by classifying the data, because the right tier depends on how much staleness each class can tolerate and how expensive it is to rebuild.

Cache correctness-critical data only when you have explicitly decided which stale reads the business can accept, and how stale they can be.

A shared cache is justified when values must look identical on every instance, when a computation is too expensive to repeat N times, or when the working set is large enough that copying it into every replica's heap wastes memory. It also protects the database during a deploy. Without it, every new instance starts cold and sends its misses to the database at the same moment.

A shared cache also adds a failure domain. Give cache calls a timeout shorter than the database query they replace, so a slow cache never costs more than a miss. Put a circuit breaker in front of the cache so a failing node stops stalling request threads. Then check that the database can handle the fallback. If it can only sustain the load because the cache absorbs most reads, falling back to it during a cache outage turns that outage into a database outage. In that case, shed load or serve degraded responses rather than passing every miss through to the database.

The two-tier pattern: near cache in front of a shared cache

A common alternative to choosing one tier is to run both: a small in-process cache on each replica, sitting in front of the shared cache, which sits in front of the database. The near cache handles the hottest keys at memory speed. The shared cache keeps the database from seeing a miss from every replica.

The read path

A request checks the local cache first. On a miss, it queries the shared cache. On a second miss, it reads from the database. Values are filled in on the way back. The database result is written to the shared cache, and then to the local cache, before it is returned. A hit in the shared cache fills only the local tier. Each replica therefore warms its own near cache from a store that other replicas have already populated.

Staleness compounds

Each tier has its own staleness window, and the windows add up. Suppose a value is written to the shared cache just before the underlying row changes. That value can stay there for the full shared TTL. A replica can then copy it into its local cache at the last moment and keep it for the full local TTL as well. In the worst case, total staleness is roughly the sum of the two TTLs.

To bound it, start from the inconsistency your product can tolerate. Set the local TTL to a small fraction of that window, because the local tier is the one you cannot inspect centrally.

Harder invalidation and debugging

A write now has to evict the key from the shared cache and also notify every replica to drop its local copy. That adds a second invalidation path, and the second path can fail silently. When a user reports a wrong value, it could have come from either tier on any replica. Reproducing the bug means knowing which replica served the request and what was in its local cache at that moment.

Copy  +---------------------------+        +---------------------------+
  | Replica A                 |        | Replica B                 |
  |   request                 |        |   request                 |
  |      | (1) get   ^ return |        |      | (1) get   ^ return |
  |      v           |        |        |      v           |        |
  |   +--------------------+  |        |   +--------------------+  |
  |   | local cache  TTL L |  |        |   | local cache  TTL L |  |
  |   +--------------------+  |        |   +--------------------+  |
  +------|-----------^--------+        +------|-----------^--------+
    (2)  | miss      | (5) fill local         |           |
         v           |                        v           |
  +----------------------------------------------------------------+
  |                    shared cache   TTL S                        |
  +----------------------------------------------------------------+
    (3)  | miss      | (4) fill shared
         v           |
  +----------------------------------------------------------------+
  |                          database                              |
  +----------------------------------------------------------------+

  write path:  write --> database
                 |--> evict key in shared cache        (path 1)
                 +--> notify EVERY replica: drop local (path 2,
                                               can fail silently)

  shared hit: skip (3)-(4), do (5) only
  worst-case staleness ~= S + L   -->  keep L well below the window

Two-tier read path: a request falls through the replica's local cache and the shared cache to the database, then fills shared and local tiers on the way back, so worst-case staleness approaches the sum of both TTLs and every write needs two invalidation paths.

Stampede protection and what to measure in production

Stampede protection has the same split between local and fleet behavior.

Coalescing, jitter and early refresh

Within one process, Caffeine's loading get coalesces concurrent misses for the same key: one thread loads while the others wait for that result. Across a fleet this guarantee shrinks. With N replicas, a hot key that expires can still produce N simultaneous database queries, one per instance. Usually that is acceptable. It becomes a problem when N is large or the query is expensive.

Jittered TTLs prevent many keys, or many replicas, from expiring on the same tick. Early refresh with refreshAfterWrite reloads an entry in the background while callers keep receiving the old value, so a hot key never goes fully cold.

Deploys and autoscaling erase all of this. Every new instance starts with an empty heap cache. A rolling deploy across the whole fleet is therefore a cold start that arrives in waves. Warm critical keys before an instance receives traffic, or ramp traffic to it gradually.

What to measure

• Hit ratio per cache, per tier and per instance
• Eviction counts, which show whether entries are pushed out by size limits or simply expire
• Load latency, the time a miss actually costs
• Heap usage and GC pause time
• Shared-cache error and timeout rates

A fleet-wide hit ratio can hide individual instances. If 95 percent of the fleet sits warm, a few freshly scaled replicas that are hammering the database barely move the average.

These numbers also show whether a shared cache is worth building. If local hit ratios are already high and misses are cheap, a shared tier adds a network hop and a new failure mode without improving anything. You can confirm that from data you already collect.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to monolith caching strategy, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.