How to Reduce Redis Memory Cost: Expiry, Compression, Right-Sizing

Where does Redis memory actually go?

Before touching instance types or codecs, measure what is consuming memory. Data volume and retention drive the bill far more than the per-GB price of a node, so the baseline tells you where engineering effort will pay off.

Start with INFO memory. Three fields matter most:

• used_memory: bytes Redis has allocated for data, data structures, and internal buffers. This is your logical footprint.
• used_memory_rss: memory the operating system reports as resident for the process. This is what you actually pay for.
• mem_fragmentation_ratio: roughly RSS divided by used_memory. A ratio well above 1 means the allocator holds pages it cannot reuse efficiently.

RSS often stays high after the dataset shrinks. Freed memory is scattered across partially used pages that the allocator does not return to the OS. A post-cleanup measurement can therefore understate your savings until fragmentation is reclaimed through active defragmentation or a restart or failover.

Next, find out which keys hold the bytes:

• redis-cli --bigkeys reports the largest key per data type by element count.
• --memkeys ranks keys by actual memory usage.
• MEMORY USAGE on individual sampled keys gives per-key byte counts, including overhead.

Both scanning modes iterate with SCAN, but run them against a replica where possible.

The most useful view comes from grouping keys by prefix, such as session:, cache:product:, or ratelimit:. Sample keys with SCAN, sum MEMORY USAGE per prefix, and multiply by each prefix's key count. Typically a few patterns dominate. Those patterns are where expiry and compression work belongs.

Copy# Estimate memory per key prefix. Run against a replica, not the primary.
import random
from collections import defaultdict
import redis

r = redis.Redis(host="redis-replica.internal", port=6379)

# Baseline: logical footprint vs. what the OS actually holds resident.
mem = r.info("memory")
print("used_memory:", mem["used_memory_human"])
print("used_memory_rss:", mem["used_memory_rss_human"])
print("mem_fragmentation_ratio:", mem["mem_fragmentation_ratio"])

SAMPLE_RATE = 0.05  # call MEMORY USAGE on ~5% of keys to limit load
counts = defaultdict(int)
sampled_bytes = defaultdict(int)
sampled_n = defaultdict(int)

# SCAN walks the keyspace incrementally instead of blocking like KEYS.
for key in r.scan_iter(count=1000):
    name = key.decode("utf-8", "replace")
    # Group by the first two segments, e.g. "cache:product:" or "session:".
    parts = name.split(":")
    prefix = ":".join(parts[:2]) + ":" if len(parts) > 2 else parts[0] + ":"
    counts[prefix] += 1
    # Always sample the first key of a prefix so small groups still get an estimate.
    if sampled_n[prefix] == 0 or random.random() < SAMPLE_RATE:
        size = r.memory_usage(key, samples=0)  # samples=0: measure all nested values
        if size is not None:  # key may have expired between SCAN and MEMORY USAGE
            sampled_bytes[prefix] += size
            sampled_n[prefix] += 1

# Extrapolate: average sampled size x total key count per prefix.
estimates = {
    p: (sampled_bytes[p] / sampled_n[p]) * counts[p]
    for p in counts if sampled_n[p]
}
for prefix, est in sorted(estimates.items(), key=lambda kv: -kv[1])[:15]:
    print(f"{prefix:<30} keys={counts[prefix]:>10}  est={est / 1024**2:>10.1f} MiB")

A Python script using redis-py that prints the key INFO memory fields, then scans the keyspace, samples MEMORY USAGE, and extrapolates estimated memory per key prefix to show which patterns dominate.

Why do key expiry and per-key overhead matter more than instance price?

Encoding changes pay off before any instance change does, and they cost nothing to try.

Find keys that never expire

INFO keyspace reports, per database, keys (total) and expires (keys with a TTL). A large gap between them means data that stays in memory until something deletes it or evicts it. Session blobs and cached API responses usually have no reason to live forever. Set TTLs that match how long the data stays useful, and add random jitter (for example, base TTL ±10%) so a batch written together does not expire together and send a burst of misses to the backend.

Pick maxmemory-policy deliberately. The volatile-* policies evict only keys that have a TTL. If most keys have none, Redis has little it can evict and starts rejecting writes once it reaches the limit. The allkeys-* policies can evict any key, so they are only safe when every key can be rebuilt from another source.

Cut per-key overhead

Every top-level key carries fixed metadata: a dictionary entry, an object header, and an expiry entry if it has a TTL. For small values, this overhead can be larger than the data itself. If you pack related fields into one hash, such as user:42 with fields instead of user:42:name and user:42:email, the hash can use a compact listpack encoding. The thresholds are hash-max-listpack-entries and hash-max-listpack-value (named *-ziplist-* on older versions).

Trade-offs

• Short TTLs raise miss rates and backend load.
• Encoding gains stop without warning. Once a hash exceeds either threshold, it converts to a regular hash table. Check the result with OBJECT ENCODING.
• Expiry granularity gets coarser. A TTL normally applies to the whole hash. Per-field expiry is not available on every version.

Copy#!/usr/bin/env bash
set -euo pipefail
R=(redis-cli -h 127.0.0.1 -p 6379)

# 1. Per database, compare total keys with keys that have a TTL.
#    A large no_ttl count means data that never leaves memory on its own.
"${R[@]}" INFO keyspace | tr -d '\r' | awk -F'[:=,]' '/^db/ {
  printf "%s keys=%s expires=%s no_ttl=%d\n", $1, $3, $5, $3 - $5 }'

# 2. Check the eviction policy. volatile-* only evicts keys with a TTL.
"${R[@]}" CONFIG GET maxmemory-policy

# 3. Write a session with a base TTL of 1 hour, +/-10% random jitter,
#    so keys written together do not all expire together.
base=3600
jitter=$(( RANDOM % (base / 5 + 1) - base / 10 ))
"${R[@]}" SET session:abc123 '{"cart":[]}' EX $(( base + jitter ))

# 4. Pack related fields into one hash instead of one key per field.
"${R[@]}" HSET user:42 name "Ada" email "ada@example.com"
"${R[@]}" EXPIRE user:42 86400   # the TTL applies to the whole hash

# 5. Confirm that the compact encoding is in use: expect "listpack"
#    ("ziplist" on versions before 7.0). "hashtable" means a threshold was exceeded.
"${R[@]}" OBJECT ENCODING user:42
"${R[@]}" CONFIG GET 'hash-max-listpack-*'
"${R[@]}" MEMORY USAGE user:42

A redis-cli audit that finds keys without TTLs, checks the eviction policy, writes a TTL with jitter, and confirms that a small hash uses the compact listpack encoding.

How does client-side Redis compression work, and when does it pay off?

Redis never sees the compression. The application compresses the serialized value before SET and decompresses after GET, so the server stores opaque bytes and needs no module or configuration change. The memory saving shows up directly in used_memory. The cost is CPU time on your app hosts, which are usually cheaper to scale than cache nodes.

When it pays off

Compression works by exploiting repetition. Large JSON documents with repeated field names, rendered HTML fragments, and verbose serialized objects often shrink substantially. It does nothing useful for:

• Small values. Codec headers and per-key overhead dominate, and the output can be larger than the input.
• Images, video and archives. These are already compressed.
• Encrypted blobs. Ciphertext looks random, and random data does not compress.

Threshold and marker byte

The wrapper needs two rules:

1. Skip small values. Below a threshold, such as 1 KB (tune it against your own payloads), store the raw bytes. Also store the raw bytes if compressing did not actually make the value smaller.
2. Prefix a marker byte. Prepend one byte that records how the value was encoded, for example 0x00 for raw and 0x01 for codec A. On read, check this byte first and dispatch to the matching decoder.

The marker lets you switch codecs later without flushing the cache. Old and new formats can coexist until the old entries expire.

SET path: value → size < threshold? → raw → [0x00|bytes] / else compress → [0x01|bytes] GET path: bytes → read byte[0] → 0x00: return rest | 0x01: decompress rest | unknown: treat as miss

Treat an unknown marker as a cache miss rather than an error. A rollback to an older client then degrades to extra backend reads instead of failed requests.

Copyimport zlib

RAW = b"\x00"         # marker: value stored uncompressed
ZLIB = b"\x01"        # marker: value compressed with zlib
THRESHOLD = 1024       # bytes; tune against your own payloads

def encode(data: bytes) -> bytes:
    # Small values: codec overhead outweighs any saving, so store raw.
    if len(data) < THRESHOLD:
        return RAW + data
    packed = zlib.compress(data, 6)
    # Incompressible input (images, ciphertext) can grow, so keep raw.
    if len(packed) >= len(data):
        return RAW + data
    return ZLIB + packed

def decode(stored: bytes | None) -> bytes | None:
    if not stored:
        return None                       # key absent or empty: miss
    marker, body = stored[:1], stored[1:]
    if marker == RAW:
        return body
    if marker == ZLIB:
        return zlib.decompress(body)
    # Unknown marker (e.g. written by a newer client): treat as a miss
    # so a rollback causes backend reads, not failed requests.
    return None

# Usage with redis-py; Redis only ever sees opaque bytes.
# r = redis.Redis(host="cache.internal", port=6379)
# r.set("page:42", encode(html.encode()), ex=3600)
# value = decode(r.get("page:42"))
# if value is None:
#     value = load_from_backend(42)   # miss path

A minimal Python encode/decode wrapper that skips compression below a size threshold or when it doesn't help, prefixes a marker byte, and treats unknown markers as cache misses.

Which compression codec should you choose?

Codecs fall into three practical classes, and the right one depends on whether latency or memory is the constraint you are paying for.

Dictionaries matter because a generic codec has little history to exploit in a few hundred bytes. A dictionary trained on representative values supplies that shared structure up front. The cost is that the dictionary must be versioned alongside your format marker, because a value compressed with one dictionary can only be read with that same dictionary.

Do not choose a codec from published ratios. Compression ratios depend on your data, so pull a sample of real values from production. Measure the ratio, compress time and decompress time at each level, and look at p99 as well as the mean.

Compression also moves cost elsewhere:

• CPU and latency shift to the app tier. You are trading cache memory for application CPU, which you also pay for.
• Values become opaque to Redis. GETRANGE, APPEND, partial reads and Lua scripts that inspect content no longer work on compressed keys.
• Debugging gets harder. redis-cli GET returns binary, so on-call engineers need a decode tool.

Illustrative estimate: how much can expiry and compression shrink a 40 GB dataset?

Illustrative estimate (assumed numbers, not a benchmark). None of these figures were measured. Replace them with values from INFO memory and a key-space sample from your own cluster.

Assumed inputs:

• 40 GB used memory
• 70% of it in JSON values averaging 4 KB
• 3:1 compression ratio on those values
• 20% of keys stale with no TTL

The order of the steps matters. Do expiry first, so you don't spend CPU compressing data you are about to delete.

The general formula is:

Target = M × (1 − s) × ((1 − j) + j ÷ r) ÷ (1 − h)

The estimate depends on these simplifying assumptions:

• Staleness is spread evenly across key types. If most stale keys are JSON, the compression savings shrink, because there is less JSON left to compress.
• The 70% is all compressible payload. It is not. Key names, dictionary entries and allocator rounding remain uncompressed. That makes 17.1 GB an optimistic floor.
• 3:1 applies uniformly. At a 4 KB average, nearly every value clears a typical minimum-size threshold, so this is plausible. Small or already-dense values will compress far less.

Headroom is applied by dividing by (1 − h) rather than adding a flat 25%. That way the post-change dataset fills only 75% of the target size, which leaves the remainder for fragmentation, replication and client buffers, and growth.

How should you size a smaller ElastiCache node beyond peak memory?

Peak used_memory describes the dataset. It does not describe the process under stress. Memory fragmentation inflates resident size beyond what Redis reports as used. Replication backlogs and client output buffers grow when replicas or pub/sub consumers fall behind. During BGSAVE or a full resync, the forked child shares pages with the parent until writes touch them. Each page a write touches is copied, so write-heavy workloads can briefly need much more than the dataset size. A node sized on peak memory alone tends to hit eviction or OOM at exactly these moments.

ElastiCache also sets aside reserved memory for overhead outside the dataset, so usable capacity is lower than the node's nominal size. Node families differ in their memory-to-vCPU and memory-to-network ratios. A smaller node can fit the data and still saturate its network allowance or its main thread, because command execution is largely single-threaded. For large datasets with a cold tail, data tiering or memory-optimized families may cost less than scaling RAM. Reserved pricing changes the comparison too, because a committed larger node can undercut an on-demand smaller one. Confirm current specs, reserved-memory parameters and pricing in the provider's documentation before deciding.

What is a safe rollout sequence, and when should you avoid downsizing?

Change one variable at a time so each metric shift has a clear cause:

1. Baseline. Record used_memory, used_memory_rss, hit rate, evicted_keys, p50/p99 latency, CPU and network for at least one full traffic cycle.
2. Add TTLs and let expiry reach a steady state.
3. Ship dual-read code that accepts compressed and uncompressed values before any writer compresses.
4. Migrate gradually, starting with the largest values, by key prefix or percentage.
5. Re-measure, including RSS. Fragmentation can hide part of the logical savings.
6. Resize last, when there is headroom left over.

Watch evictions, latency percentiles, CPU and network at every step. Rollback is the main risk. Without a format marker such as a prefix byte or header, old readers cannot tell compressed bytes from raw ones. A reverted deploy then returns garbage until those keys expire.

Signs you went too far

• evicted_keys rising or hit rate falling
• p99 latency climbing
• EngineCPU or host CPU reaching its ceiling
• Swap or OOM during snapshots, when fork copy-on-write needs extra memory

When to avoid it

Skip compression on hot paths with small values, where codec overhead exceeds the savings. Keep headroom for bursty workloads. Avoid repeated resizes, since each one moves data and may depend on maintenance windows.

Alternatives

Consider these options instead:

• Scale out with sharding instead of up.
• Fix bloated data models.
• Move cold data to a cheaper store.

Teams with large, compressible, read-heavy values and steady traffic gain the most from this approach.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reduce redis memory cost, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.