What are Redis hot keys and big keys, and why do they stall a whole node?
A hot key is a key that receives a disproportionate share of a node's requests, such as a feature-flag hash read on every API call or a counter incremented by every worker. A big key is large in bytes (a multi-megabyte string) or in element count (a set, list, hash or sorted set with millions of members). The two often overlap, and both cause the same failure: one key consumes time that every other client is waiting for.
The cause is Redis's execution model. Commands run one at a time on a single main thread, even when network I/O is offloaded to other threads. A HGETALL on a hash with millions of fields, or a DEL of a large set, blocks every other command until it finishes. A hot key works more gradually. Each request is cheap, but together they saturate the one core that does all the work.
Redis Cluster does not solve this. Each key hashes to exactly one of 16,384 slots, and each slot belongs to one primary. Adding nodes moves other slots away, but the problem key's load stays on whichever shard owns its slot:
| Shard | Slots | Owns config:flags? |
CPU (illustrative) |
|---|---|---|---|
| A | 0–5460 | yes | ██████████ ~100% |
| B | 5461–10922 | no | ██ ~20% |
| C | 10923–16383 | no | ██ ~20% |
This section is for backend and platform engineers who see p99 latency spikes or one shard running much hotter than its peers. Identifying the specific key is the first step. It prevents spending time on bigger instances or resharding, neither of which reduces the cost of a single slot.
┌────────────────────────┐
│ app servers / workers │
└────────────┬───────────┘
│ every request: HGETALL config:flags
│ plus light traffic on other keys
▼
┌────────────────────────┐
│ slot = CRC16(key) │
│ mod 16384 │
│ config:flags → Shard A │
└────────────┬───────────┘
┌──────────────────┼──────────────────┐
▼ heavy ▼ light ▼ light
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Shard A │ │ Shard B │ │ Shard C │
│ 0–5460 │ │ 5461–10922 │ │ 10923–16383 │
│ config:flags │ │ │ │ │
│ main thread: │ │ main thread: │ │ main thread: │
│ [██████████] │ │ [██ ] │ │ [██ ] │
│ ~100% busy │ │ ~20% busy │ │ ~20% busy │
│ queue: ▓▓▓▓▓ │ │ queue: empty │ │ queue: empty │
└──────────────┘ └──────────────┘ └──────────────┘
other keys on A wait behind config:flags
┌──────────────┐
│ Shard D (new)│ ◄── receives some slots from A, B and C,
└──────────────┘ but config:flags' slot still has one owner
Because config:flags hashes to a single slot owned by Shard A, every request for it queues on A's one main thread while B and C stay mostly idle, and adding Shard D cannot split that slot's load (CPU figures are illustrative, not measured).
How do latency spikes and uneven shard load point to a problem key?
Two built-in tools narrow the search before you scan any keyspace.
SLOWLOG GET lists commands that exceeded the slowlog-log-slower-than threshold, along with their arguments. Look for O(N) commands such as HGETALL, SMEMBERS, LRANGE key 0 -1, or DEL against a large collection. The cost of each grows with the size of the collection, and Redis executes commands on a single main thread. One 200 ms HGETALL therefore delays every other client on that node for 200 ms.
Latency monitoring is off until you set latency-monitor-threshold. Once it is on, LATENCY LATEST and LATENCY HISTORY show when spikes happened and which event class caused them, such as command execution or fork. Match those timestamps against your deploys and traffic patterns.
A hot key that is also large often saturates the network before the CPU. Illustrative estimate, all inputs assumed: a 100 KB value read 10,000 times per second sends about 1 GB/s, roughly 8 Gbit/s. That is most of a 10 Gbit/s link. The main thread could still look only moderately busy at that point, while clients see timeouts.
| Symptom | Likely cause | Next check |
|---|---|---|
| Periodic multi-ms stalls for all clients | O(N) command on a big key | SLOWLOG GET |
| One cluster node's CPU far above its peers | Hot key in one slot | Per-node INFO commandstats |
| NIC near line rate, CPU moderate | Large values read frequently | Value size × read rate |
| Spikes during deletes or expiry | Synchronous free of a big key | LATENCY HISTORY, UNLINK usage |
#!/usr/bin/env bash
# Triage a node with uneven load. Host, port and key name are placeholders.
HOST=10.0.0.11
PORT=6379
R="redis-cli -h $HOST -p $PORT"
# Log commands slower than 10 ms (threshold is in microseconds).
$R CONFIG SET slowlog-log-slower-than 10000
# Show the 10 most recent slow entries with their arguments.
# Look for HGETALL, SMEMBERS, LRANGE key 0 -1, or DEL on large keys.
$R SLOWLOG GET 10
# Turn on latency monitoring (threshold is in milliseconds).
$R CONFIG SET latency-monitor-threshold 50
# Latest spike per event class, then the history for command execution.
$R LATENCY LATEST
$R LATENCY HISTORY command
# Per-command call counts and time on this node.
# Run on every cluster node and compare the totals.
$R INFO commandstats
# Current network throughput reported by Redis, in KB/s.
$R INFO stats | grep -E 'instantaneous_(input|output)_kbps'
# Approximate size of a suspected hot key, in bytes.
$R MEMORY USAGE user:session:12345
# Illustrative estimate (assumed inputs, not measured):
VALUE_KB=100 # assumed value size
READS_PER_SEC=10000 # assumed read rate
KB_PER_SEC=$(( VALUE_KB * READS_PER_SEC ))
echo "~${KB_PER_SEC} KB/s, about $(( KB_PER_SEC * 8 / 1000000 )) Gbit/s of a 10 Gbit/s link"
A redis-cli triage script that enables and reads SLOWLOG and latency monitoring, collects per-node command statistics and network throughput, checks the size of a suspect key, and computes the illustrative value-size × read-rate bandwidth estimate.
How do you find Redis big keys without hurting production?
Never run KEYS * on a busy primary. Redis executes commands on a single thread, so KEYS blocks every other client until it has walked the whole keyspace. On a node with tens of millions of keys, that is a self-inflicted outage. DEBUG OBJECT and scripts that call MEMORY USAGE ... SAMPLES 0 across every key are also unsafe: each call fully inspects a value, and doing that to the large values you are hunting for causes the same latency spikes you are diagnosing.
The safer live tools are built on SCAN, which iterates in small batches between other commands:
redis-cli --bigkeysreports the largest key per type by length or element count. This is cheap, but a hash with few huge fields can rank low.redis-cli --memkeysreports per-key memory usage, which is closer to what you care about. It costs more per key.MEMORY USAGE <key>on a short list of candidates confirms actual byte size. It samples nested elements by default, so treat the result as an estimate.
These scans still add CPU and network load. Use the -i interval option to throttle them. SCAN also has weak guarantees: keys created, deleted or resized during the iteration may be missed or reported more than once, so one pass is a snapshot, not an inventory.
Where possible, point scans at a replica that serves no client reads. It holds the same data and takes the extra load instead of the primary. In a cluster, run the scan against each node.
The lowest-risk option is offline analysis. Copy an existing RDB backup, or one produced on a replica, to another machine and parse it there. This gives exact per-key sizes with zero load on production. The data will be as old as the snapshot.
#!/usr/bin/env bash
# Find big keys while keeping load off the primary.
set -euo pipefail
HOST=10.0.2.15 # a replica that serves no client reads
PORT=6379
export REDISCLI_AUTH='example-password' # placeholder; keeps it out of argv
# 1. Refuse to run against a primary.
ROLE=$(redis-cli -h "$HOST" -p "$PORT" ROLE | head -n 1)
if [ "$ROLE" != "slave" ]; then
echo "Refusing to scan: $HOST:$PORT reports role '$ROLE'" >&2
exit 1
fi
# 2. Cheap pass: largest key per type by length/element count.
# -i 0.1 sleeps 0.1s every 100 SCAN calls to throttle the load.
redis-cli -h "$HOST" -p "$PORT" --bigkeys -i 0.1
# 3. Costlier pass: per-key memory usage, closer to real impact.
redis-cli -h "$HOST" -p "$PORT" --memkeys -i 0.1
# 4. Confirm a short list of candidates. Default sampling makes this
# an estimate; avoid SAMPLES 0 on huge values.
for KEY in 'session:index' 'feed:user:42' 'cart:pending'; do
BYTES=$(redis-cli -h "$HOST" -p "$PORT" MEMORY USAGE "$KEY")
echo "$KEY -> ${BYTES:-missing} bytes"
done
# 5. Lowest-risk option: pull an RDB snapshot from the replica and
# analyse it on another machine with an RDB parser.
redis-cli -h "$HOST" -p "$PORT" --rdb /var/tmp/replica-dump.rdb
# Never run on a busy primary: KEYS '*' (blocks the server) or a
# loop of DEBUG OBJECT / MEMORY USAGE ... SAMPLES 0 over every key.
A throttled scan script that checks the target is a replica, runs --bigkeys and --memkeys with -i, confirms candidates with MEMORY USAGE, and downloads an RDB snapshot for offline analysis.
How do you detect Redis hot keys reliably?
redis-cli --hotkeys walks the keyspace with SCAN and calls OBJECT FREQ on each key. That command returns the access-frequency counter stored in each key's object header, and Redis only maintains that counter as a frequency when maxmemory-policy is allkeys-lfu or volatile-lfu. Under any other policy the same bits hold an LRU timestamp, so OBJECT FREQ returns an error and --hotkeys has nothing to report.
Changing the policy only to enable detection has a cost. Once memory reaches maxmemory, LFU evicts keys that are accessed rarely rather than keys that were touched least recently. A cache that depends on recency will behave differently. The LFU counter is also logarithmic and decays over time, so --hotkeys ranks keys approximately and does not give exact request rates.
MONITOR streams every command the server executes. That makes it useful for a sampling window of a few seconds, during which you count keys offline. Every command must be formatted and written to the monitoring client, so throughput drops noticeably while it runs. Never leave it attached.
The most reliable data comes from the request path. A client wrapper or proxy can count keys per time window, or sample a fraction of requests. It sees real traffic per key and costs the Redis server nothing.
| Method | Accuracy | Server overhead | Prerequisites |
|---|---|---|---|
redis-cli --hotkeys |
Approximate ranking | Full keyspace scan | LFU eviction policy |
MONITOR sampling |
Exact for the window | High while attached | Short windows only |
| Client/proxy counting | Exact or sampled, per key | None | Instrumented clients or a proxy |
#!/usr/bin/env bash
# Hot-key detection on a single Redis node. Run against a replica or
# during low traffic where possible.
set -euo pipefail
HOST=10.0.0.12 # placeholder address
PORT=6379
# 1. --hotkeys only works under an LFU policy. Record the current policy
# so you know what you are changing (and can revert it).
redis-cli -h "$HOST" -p "$PORT" CONFIG GET maxmemory-policy
# Switching changes eviction behavior from recency to frequency.
redis-cli -h "$HOST" -p "$PORT" CONFIG SET maxmemory-policy allkeys-lfu
# SCAN + OBJECT FREQ over the whole keyspace. Gives an approximate ranking.
redis-cli -h "$HOST" -p "$PORT" --hotkeys
# Spot-check one key's logarithmic, decaying LFU counter.
redis-cli -h "$HOST" -p "$PORT" OBJECT FREQ user:42:profile
# 2. MONITOR sample: hard-capped at 5 seconds because MONITOR slows
# the server while it is attached. `|| true` keeps set -e from
# aborting on timeout's non-zero exit status.
# Line format: <ts> [<db> <client>] "CMD" "key" ...
# $4 = command, $5 = first argument (usually the key)
timeout 5 redis-cli -h "$HOST" -p "$PORT" MONITOR > /tmp/monitor.sample || true
# Count requests per key offline and show the top 20.
awk '$4 ~ /^"/ && NF >= 5 { print $5 }' /tmp/monitor.sample \
| sort | uniq -c | sort -rn | head -20
A shell session that switches to allkeys-lfu so redis-cli --hotkeys and OBJECT FREQ can report access frequency, then takes a strictly time-limited MONITOR sample and counts requests per key offline.
How do you fix big keys: safe deletion and restructuring
Deleting a big key with DEL frees every element synchronously on the main thread, so removing a multi-million-field hash can block all other clients for as long as the free takes. Use UNLINK instead. It removes the key from the keyspace immediately and reclaims large values on a background thread. Several deletions happen implicitly, so also review the lazy-free settings:
lazyfree-lazy-user-delmakes plainDELbehave likeUNLINK.lazyfree-lazy-expireandlazyfree-lazy-evictioncover expiry and eviction.lazyfree-lazy-server-delcovers server-side overwrites such asRENAMEonto an existing key.
Freed memory is not available right away. The background thread needs time to finish, and the allocator may keep freed pages instead of returning them to the operating system. A node close to maxmemory can keep evicting or rejecting writes for a while after a large UNLINK.
Stop reading whole collections. Replace HGETALL, SMEMBERS and unbounded ZRANGE or LRANGE with HSCAN, SSCAN and ZSCAN, or with bounded ranges and cursor-style pagination. COUNT is only a hint, so callers must handle variable batch sizes.
To restructure a large hash, split it into buckets. Compute a stable hash of the field name, take it modulo N, and write to user:123:b{n}. Every reader and writer must use the same function and the same N. Changing N later means migrating data.
Keep large blobs in object storage or a database, and store only a reference in Redis.
Bucketing has a cost in Cluster. Buckets land on different slots, so multi-key commands, transactions and Lua scripts across them fail. Hash tags such as {user:123}:b7 restore atomicity, but they put every bucket back on one slot and one node. That fixes per-command latency but does not spread the load.
How do you spread load from a Redis hot key?
A hot key is a throughput problem, not a size problem: one slot, one node, one thread serving every request. The fix is to give that traffic more than one place to land.
Read-hot keys
Replicate across suffixed keys. Write the value to N copies, config:flags:0 through config:flags:7, and have each client read one at random. Without hash tags, the copies hash to different slots and usually different nodes, so per-node load drops roughly by N. Writes now cost N commands, and readers can briefly see mixed versions during an update. This suits values that change rarely.
Cache in-process with a short TTL. Even a one-to-five-second local cache means each application instance hits Redis at most once per TTL instead of once per request. You accept bounded staleness in exchange.
Read from replicas. This adds capacity without changing key names. Replication is asynchronous, so these reads can lag the primary.
Write-hot keys
Shard the counter. Split it into sub-keys such as views:42:0 through views:42:15, increment a random one, and sum them on read with pipelined GETs. In cluster mode a plain MGET across slots fails. Reads get more expensive, and writes spread out.
Batch client-side. Accumulate increments in memory and flush them with one INCRBY every few hundred milliseconds. A crash loses the unflushed delta.
Why hash tags work against you
Every key that shares a {...} tag maps to the same slot. If you wrap suffixed copies or counter shards in a tag, you recreate the hot key. Use tags only where multi-key atomicity justifies that concentration.
The example below shows suffixed read copies and a sharded counter.
When should you avoid these fixes, and what are the alternatives?
Each fix removes one bottleneck and introduces a different cost. Choose the fix whose cost your workload can absorb.
| Fix | What it costs | Avoid it when |
|---|---|---|
| Suffixed read copies | Memory and write fan-out grow with the copy count. Copies can disagree briefly during an update. | The key is written often or readers need read-your-writes consistency |
| Local in-process cache | Stale reads, invalidation logic, and thundering-herd reloads when many TTLs expire at once | The data changes faster than your staleness tolerance |
| Replica reads | Stale reads, because replication is asynchronous | The key is write-hot. Replicas only absorb reads. |
| Sharded counters | Each read must fetch and sum every shard, and there is no atomic read-modify-write across shards | You need exact, atomic checks such as quota or inventory enforcement |
Sometimes you should not change the data model at all. If the hot key comes from a traffic pattern, the key itself is fine and the demand is the problem. A viral post and a client retry storm are typical cases. Restructuring the key leaves the extra requests in place and adds permanent complexity. Application-layer rate limiting caps the load. Request coalescing lets one in-flight fetch serve all concurrent callers asking for the same key. Both fixes target the cause and stay inactive under normal traffic.
Development cost also matters. Every fix adds code paths, failure modes, and things to monitor: copy fan-out, invalidation channels, shard aggregation. Measure the per-key cost first, using request rate, latency contribution, and memory footprint. Then apply the smallest fix that those numbers justify.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to redis hot keys, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.