Why doesn't Redis Cluster key sharding fix a hot key?

A hot key is a single key that receives a disproportionate share of requests, enough that the node holding it runs out of CPU or network capacity while the rest of the cluster sits mostly idle. Redis Cluster cannot spread that load, because sharding works at the granularity of a key, not a request.

How keys map to nodes

Every key is assigned to one of 16384 hash slots by computing CRC16 of the key and taking the result modulo 16384. Each slot is owned by exactly one primary at any moment. If the key contains a hash tag (a non-empty substring between the first { and the next }), only that substring is hashed. The mapping is deterministic, so every GET, INCR or HSET against product:42 resolves to the same slot and therefore the same primary. Going from 3 shards to 30 changes which node owns that slot. It does not change the fact that one node owns it.

Hash tags concentrate load

Hash tags exist so that multi-key operations, transactions and Lua scripts can touch related keys in one slot. The cost is that tagging {user:42}:profile, {user:42}:cart and {user:42}:session together puts all of that user's traffic on one node. A popular tag turns several warm keys into one hot slot.

Who gets hit, and why scaling out fails

The usual victims are high-traffic paths: a global rate-limit counter, a feature-flag blob read on every request, or a celebrity's timeline. Resharding moves whole slots, and a slot cannot be subdivided below one key. Adding nodes only relocates the bottleneck to another node.

 Clients: many app servers, every request names a key
 GET product:42      GET {user:42}:profile      GET {user:42}:cart
        |                     |                           |
        v                     v                           v
 +-----------------------------------------------------------+
 | slot = CRC16(key or {hash tag}) mod 16384                 |
 | product:42         hashes "product:42" -> slot A          |
 | {user:42}:profile  hashes "user:42"    -> slot B          |
 | {user:42}:cart     hashes "user:42"    -> slot B          |
 +-----------------------------------------------------------+
        |                       |
        | every product:42 req  | every {user:42} req
        v                       v
 +----------------+      +----------------+      +----------------+
 | Primary 1      |      | Primary 2      |      | Primary 3      |
 | owns slot A    |      | owns slot B    |      | other slots    |
 | CPU/NIC: HOT   |      | CPU/NIC: HOT   |      | mostly idle    |
 +----------------+      +----------------+      +----------------+

 Scale out: add Primary 4 and reshard slot A onto it
 +----------------+   move slot A    +----------------+
 | Primary 1      | ---------------> | Primary 4      |
 | now idle       |  whole slot; a   | owns slot A    |
 |                |  key can't split | CPU/NIC: HOT   |
 +----------------+                  +----------------+

CRC16 maps each key, or its hash tag, to one slot owned by one primary, so a hot key or popular tag saturates a single node, and resharding only moves that hot slot to a different node.

Is it a hot key, a big key or a hot slot?

An overloaded shard has three common causes, and each needs a different fix. Before you split anything, work out which one you have.

A hot key is one key receiving a disproportionate share of operations. Each request may be cheap, such as a GET or INCR, but the volume saturates one node's single command-processing thread. The node's CPU climbs while its bytes per command stay small.

A big key puts its cost into each request rather than into the request rate. A SMEMBERS on a set with hundreds of thousands of members, or an unbounded LRANGE 0 -1, blocks the event loop while the reply is built and then fills the network buffers. The node can show modest ops/sec alongside high latency, slowlog entries and heavy outbound traffic. Splitting this key into copies only multiplies the expensive reads.

A hot slot has no single culprit. Many moderately busy keys hash to the same slot, often because a broad hash tag like {tenant:7} pins a whole tenant's data together. No individual key looks unusual, but their combined load lands on one node.

Cause Telltale signal Fix it needs
Hot key High ops/sec concentrated on one key, small replies Key splitting, replica reads, client-side caching
Big key Slowlog entries, high bytes out, latency spikes at moderate ops/sec Data model change: paginate with SSCAN/bounded ranges, bucket the structure
Hot slot Load spread across many keys sharing one slot Narrow or remove hash tags, move the slot to a less loaded node

How do you detect redis hot keys on the server without hurting production?

Start by localizing the problem to a node, then narrow it to a key. Compare INFO commandstats across every primary: one node with much higher calls and usec than its peers, together with higher CPU and network egress, points to a hot shard. The work is cheap because each node returns counters it already maintains. Reset them with CONFIG RESETSTAT before a measurement window so all nodes cover the same interval.

On the suspect node, redis-cli --hotkeys walks the keyspace with SCAN and calls OBJECT FREQ on each key. Both rely on the LFU access counter, so the node must run an LFU maxmemory-policy (allkeys-lfu or volatile-lfu). Under any other policy the command returns an error. The counter is logarithmic and decays over time, so read it as a relative rank, not a request count. A full scan adds one extra command per key, so on a large keyspace it is better to run OBJECT FREQ only on keys you already suspect. In Redis Cluster, --hotkeys scans only the node it connects to.

MONITOR streams every command the node executes. That gives you exact data, but on a busy node it can significantly reduce throughput. Keep sessions to a few seconds, sample from one client, and stop the session before you analyze the output.

Method What it tells you Cost on the node Prerequisite
INFO commandstats, CPU, network Which shard is hot Negligible None
redis-cli --hotkeys Ranked keys on one node Full SCAN + per-key call LFU policy
OBJECT FREQ <key> Frequency of one key Negligible LFU policy
MONITOR Exact command stream High; keep it brief None
#!/usr/bin/env bash
# Localize a hot shard cheaply, then narrow to keys on that node only.
set -euo pipefail
SEED="10.0.1.10:6379"             # any cluster node (placeholder)
WINDOW=60                          # seconds per measurement window
# export REDISCLI_AUTH=...         # password via env, not on the command line

# 1. List primaries from CLUSTER NODES (address field is ip:port@cport).
PRIMARIES=$(redis-cli -h "${SEED%:*}" -p "${SEED#*:}" CLUSTER NODES \
  | awk '$3 ~ /master/ {split($2, a, "@"); print a[1]}')

# 2. Reset counters on every primary so all nodes cover the same interval.
for n in $PRIMARIES; do redis-cli -h "${n%:*}" -p "${n#*:}" CONFIG RESETSTAT >/dev/null; done
sleep "$WINDOW"

# 3. Compare total calls and usec per primary; one outlier = hot shard.
for n in $PRIMARIES; do
  redis-cli -h "${n%:*}" -p "${n#*:}" INFO commandstats | tr -d '\r' \
    | awk -F'[=,]' -v node="$n" '/^cmdstat_/ {c += $2; u += $4}
        END {printf "%-22s calls=%d usec=%d\n", node, c, u}'
done | sort -t= -k2 -nr

# 4. On the suspect node: OBJECT FREQ needs an LFU policy, so check first.
HOT="10.0.1.12:6379"               # chosen from the output above
H="${HOT%:*}"; P="${HOT#*:}"
redis-cli -h "$H" -p "$P" CONFIG GET maxmemory-policy

# Prefer targeted checks over a full --hotkeys scan on large keyspaces.
for k in "session:{42}" "product:{1001}:stock"; do
  echo "$k freq=$(redis-cli -h "$H" -p "$P" OBJECT FREQ "$k")"  # relative rank, not a count
done

# 5. Last resort: a 5-second MONITOR sample, stopped before analysis.
timeout 5 redis-cli -h "$H" -p "$P" MONITOR > /tmp/monitor.txt || true
awk 'NF >= 5 {print $4, $5}' /tmp/monitor.txt | sort | uniq -c | sort -nr | head -20

A bash workflow that resets and compares INFO commandstats across all cluster primaries to find the hot shard, then checks the LFU policy, runs OBJECT FREQ on suspect keys, and takes a brief time-boxed MONITOR sample on that node only.

How can clients and proxies find hot keys before a node saturates?

Server-side tools only see one node at a time, and most need either a full keyspace scan or an expensive command stream. The client already sees every key it sends, so it is a better place to count. Add a hook to your client library's command pipeline, or to the proxy if all traffic passes through one, that records the key of each command.

Counting every key exactly is too expensive. A hash map with one entry per distinct key grows with your keyspace and adds a lookup and allocation to every request. Two techniques keep the cost bounded:

  • Sampling. Record about 1 in 100 commands. A key hot enough to saturate a shard still appears often in the sample. Keys you miss are, by definition, not your problem.
  • Top-k sketch. Feed the sampled keys into a fixed-size structure, such as Space-Saving or a Count-Min Sketch paired with a small heap. It tracks the heaviest hitters in constant memory and gives approximate counts.

Reset or decay the sketch on a short window, for example every 10 to 60 seconds, so the counts reflect current traffic and not history.

Be careful when you export. If you emit one time series per key, your metrics backend inherits the cardinality problem. Export only the top 10 or 20 keys per window, either as a capped label set or as a periodic log line. Tag each entry with the slot number so you can map it to a shard.

This saves time in two ways. You see a key climbing before its node's CPU is pinned. And application teams can diagnose the problem from their own dashboards, without asking for server access or running MONITOR during an incident.

import random, threading

SAMPLE_RATE = 0.01  # record ~1 in 100 commands; hot keys still show up
CAPACITY = 200      # Space-Saving counters: memory is fixed, not keyspace-sized
TOP_N = 20          # export only this many keys per window (caps cardinality)

def key_slot(key: bytes) -> int:
    # Redis Cluster slot: CRC16-XMODEM of the key (or its {hash tag}) mod 16384
    s = key.find(b"{")
    if s != -1 and (e := key.find(b"}", s + 1)) > s + 1:
        key = key[s + 1:e]
    crc = 0
    for b in key:
        crc ^= b << 8
        for _ in range(8):
            crc = ((crc << 1) ^ 0x1021 if crc & 0x8000 else crc << 1) & 0xFFFF
    return crc % 16384

class HotKeyTracker:
    def __init__(self):
        self.counts, self.lock = {}, threading.Lock()

    def record(self, key: bytes):  # call from your client/proxy command hook
        if random.random() >= SAMPLE_RATE:
            return  # fast path for ~99% of commands: no lock, no allocation
        with self.lock:
            if key in self.counts or len(self.counts) < CAPACITY:
                self.counts[key] = self.counts.get(key, 0) + 1
            else:  # Space-Saving: evict the smallest counter, inherit its count
                victim = min(self.counts, key=self.counts.get)
                self.counts[key] = self.counts.pop(victim) + 1

    def flush(self):  # run every 10-60 s; the reset keeps counts current
        with self.lock:
            snapshot, self.counts = self.counts, {}
        top = sorted(snapshot.items(), key=lambda kv: -kv[1])[:TOP_N]
        for key, n in top:  # one log line per key; scale sample count back up
            print(f"hotkey key={key!r} slot={key_slot(key)} est_ops={int(n / SAMPLE_RATE)}")

A client-side tracker that samples about 1% of commands into a fixed-size Space-Saving sketch. Each window it emits only the top 20 keys, tagged with their cluster slot so you can map them to a shard.

How do you split a hot key across cluster shards?

For a read-heavy key, write N copies under distinct names (product:42:0 through product:42:N-1) and have each read pick one, either at random or by hashing the client ID so a given client keeps hitting the same copy. Run CLUSTER KEYSLOT on every name, then map those slots to primaries with CLUSTER SLOTS. Different slots can still live on the same primary, so check placement against primaries, not slots. Avoid a hash tag that wraps the shared part, such as {product:42}:3, because it forces every copy into one slot. Tagging only the suffix (product:42:{3}) spreads the copies, but copy 3 of every split key then shares one slot.

Illustrative estimate, assuming 120,000 reads/s and 1,000 writes/s on one key across 6 primaries:

N Reads/s per copy Writes/s cluster-wide
1 120,000 1,000
4 30,000 4,000
6 20,000 6,000

For write-heavy counters and sets, split the value instead of copying it. Increment a random sub-counter hits:k, or add each member to the partition chosen by its hash. To read, sum the counters or union the sets.

The costs:

  • Writes and memory grow N times.
  • No cross-slot atomicity. MULTI and Lua cannot span all the pieces.
  • Briefly inconsistent copies. Readers can see stale data during an update.
  • Fan-out reads. Every read of a split value touches N keys.

Because no single key holds the true total, exact rate limits and compare-and-set no longer work.

import random, zlib
from redis.cluster import RedisCluster

N = 6  # copies / sub-counters per hot key
rc = RedisCluster(host="redis-node-1.internal", port=6379, decode_responses=True)

def copy_names(base):
    # No hash tag, so each name hashes to its own slot
    return [f"{base}:{i}" for i in range(N)]

def check_placement(base):
    # Distinct slots are not enough; copies must sit on distinct primaries
    primaries = {}
    for name in copy_names(base):
        node = rc.get_node_from_key(name)  # primary that owns the slot
        primaries.setdefault(node.name, []).append((name, rc.keyslot(name)))
    for primary, keys in primaries.items():
        print(primary, keys)
    return len(primaries) == N

def write_copies(base, value, ttl=300):
    # N writes; readers may briefly see old and new values side by side
    for name in copy_names(base):
        rc.set(name, value, ex=ttl)

def read_copy(base, client_id=None):
    # Stable per-client choice (crc32 is the same in every process) or random
    i = zlib.crc32(client_id.encode()) % N if client_id else random.randrange(N)
    return rc.get(f"{base}:{i}")

def incr_counter(base, amount=1):
    # Write-heavy: spread increments over N sub-counters
    rc.incrby(f"{base}:{random.randrange(N)}", amount)

def read_counter(base):
    # Fan-out read touching N keys; the sum is not an atomic snapshot
    return sum(int(rc.get(name) or 0) for name in copy_names(base))

print(check_placement("product:42"))

A redis-py cluster sketch that writes N copies of a read-heavy key, checks they land on distinct primaries rather than just distinct slots, picks a copy per read, and spreads a write-heavy counter over N sub-counters summed on read.

What are the alternatives to key splitting?

Replica reads, local caching and request coalescing each remove load without changing the keyspace.

Replica reads

A client that sends READONLY on a connection to a replica can read keys in slots that the replica's primary owns. Without it, the replica answers with a MOVED redirect. Cluster-aware clients typically expose this as a routing option that sends reads to replicas. Capacity grows only with the number of replicas in the hot shard, so a shard with two replicas gives you roughly three nodes' worth of read throughput, not cluster-wide capacity. Replication is asynchronous, so reads can be stale. They can also go backwards: a client may read from a replica that is ahead and then from one that is behind. After a failover, a promoted replica may never have received the most recent writes.

Client-side caching

An in-process cache with a TTL of one to a few seconds absorbs nearly all reads of a hot key. The cost is staleness for up to the TTL.

CLIENT TRACKING lets the server send invalidation messages when a key changes. These arrive inline over RESP3, or through a redirect connection under RESP2.

  • Default mode: the server records which keys each client has read, which costs server memory.
  • BCAST mode: the server tracks key prefixes instead, which avoids per-key server memory but sends more invalidations.

If the connection drops, invalidations can be lost, so flush the local cache on every reconnect.

Request coalescing

When a cached value expires, single-flight lets one caller refetch the value while concurrent callers wait for that result. This prevents a stampede against the shard.

Option Load removed Consistency Memory Implementation cost
Replica reads Scales with replicas per shard Stale; can go backwards Full replica per shard Low (client config)
Local TTL cache Most reads Stale up to TTL Per app process Low
CLIENT TRACKING Most reads Near-fresh; at risk on reconnect Client cache plus server tracking table Medium
Single-flight Expiry spikes only Unchanged Negligible Low to medium

When should you avoid these fixes, and what do they cost your team?

Alert on the hottest shard, not the cluster average

A cluster-wide ops/sec or CPU average hides a hot key. One saturated primary among many idle ones barely moves the mean. Alert on per-shard skew instead: the busiest primary's ops/sec divided by the median primary's, plus per-node CPU. Redis executes commands on a single main thread, so one node's CPU is what saturates. Sustained skew that tracks one slot range is your trigger to investigate.

Keep a runbook for temporary relief

Most hot keys are incidents, not architecture problems. Prepare two levers in advance:

  • A short-TTL local cache that can be enabled per key prefix.
  • Key splitting behind a feature flag, so you can turn it off once traffic subsides.

For each lever, document who can turn it on and how to confirm that the load actually moved.

When not to split

Avoid static splitting for transient hotspots, such as a viral item or a cron burst. By the time the change ships, the spike is often over, and you are left maintaining N copies of the key indefinitely.

Splitting also does nothing for big-key or slow-command bottlenecks. Fanning out an HGETALL on a huge hash multiplies the work instead of spreading it.

Adaptive splitting does handle transient hotspots, but it brings its own costs:

  • Detection thresholds that need tuning.
  • Coordination of the shard count across every client.
  • New failure modes when clients disagree about how a key is split.

The cost lands in client code

Every one of these fixes moves complexity into client code and into conventions that every service must follow. A single script or service that reads the original key directly can recreate the hotspot. Ship the logic in a shared library and audit regularly for clients that bypass it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to redis hot keys, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.