What is Redis memory fragmentation and why does RSS stay high?
Redis does not manage memory itself. It asks its allocator, usually jemalloc, for blocks of specific sizes. jemalloc groups same-sized allocations into size classes and packs them into runs of memory pages. When Redis frees a key, that block goes back to jemalloc, not to the operating system. jemalloc can reuse the slot for another allocation of the same size class. It can only return a page to the OS once every allocation on that page is gone.
That rule causes fragmentation. Suppose a dataset grows to its peak and then you delete half the keys, trim large hashes, or overwrite 300-byte strings with 60-byte ones. The surviving allocations are scattered across the pages:
| Page | Slots (█ live, ░ freed) | Can go back to the OS? |
|---|---|---|
| 1 | █░░░░░░░ | No, one live object |
| 2 | ░░░█░░░░ | No, one live object |
| 3 | ░░░░░░░█ | No, one live object |
| 4 | ░░░░░░░░ | Yes |
In this example 29 of 32 slots are free, yet only one page out of four is released. The freed bytes are counted as free inside jemalloc, but they stay resident in the process.
Size changes make this worse. A shrunken value moves to a smaller size class on different pages. The new allocation lands somewhere else, and the old page is left with one fewer tenant but still pinned.
From the outside, used_memory (what Redis thinks it holds) drops, while RSS (what the kernel has mapped for the process) stays close to its historical peak. The gap is held by mostly empty pages that jemalloc cannot return.
+-------------+ free(key) +-------------------------------------------+
| Redis | --------------> | jemalloc: one size class, 8 slots/page |
| used_memory | | |
| drops | | Page 1 [#][ ][ ][ ][ ][ ][ ][ ] pinned |
+-------------+ | Page 2 [ ][ ][ ][#][ ][ ][ ][ ] pinned |
| Page 3 [ ][ ][ ][ ][ ][ ][ ][#] pinned |
| Page 4 [ ][ ][ ][ ][ ][ ][ ][ ] empty |
+---------+-------------------+-------------+
| |
v v
Pages 1-3 stay resident Page 4 returned to OS
(29 of 32 slots free) (only empty pages go)
used_memory [###.............................] 3 of 32 slots live
RSS [########################........] 3 of 4 pages mapped
[#] live object [ ] freed slot (free inside jemalloc, still resident)
Redis returns freed blocks to jemalloc, but jemalloc can release only a page with no live allocations, so three scattered live objects keep three of four pages resident and RSS stays high while used_memory drops.
How do you read mem_fragmentation_ratio and the allocator fields in INFO memory?
The top-level numbers are used_memory (bytes Redis has requested from the allocator) and used_memory_rss (bytes the kernel reports as resident). mem_fragmentation_ratio is RSS divided by used_memory, and mem_fragmentation_bytes is the difference between the two. Start with the bytes figure, because the ratio misleads on small instances. A process carries a fixed baseline of binary, libraries and allocator metadata. Illustrative estimate: an instance with 2 MB of data and 9 MB of RSS reports a ratio of 4.5, yet only 7 MB is unaccounted for. A 40 GB instance at 1.3 has 12 GB stranded. The second case is the real problem.
To find where the gap sits, split RSS into three layers using the allocator fields. A ratio below 1.0 means RSS is smaller than the data set. That usually indicates the kernel has swapped pages out, which is a latency problem rather than a fragmentation problem.
| Metric | Compares | What a high value means | Matching fix |
|---|---|---|---|
allocator_frag_ratio / allocator_frag_bytes |
allocator_active vs allocator_allocated |
Internal fragmentation: partly filled allocator pages | activedefrag |
allocator_rss_ratio / allocator_rss_bytes |
allocator_resident vs allocator_active |
Pages freed by Redis but retained by jemalloc | MEMORY PURGE, allocator decay behaviour |
rss_overhead_ratio / rss_overhead_bytes |
used_memory_rss vs allocator_resident |
Memory outside the allocator, such as fork copy-on-write, buffers or modules | Investigate the source; defrag does not help |
mem_fragmentation_ratio < 1.0 |
RSS vs used_memory |
Swapping | Add RAM, lower maxmemory, disable swap |
Only the first row responds to activedefrag. If the gap sits in the other two layers, enabling defrag spends CPU without reclaiming memory.
# redis-cli INFO memory (excerpt; illustrative values for a ~40 GB instance, not a measurement)
# Top level: what Redis asked for vs what the kernel says is resident
used_memory:42949672960 # 40 GB requested from the allocator
used_memory_rss:55834574848 # 52 GB resident according to the kernel
mem_fragmentation_ratio:1.30 # rss / used_memory
mem_fragmentation_bytes:12884901888 # 12 GB gap; check this before the ratio
# Layer 1: internal fragmentation (active vs allocated). activedefrag addresses this
allocator_allocated:42991616000 # bytes held in live allocations
allocator_active:51539607552 # bytes in pages that contain live allocations
allocator_frag_ratio:1.20
allocator_frag_bytes:8547991552 # ~8 GB in partly filled pages
# Layer 2: retained pages (resident vs active). MEMORY PURGE or allocator decay
allocator_resident:53687091200 # pages jemalloc still holds from the OS
allocator_rss_ratio:1.04
allocator_rss_bytes:2147483648 # ~2 GB freed by Redis, not yet returned
# Layer 3: non-allocator memory (rss vs allocator_resident). Defrag does not help
rss_overhead_ratio:1.04
rss_overhead_bytes:2147483648 # ~2 GB of fork COW, buffers, modules, etc.
# Check: 8.5 GB + 2 GB + 2 GB + (allocated - used_memory, ~0.04 GB) ~= 12 GB gap
# Here most of the gap is in layer 1, so activedefrag is the right tool.
# If mem_fragmentation_ratio were < 1.0, look for swapping, not fragmentation.
An annotated INFO memory excerpt with illustrative values that splits a 12 GB RSS gap into allocator fragmentation, retained pages and non-allocator overhead, so you can see which fix applies.
What else inflates Redis RSS besides fragmentation?
Work through these before concluding the allocator is the problem.
- Fork copy-on-write.
BGSAVEand AOF rewrites fork the process. Parent and child share pages until the parent writes to one. Each write then copies that page, so a write-heavy workload during a long save can come close to doubling physical memory. CheckINFO persistence.rdb_bgsave_in_progressandaof_rewrite_in_progressshow whether a fork is active, and the COW size fields, such asrdb_last_cow_size, report how much was copied. - Transparent huge pages. With THP enabled, the copy unit becomes a 2 MB huge page instead of a 4 KB page. A one-byte write then duplicates far more memory. Redis logs a startup warning when THP is on, and disabling it is the standard recommendation.
- Client output buffers. Lagging replicas, slow pub/sub subscribers and
MONITORsessions accumulate unsent replies.INFO clientsshowsclient_recent_max_output_buffer. InCLIENT LIST, look for largeomemvalues. Theflagsfield identifies the client:Sfor a replica,Pfor pub/sub,Ofor monitor. - Lua and modules. Cached scripts take up space. Modules that allocate outside the Redis allocator add to RSS without appearing in
used_memory.
MEMORY STATS breaks overhead into replica and normal client buffers, the replication backlog and Lua caches. MEMORY DOCTOR gives a quick heuristic diagnosis in plain text. MEMORY MALLOC-STATS dumps raw allocator statistics.
None of these causes responds to activedefrag. It only compacts fragmented allocations, and none of the above is fragmentation.
#!/usr/bin/env bash
# Rule-out pass for high RSS before blaming fragmentation.
# Read-only commands. Run on the Redis host so the THP check is meaningful.
set -uo pipefail
R="redis-cli -h ${REDIS_HOST:-127.0.0.1} -p ${REDIS_PORT:-6379}"
echo "== Fork state and copy-on-write =="
# A *_in_progress value of 1 means a fork is active and COW may be inflating RSS.
# The *cow_size fields report bytes copied (the set varies by Redis version).
$R INFO persistence | tr -d '\r' \
| grep -E '^(rdb_bgsave_in_progress|aof_rewrite_in_progress|.*cow_size):'
echo "== Transparent huge pages (the bracketed value is active) =="
# [always] means each COW copy can be 2 MB instead of 4 KB.
cat /sys/kernel/mm/transparent_hugepage/enabled
echo "== Client output buffers =="
$R INFO clients | tr -d '\r' \
| grep -E '^(connected_clients|client_recent_max_output_buffer):'
# Top 10 clients by omem. Flags: S=replica, P=pub/sub, O=MONITOR.
$R CLIENT LIST | awk '{
for (i = 1; i <= NF; i++) { split($i, kv, "="); f[kv[1]] = kv[2] }
print f["omem"], "flags=" f["flags"], f["addr"], "cmd=" f["cmd"]
}' | sort -rn | head -n 10
echo "== Overhead breakdown (key, then value) =="
# Replica and normal client buffers, replication backlog, Lua caches.
$R MEMORY STATS \
| grep -A1 -E '^(clients\.slaves|clients\.normal|replication\.backlog|lua\.caches)$'
echo "== Heuristic read =="
$R MEMORY DOCTOR
echo "== Raw allocator stats (first lines only; output is long) =="
$R MEMORY MALLOC-STATS | head -n 20
# Only if none of the above explains the RSS gap is activedefrag worth considering.
A read-only redis-cli script that checks fork state and copy-on-write, the transparent huge pages setting, client output buffers by client type, and the MEMORY STATS overhead breakdown, then prints MEMORY DOCTOR and the first lines of MEMORY MALLOC-STATS, all before activedefrag is considered.
How does activedefrag work inside Redis?
Active defragmentation relies on the jemalloc build bundled with Redis, which exposes an extra hint that standard jemalloc does not. For any allocation, the hint reports whether the slab holding it is less utilized than the average slab for that size class. Redis uses this to decide what to move. If a value sits in a sparsely used slab, Redis allocates a fresh copy (bypassing the thread cache so the copy lands in a denser slab), copies the bytes, updates every pointer that referenced the old address, and frees the original. Once enough neighbors move out, a slab becomes completely empty and jemalloc can return its pages to the OS. That is the point where RSS actually drops.
Pointer updates are the reason this cannot run on a background thread. Redis's data structures are not locked, so defrag runs on the main thread. The cron triggers it, and it works in short time slices between commands, with each slice bounded by the configured CPU budget. It walks the keyspace with a scan cursor, so it can stop at any time and resume later. Large collections are handled in bounded steps rather than in one pass, which keeps a single huge hash or sorted set from stalling the event loop.
The hint exists only in the bundled jemalloc, so builds linked against libc malloc, and some distribution packages that link a system jemalloc, cannot use the feature. Check the mem_allocator field in INFO memory. It should start with jemalloc. On an unsupported build, enabling activedefrag returns an error. Some managed services hide this setting entirely.
Tuning activedefrag: thresholds, CPU bounds and a conservative starting config
Six directives control the feature, and they interact. activedefrag yes only permits defragmentation. Whether a cycle starts depends on two gates that must both be crossed. active-defrag-ignore-bytes sets the minimum amount of fragmentation waste, in bytes. active-defrag-threshold-lower sets the minimum fragmentation percentage. The byte gate stops a small instance with a high ratio from spending CPU to recover a few megabytes.
Once a cycle is running, effort scales between active-defrag-cycle-min and active-defrag-cycle-max. These are percentages of CPU time. Effort rises as fragmentation moves from the lower threshold toward active-defrag-threshold-upper, where it reaches the maximum. active-defrag-max-scan-fields limits how many fields of a single large hash, set, sorted set or list are processed in one step. Above that limit, the key is handled in later passes, so one huge key cannot stall the event loop.
For a conservative start, use these values:
active-defrag-ignore-bytes 200mb, so that small amounts of waste are ignored.threshold-lower 15, so that normal allocator slack does not trigger cycles.threshold-upper 50.cycle-min 1.cycle-max 10, which keeps peak effort low.max-scan-fields 1000.
Raise cycle-max only if INFO memory shows that fragmentation is not falling.
| Knob | Effect | Risk if too aggressive |
|---|---|---|
active-defrag-ignore-bytes |
Byte gate for starting | Too low: CPU spent on trivial waste |
active-defrag-threshold-lower |
Percentage gate for starting | Too low: near-constant cycling |
active-defrag-threshold-upper |
Point where maximum effort applies | Too low: full effort at mild fragmentation |
active-defrag-cycle-min / active-defrag-cycle-max |
CPU effort bounds | Too high: higher tail latency |
active-defrag-max-scan-fields |
Work per step on large keys | Too high: latency spikes on big collections |
# redis.conf: conservative active defragmentation settings
# Requires a Redis build that uses jemalloc (the default allocator on Linux).
# redis.conf does not support trailing comments, so each comment sits on its own line.
# Allow defragmentation. Cycles start only when BOTH gates below are crossed.
activedefrag yes
# Gate 1 (bytes): ignore fragmentation waste below 200 MB, so a small
# instance with a high ratio does not spend CPU to recover a few megabytes.
active-defrag-ignore-bytes 200mb
# Gate 2 (percent): start only above 15% fragmentation, so normal
# allocator slack does not trigger cycles.
active-defrag-threshold-lower 15
# Effort rises from cycle-min at 15% to cycle-max at 50% and above.
active-defrag-threshold-upper 50
# CPU effort bounds, as a percentage of CPU time.
# A low ceiling limits added tail latency. Raise cycle-max only if
# INFO memory shows that fragmentation is not falling.
active-defrag-cycle-min 1
active-defrag-cycle-max 10
# Process at most 1000 fields of a large hash, set, sorted set or list
# per step. Larger keys are finished in later passes, so one huge key
# cannot stall the event loop.
active-defrag-max-scan-fields 1000
# Verify from a shell after applying the config:
# redis-cli INFO memory | grep -E 'frag|active_defrag_running'
# Watch allocator_frag_ratio and allocator_frag_bytes over time.
# active_defrag_running reports the CPU percentage the current cycle is using.
A conservative redis.conf block that sets both start gates, the effort ramp and the CPU bounds for active defragmentation, followed by an INFO memory check to confirm that fragmentation is falling before you raise cycle-max.
How do you enable activedefrag safely in production?
Treat the rollout as an experiment with a control.
-
Capture a baseline. Record
INFO memoryat regular intervals for at least one full traffic cycle. Noteallocator_frag_ratio,allocator_frag_bytes,mem_fragmentation_bytesandused_memory_rss. For latency, setlatency-monitor-thresholdand readLATENCY LATEST, or use the p99 your clients already report. Client-side percentiles are better because they include network and queueing time. Without a baseline, you cannot attribute a later latency bump to defrag. -
Enable on one replica. Run
CONFIG SET activedefrag yeson a replica that serves little or no read traffic. If it returns an error, the build is probably not using jemalloc, and defrag is unavailable on that node. -
Watch both effects together.
allocator_frag_ratioandmem_fragmentation_bytesshould trend down over minutes to hours. At the same time, watch Redis CPU and p99 latency. If latency rises, loweractive-defrag-cycle-maxrather than turning defrag off. -
Roll out to the primary. Do this during a lower-traffic window, using the same observation loop.
-
Persist the setting on every node. Use
CONFIG REWRITEor edit the config file.CONFIG SETchanges are lost on restart and are not propagated to replicas. A promoted replica that lacks the setting brings the fragmentation back after failover.
For alerting, combine absolute allocator_frag_bytes with allocator_frag_ratio. The top-level mem_fragmentation_ratio misleads on small datasets and is skewed by swap and copy-on-write.
This sequence saves engineering time because each change produces evidence. Tuning becomes a single bounded adjustment instead of repeated guesses made during incidents.
When should you avoid activedefrag, and what are the alternatives?
Defrag only compacts allocator fragmentation inside Redis's own heap. If the extra RSS comes from copy-on-write during a fork, client output buffers, pages the allocator has kept but not returned, or swapped memory, moving values around does nothing useful. The diagnosis from earlier sections has to point at fragmentation before defrag is worth considering.
Avoid it on instances that are already CPU-bound or have tight tail-latency budgets. Defrag runs on the main thread, so every cycle it spends delays commands. It also cannot save you from an imminent OOM kill, because it reclaims memory gradually over many cycles and the kernel will act first. Workloads whose value sizes keep changing will fragment again soon after each pass. In that case defrag becomes a permanent background CPU cost rather than a one-time cleanup.
| Option | What it fixes | Cost | Dev / ops effort |
|---|---|---|---|
| activedefrag | Fragmentation in the jemalloc heap | Ongoing main-thread CPU and added latency | Low dev effort; ops must tune and monitor it |
| MEMORY PURGE | Pages jemalloc retains but no longer uses | Brief blocking call; does not touch fragmentation | Minimal; run on demand |
| Restart or failover to a freshly synced replica | All heap fragmentation at once | A failover event, plus fork and copy-on-write load during the sync | Moderate ops effort; needs reliable failover |
| Uniform value sizes in the data model | The root cause of refragmentation | Schema and code changes | High dev effort, then little ongoing ops work |
| More headroom | OOM risk from any of these sources | More memory per instance | Low effort; capacity planning |
How much memory headroom does Redis need beyond maxmemory?
maxmemory caps used_memory, the bytes Redis has asked the allocator for. The kernel and the cgroup see RSS. If the container limit equals maxmemory, eviction never triggers before the OOM killer does, because fragmentation, copy-on-write pages from a BGSAVE or AOF rewrite fork, and buffers all sit above the eviction line.
Illustrative estimate
The inputs below are assumptions, not measurements: a 10 GB dataset at maxmemory, a mem_fragmentation_ratio of 1.3, 25% of pages written while the fork child is alive, and a 1 GB allowance for client output buffers, the replication backlog and rewrite buffers.
| Component | Assumption | Memory |
|---|---|---|
Dataset (used_memory) |
At maxmemory |
10 GB |
| Fragmentation overhead | Ratio 1.3 | +3 GB (RSS 13 GB) |
| Fork copy-on-write | 25% of 13 GB RSS duplicated | +3.25 GB |
| Buffers | Stated allowance | +1 GB |
| Recommended cgroup limit | Rounded up | ≈18 GB |
The COW term scales with RSS rather than the dataset. The kernel copies whole pages, so a single small write duplicates the entire page that holds it. Write-heavy workloads during a long persistence window push that 25% higher. Measure your own rate from the copy-on-write figure Redis logs after each fork.
Undersizing looks cheaper on the bill, but the costs show up elsewhere. An OOM kill drops the whole dataset. Recovery then requires a reload from disk or a full resync, which loads the primary and the network while someone is paged. Running with deliberate headroom costs a predictable amount of memory, and that cost is easy to plan for. Reacting to OOM kills means repeated incidents and emergency resizing.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to redis memory fragmentation, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.