How does Redis persistence work? RDB snapshots and AOF logging

Redis keeps the dataset in memory and offers two ways to get it onto disk.

RDB writes a point-in-time snapshot of the whole dataset. A save rule (N seconds and at least M changes) or an explicit BGSAVE makes Redis fork. The child serializes its frozen copy-on-write view of memory to a temporary file and renames it over the old snapshot. The parent keeps serving traffic during this. A snapshot only reflects writes up to the moment of the fork.

AOF appends every write command to a log. On startup, Redis replays that log to rebuild memory. The log grows without bound, so Redis periodically runs an AOF rewrite. This is another fork, and the child writes the minimal commands needed to recreate the current state. With the RDB preamble enabled, the rewritten base is in RDB format, and an incremental AOF tail follows it. Newer releases split this into multiple files tracked by a manifest. Check how your version lays these files out on disk instead of assuming a single appendonly.aof.

Write path:

  • Client write → executed in memory
    • → AOF buffer → write() to OS page cache → fsync() to disk (timing set by appendfsync)
    • → fork → BGSAVE child → temp RDB file → rename
    • → fork → rewrite child → new base file → swapped in

For a queue, RDB alone has a direct consequence. Suppose snapshots run every 60 seconds and Redis crashes 55 seconds after the last one completed. Every job enqueued in those 55 seconds is gone, and so is every acknowledgment and every state change made in that window.

   +--------+
   | Client |
   +--------+
       | write command
       v
 +---------------------------------------------------------------+
 |      Redis main process: executes command on in-memory data   |
 +---------------------------------------------------------------+
       |                       |                       |
       | append                | fork()                | fork()
       |                       | (save rule/BGSAVE)    | (AOF rewrite)
       v                       v                       v
 +-------------+        +--------------+       +----------------+
 | AOF buffer  |        | BGSAVE child |       | rewrite child  |
 +-------------+        | COW snapshot |       | COW snapshot   |
       | write()        +--------------+       +----------------+
       v                       |                       |
 +-------------+               v                       v
 | OS page     |        +--------------+       +----------------+
 | cache       |        | temp RDB     |       | new base file  |
 +-------------+        +--------------+       | (RDB preamble) |
       | fsync()               | rename()      +----------------+
       | (appendfsync)         |                       | swap in
       v                       v                       v
 +---------------------------------------------------------------+
 | Disk:  AOF base + incremental tail (manifest) |  RDB snapshot |
 +---------------------------------------------------------------+

A write is executed in memory, then reaches disk through the AOF buffer, page cache and fsync, while BGSAVE and AOF rewrite run in forked children that write a new file and rename or swap it in.

What does each appendfsync mode actually lose?

Enabling AOF only decides what gets logged. The appendfsync setting decides when the log is forced from the OS page cache to stable storage, and that is what determines durability. The distinction depends on the failure type. If the Redis process crashes, writes it already handed to the kernel usually survive, because they sit in the page cache and the OS flushes them later. A power loss or kernel panic destroys the page cache, and only data that was fsynced is safe. fsync exists to protect against this second case.

Mode When fsync happens Loss window on power loss Latency / throughput cost
always Before replying to clients, per batch of writes Effectively none for acknowledged writes Every write waits on disk; throughput bounded by fsync latency
everysec Background fsync about once per second Typically about 1 second, can be more Low; stalls when the disk is slow
no Whenever the OS flushes Many seconds, depends on kernel settings Lowest

The one-second bound for everysec is typical, not guaranteed. If the disk is slow and an fsync is still running, Redis postpones further AOF writes. If the stall continues, Redis blocks writes. The data at risk can then exceed one second.

Illustrative estimate: assume each fsync takes 2 ms. Under always, a single client issuing enqueues one at a time waits for one fsync per command, so it is capped near 500 operations per second. Many concurrent clients can exceed this because Redis groups writes from one event-loop iteration into a single fsync. Even then, disk latency sets the ceiling.

When should you use RDB, AOF, or both for a queue?

Start from what a lost job costs you. That cost should drive the choice more than Redis defaults or habit.

Setup Typical loss window on crash Fits
RDB only Everything since the last snapshot Caches, or queues rebuilt from a source of truth
AOF everysec + RDB Roughly one second of writes Most job queues
AOF always + RDB Acknowledged writes survive, if the disk honours fsync Enqueues that must never vanish after an ack

RDB only works when the queue holds derived work. A crawler frontier that can be rebuilt from a database, or a fan-out queue whose producer can replay, fits here. Losing minutes of data costs a rebuild and nothing more, so you avoid paying for continuous disk writes.

AOF with everysec plus RDB is the sensible default for job queues. A second of exposure is usually acceptable when producers retry or jobs are idempotent. Keeping RDB alongside AOF gives you compact snapshot files that are easy to copy off-host for backups.

appendfsync always is for enqueues that a client treats as committed, such as payment or order events. Use it only after measuring your disk's fsync latency under load. That latency becomes the floor for every write.

The tradeoff lands on your team either way. Tighter durability means more spend on low-latency storage and more time tuning tail latency. Looser durability moves that work into the application: retries, deduplication, and reconciliation jobs that find and replay what Redis dropped.

Who should use Redis persistence for a durable queue?

This guidance is aimed at teams that already run Redis, or are about to, and use it as a job or task queue: background jobs, email and notification sends, webhook delivery, report generation, and similar work where low enqueue latency and operational simplicity matter.

It fits best when:

  • Producers can retry and consumers can be made idempotent. The patterns later in this article assume at-least-once delivery. If your jobs can tolerate being run twice, Redis with AOF covers most crash scenarios.
  • The queue is not the system of record. Jobs describe work derived from data that lives elsewhere, such as a database row that a reconciliation job can rescan.
  • You can operate the topology properly. Someone owns redis.conf, kernel settings like vm.overcommit_memory, restart policies, monitoring of INFO persistence, and failover behaviour.

The people who should read the configuration and verification sections closely are the application engineers who choose the dequeue pattern and the platform or SRE engineers who own the Redis deployment. Durability depends on both, and a gap in either one loses jobs.

It is a poor fit when a lost acknowledged message carries regulatory or contractual consequences, or when you need long retention and replay as a guaranteed feature. The final section covers that case.

How can choosing persistence deliberately save engineering time?

Most of the engineering cost of a Redis queue comes from incidents and rework, not from the initial setup. Making the persistence decision explicitly, using the analysis above, reduces that cost in several concrete ways:

  • Fewer lost-job investigations. A queue left on RDB-only defaults can drop minutes of work on a crash. Afterward, someone has to work out which jobs vanished, cross-reference logs or the database, and re-enqueue by hand. Enabling AOF with everysec shrinks that window to about a second. Combined with idempotent consumers and retrying producers, recovery becomes automatic and no longer needs a manual cleanup.
  • No premature latency tuning. Choosing appendfsync always by reflex makes disk fsync latency the floor for every enqueue. That can lead to time spent chasing tail latency and storage upgrades for jobs that would have been fine with everysec. Deciding from the cost of a lost job avoids that work.
  • Reusing infrastructure you already run. If Redis is already in your stack, configuring persistence correctly and using Streams consumer groups or BLMOVE can meet the durability needs of many job queues. You avoid deploying, learning and operating a separate broker.
  • Shorter recovery after restarts. aof-use-rdb-preamble yes keeps the rewritten AOF base compact. Restarts load a snapshot plus a short tail instead of replaying every command ever issued, which reduces the time the queue is unavailable and the time engineers spend waiting on it.
  • Catching silent failures early. Alerting on rdb_last_bgsave_status, aof_last_bgrewrite_status and aof_last_write_status turns a persistence failure that could go unnoticed for days into a page on the day it happens. The MISCONF behaviour from stop-writes-on-bgsave-error makes the failure visible instead of letting persistence quietly stop.
  • Rehearsed repairs. Running restart, kill -9, VM power-off and redis-check-aof --fix drills ahead of time means the first time you perform these steps is not during an outage.

Illustrative estimate: assume a team has two Redis crashes a year, and each lost-job cleanup on an RDB-only queue takes one engineer a full day of log correlation and manual re-enqueueing. Configuring AOF and idempotent consumers once can remove most of those two days of work per year, plus the on-call disruption. Your actual numbers depend entirely on your crash frequency and how hard lost jobs are to reconstruct.

How do you configure Redis persistence for a durable queue?

For a queue, start from AOF and treat RDB as a secondary safety net. In redis.conf, set appendonly yes and choose appendfsync everysec unless the earlier analysis pushed you to always. Keep aof-use-rdb-preamble yes so rewrites produce a compact RDB-format base plus an incremental tail. This makes restarts on a large queue much faster than replaying every LPUSH ever issued. Keep a few save rules, such as save 3600 1 300 100. They give you point-in-time snapshots for backups, though not durability. auto-aof-rewrite-percentage 100 and auto-aof-rewrite-min-size 64mb control when the log is compacted, which stops it from growing without bound.

To enable AOF on a running instance, use CONFIG SET appendonly yes. Redis builds the initial file in the background. Follow it with CONFIG REWRITE so the change survives a restart. BGREWRITEAOF triggers compaction manually, which is useful before a planned maintenance window.

Every rewrite and snapshot forks the process. With copy-on-write, each page the parent modifies during the child's run gets duplicated. Under heavy write load, memory use can approach double the dataset in the worst case, so provision headroom accordingly. On large instances, the fork call itself can stall the main thread long enough to show up in tail latency. Set vm.overcommit_memory = 1 on Linux. Without it, the kernel may refuse the fork even when the child would never touch most pages.

If a background save fails and stop-writes-on-bgsave-error is yes (the default), Redis rejects writes with a MISCONF error. Producers stall rather than silently losing persistence. For a queue, that is usually the behavior you want, but alert on it.

sudo sysctl -w vm.overcommit_memory=1
echo 'vm.overcommit_memory = 1' | sudo tee /etc/sysctl.d/99-redis.conf

cat > redis-queue.conf <<'EOF'
appendonly yes                   # AOF is the durability mechanism
appendfsync everysec             # up to ~1s of acknowledged writes at risk; 'always' for tighter
aof-use-rdb-preamble yes         # rewrite = compact RDB base + AOF tail -> faster restarts
save 3600 1                      # point-in-time snapshots for backups, not durability
save 300 100
auto-aof-rewrite-percentage 100  # rewrite when AOF doubles since last rewrite...
auto-aof-rewrite-min-size 64mb   # ...but never for a log smaller than this
stop-writes-on-bgsave-error yes  # default: reject writes (MISCONF) if a snapshot fails
EOF

redis-cli CONFIG SET appendonly yes
redis-cli CONFIG SET appendfsync everysec
redis-cli CONFIG REWRITE         # persist the runtime change into the loaded redis.conf

redis-cli BGREWRITEAOF

redis-cli INFO persistence | grep -E 'aof_enabled|aof_rewrite_in_progress|aof_last_bgrewrite_status|rdb_last_bgsave_status'

redis-cli INFO stats | grep latest_fork_usec

Sets vm.overcommit_memory, writes a queue-oriented redis.conf persistence excerpt, enables AOF at runtime with CONFIG SET plus CONFIG REWRITE, triggers BGREWRITEAOF, and checks the INFO fields that reveal failed saves and fork duration.

Why isn't replication a substitute for persistence?

Replication does not give you durability. Redis replicates asynchronously: the primary applies a write, replies to the client, and streams the command to replicas afterward. If the primary crashes in that gap, the producer has an acknowledgment for a job that exists nowhere else.

WAIT narrows the gap. It blocks the client until a given number of replicas have acknowledged the preceding writes, or until a timeout expires. That still falls short of a guarantee. A timeout returns with fewer acknowledgments than requested, and the write is not rolled back. A replica's acknowledgment also means the data reached its memory, not its disk. The Redis documentation is explicit that WAIT does not make Redis strongly consistent.

Failover adds its own loss path. When Sentinel or Cluster promotes a replica, they choose among the replicas that are reachable, and the chosen one may be behind the old primary. Jobs it never received are gone. When the old primary rejoins, it resyncs from the new primary and discards whatever it had that the new primary lacks.

The worst case is a primary that has persistence disabled and restarts automatically. It comes back with an empty dataset, the replicas reconnect, and a full sync replaces their data with that empty dataset. The copies you were counting on get wiped. If you run a primary without persistence, disable automatic restarts on it, so that failover promotes a replica instead.

A durability claim for a queue therefore has to cover the whole topology: each node's persistence settings, its restart behavior, replication lag, and failover policy. One instance's appendfsync setting tells you only part of the story.

#!/usr/bin/env bash
set -euo pipefail

PRIMARY=redis-primary.internal
NODES=(redis-primary.internal redis-replica-1.internal redis-replica-2.internal)

out=$(printf 'LPUSH jobs job:1234\nWAIT 1 500\n' | redis-cli -h "$PRIMARY" -p 6379)
acked=$(printf '%s\n' "$out" | tail -n 1)

if [ "$acked" -lt 1 ]; then
  # Timeout: the job is still on the primary and is NOT rolled back.
  # Only the primary holds it. Treat it as at risk, not as failed.
  echo "warning: job:1234 acknowledged by $acked replicas" >&2
fi
for node in "${NODES[@]}"; do
  echo "== $node"
  redis-cli -h "$node" -p 6379 CONFIG GET appendonly
  redis-cli -h "$node" -p 6379 CONFIG GET appendfsync
  redis-cli -h "$node" -p 6379 CONFIG GET save
  # Replication role, link status and offsets show how far replicas lag.
  redis-cli -h "$node" -p 6379 INFO replication | grep -E '^(role|master_link_status|master_repl_offset|slave[0-9]+):'
done

A shell check that sends LPUSH and WAIT on one connection, treats a short replica acknowledgment count as a durability risk, and audits persistence and replication state on every node, since a single node's config cannot back a durability claim.

Which queue patterns close the gaps persistence leaves?

Persistence settings decide how much a crash can lose. The dequeue pattern decides whether a message a consumer has already taken can be lost, and that gap is usually wider.

Lists: BLMOVE into a processing list

A plain BRPOP deletes the item the moment a worker receives it. If the worker dies mid-job, the item is gone. BLMOVE instead moves the item atomically from the queue into a per-worker processing list. The worker runs LREM on that processing list only after the job succeeds. A separate reaper scans processing lists and pushes stale items back onto the queue. Lists store no timestamps, so the reaper needs its own record of when each item was claimed, such as a sorted set or a timestamp embedded in the payload.

Streams: consumer groups

Streams give you this bookkeeping natively. XREADGROUP records each delivered entry in the group's pending entries list, and the entry stays there until the consumer calls XACK. XPENDING reports idle time and delivery count per entry. XAUTOCLAIM reassigns entries idle longer than a threshold to a healthy consumer. A rising delivery count flags poison messages that should go to a dead-letter stream.

At-least-once is the real guarantee

A consumer can finish its work and crash before acking, so redelivery will happen. Consumers must be idempotent, for example by keying side effects on a message ID. Producers should also retry on errors or timeouts, reusing a stable ID so duplicates collapse. With both in place, a one-second everysec loss window becomes something the application can recover from. You no longer have to chase persistence settings that can never be perfect.

redis-cli BLMOVE jobs:queue jobs:processing:worker-a RIGHT LEFT 5
redis-cli LREM jobs:processing:worker-a 1 '{"job_id":"order-1042"}'

redis-cli XGROUP CREATE jobs workers '$' MKSTREAM

redis-cli XADD jobs '*' job_id order-1042 action charge_card

redis-cli XREADGROUP GROUP workers worker-a COUNT 10 BLOCK 5000 STREAMS jobs '>'

redis-cli SET processed:order-1042 1 NX EX 86400

redis-cli XACK jobs workers 1731400000000-0

redis-cli XPENDING jobs workers - + 10

redis-cli XAUTOCLAIM jobs workers worker-b 60000 0-0 COUNT 10

Redis CLI sequence showing a BLMOVE processing-list dequeue, then a Stream consumer group that reads, deduplicates by job ID, acknowledges with XACK, inspects pending entries with XPENDING, and reclaims stalled entries with XAUTOCLAIM for at-least-once delivery.

How do you verify durability, and what are the alternatives to Redis?

Settings only matter if you have checked them under failure. Treat verification as part of the queue's design.

Check what Redis reports

INFO persistence shows whether persistence is working right now. Alert when rdb_last_bgsave_status or aof_last_bgrewrite_status is not ok, and when aof_last_write_status reports an error. A failed background save can go unnoticed for days because writes keep succeeding in memory. Also track rdb_last_save_time, so a stalled snapshot schedule pages someone.

Test recovery before you need it

Check How Why it matters
Restart time Restart on a production-sized dataset and time it until loading returns to 0 Replaying a large AOF is slower than loading an RDB, and the queue is unavailable during the load
Truncated AOF tail Practice redis-check-aof --fix on a copy of the file A crash mid-write can leave a partial final command. Learn the repair steps in a drill, not during an outage
Process kill kill -9 the server under load Covers the loss window when the OS page cache survives
VM kill Hard power-off the host Covers the loss of unsynced writes. Count the missing job IDs

Always back up the AOF before running the repair tool, because --fix discards the damaged tail.

When to avoid Redis

If compliance or contracts require an audited guarantee that an acknowledged message is never lost, choose a purpose-built durable log or message broker. These systems acknowledge a write only after it has been fsynced and replicated to a quorum. They also provide consumer offsets or explicit acknowledgements, and they make retention and replay first-class, documented behaviour rather than something you assemble from configuration.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to redis rdb vs aof, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.