Redis Streams vs Kafka: Durability, Retention and Ops Cost Compared

How Redis Streams and Kafka store and order events

Both systems give you an append-only log that many consumers can read independently. That is why teams use them for event streaming. They differ in where the log lives and how it is split, and that difference sets every ordering guarantee you get.

A Redis Stream is a single key. Producers append entries with XADD orders * customer_id 42 total 19.99. The * asks Redis to generate an entry ID in the form <milliseconds>-<sequence>, and IDs always increase within the key. The stream lives in memory as part of the Redis dataset. Total order within one key is cheap because one key is served by one Redis primary.

A Kafka topic is a named set of partitions, created with something like kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 3 --bootstrap-server localhost:9092. Each partition is an ordered log of segment files on disk. Records are addressed by an integer offset that is unique only within that partition. The producer picks the partition, usually by hashing the record key.

CopyRedis                                Kafka topic "orders"
producer --XADD--> [stream "orders"]  producer --> P0 [0|1|2|3] --> group A, consumer 1
                   1-0 | 1-1 | 2-0               --> P1 [0|1|2]   --> group A, consumer 2
                        |                        --> P2 [0|1|2|3] --> group A, consumer 3
                   consumer group

Ordering follows the storage unit. Redis orders entries per stream key, and Kafka orders records per partition. If you need more throughput, you add keys or partitions, and then you no longer have a single global order. You can get global ordering in either system only by keeping everything in one key or one partition, which caps your throughput.

Copy# --- Redis: one stream key, one total order ---
# '*' lets Redis assign a monotonically increasing <ms>-<seq> ID.
redis-cli XADD orders '*' customer_id 42 total 19.99
redis-cli XADD orders '*' customer_id 7 total 5.00

# Read every entry in ID order. This is the only order the key has.
redis-cli XRANGE orders - +

# Create a consumer group that starts at the beginning of the stream.
# MKSTREAM creates the key if it does not exist yet.
redis-cli XGROUP CREATE orders billing 0 MKSTREAM

# A group member reads entries not yet delivered to the group ('>').
redis-cli XREADGROUP GROUP billing worker-1 COUNT 10 STREAMS orders '>'

# --- Kafka: one topic, three independently ordered partitions ---
kafka-topics.sh --create --topic orders \
  --partitions 3 --replication-factor 3 \
  --bootstrap-server localhost:9092

# Show partition leaders and replicas.
kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

# Produce keyed records as key:value. The default partitioner hashes the key,
# so all events for customer 42 land in the same partition and stay ordered.
printf '42:{"total":19.99}\n7:{"total":5.00}\n42:{"total":3.50}\n' | \
  kafka-console-producer.sh --topic orders \
    --bootstrap-server localhost:9092 \
    --property parse.key=true --property key.separator=:

# Consume as part of a group. Order holds within each partition,
# not across the whole topic.
kafka-console-consumer.sh --topic orders --group billing \
  --from-beginning --bootstrap-server localhost:9092 \
  --property print.key=true

Side-by-side shell session showing that a Redis Stream is one ordered key read by a consumer group, while a Kafka topic splits records across partitions by key and only guarantees order within each partition.

How consumer groups work in each system

In Redis, a consumer group is server-side state attached to a stream key. A consumer calls XREADGROUP GROUP orders worker-1 COUNT 10 STREAMS orders >. The > asks for entries never delivered to this group. Each entry returned is recorded in the group's Pending Entries List (PEL) along with the consumer name, a delivery count and a timestamp. After processing, the consumer calls XACK orders orders <id>, which removes the entry from the PEL. If a consumer dies, its entries stay in the PEL. Another consumer recovers them with XAUTOCLAIM orders orders worker-2 60000 0-0 COUNT 10, which takes ownership of entries that have been idle for more than 60 seconds. XCLAIM does the same for specific IDs. Redis tracks every message individually.

Kafka tracks one number per partition instead. Each partition is assigned to one consumer in the group, and the group commits the offset it has processed up to. When a member joins, leaves or misses heartbeats, the group rebalances and reassigns partitions. The new owner resumes from the last committed offset.

Both systems give at-least-once delivery by default, so consumers must be idempotent in both. In Redis, there are two specific failure modes. If you skip acks, the PEL grows without bound. If nothing runs XAUTOCLAIM, entries from crashed consumers stay stuck indefinitely.

Copy# Create the stream and consumer group. '$' means the group starts with new entries only.
# MKSTREAM creates the stream key if it does not exist yet.
redis-cli XGROUP CREATE orders orders '$' MKSTREAM

# Producer appends an event. '*' lets Redis assign the entry ID.
redis-cli XADD orders '*' order_id 1001 status created

# worker-1 reads up to 10 entries never delivered to this group ('>').
# Each returned entry is added to the group's PEL under worker-1.
redis-cli XREADGROUP GROUP orders worker-1 COUNT 10 STREAMS orders '>'

# Inspect the PEL: summary first, then per-entry detail
# (ID, owning consumer, idle ms, delivery count).
redis-cli XPENDING orders orders
redis-cli XPENDING orders orders - + 10

# After processing succeeds, ack by ID to remove the entry from the PEL.
# Skip this step and the PEL grows without bound.
redis-cli XACK orders orders 1700000000000-0

# Recovery: worker-2 takes ownership of entries idle for more than 60s,
# scanning the PEL from the start (0-0). The reply includes a cursor for the
# next call; repeat until the cursor is 0-0. Claiming increments the delivery
# count, so processing must be idempotent.
redis-cli XAUTOCLAIM orders orders worker-2 60000 0-0 COUNT 10

# Targeted alternative: claim specific IDs if they have been idle for 60s or more.
redis-cli XCLAIM orders orders worker-2 60000 1700000000001-0

# worker-2 must still XACK whatever it claims and finishes processing.
redis-cli XACK orders orders 1700000000001-0

A redis-cli sequence that creates a consumer group, reads entries with XREADGROUP, inspects and acknowledges entries in the PEL, and uses XAUTOCLAIM or XCLAIM to take over entries left by a dead consumer.

Durability: what each system guarantees when a node fails

Redis: persistence is a setting, not a property

Redis holds the stream in memory, so durability depends on configuration. RDB snapshots (save 900 1) write the dataset periodically, and a crash loses every XADD since the last snapshot. The append-only file (appendonly yes) logs each write, and appendfsync controls when it reaches disk:

• always fsyncs on every write. It is the safest option and the slowest.
• everysec fsyncs once a second. A crash can lose roughly the last second of writes that clients already saw acknowledged.
• no leaves flushing to the OS.

Replication is asynchronous. The primary acknowledges an XADD before replicas receive it. If the primary dies first, failover promotes a replica that never received those entries, and they are gone. WAIT 1 100 blocks until one replica confirms receipt, which narrows that window. It does not turn Redis into a consensus-replicated log. The replica may not have fsynced the write, and failover does not guarantee it promotes the replica that received it.

Check CONFIG GET appendonly on production. Teams often assume persistence is on when the instance was provisioned as a cache with appendonly no and save "".

Kafka: durability through replication

Kafka writes to the page cache and by default does not fsync each message. It protects data by copying it to several brokers. A typical setup uses these topic settings:

• replication.factor=3
• min.insync.replicas=2
• unclean.leader.election.enable=false

Producers use acks=all and enable.idempotence=true. With these settings, the leader acknowledges a write only after every in-sync replica has it. If fewer than two replicas are in sync, the producer gets an error instead of a false acknowledgement. You can lose one broker without losing acknowledged data.

The cost is latency. Each acknowledgement waits for the slowest in-sync follower to fetch the write, which adds at least one network round trip per batch.

Copy# --- Redis: redis.conf persistence fragment ---
cat > redis-persistence.conf <<'EOF'
# Keep RDB snapshots as a fallback: snapshot if >=1 change in 900s
save 900 1
# Log every write to the append-only file
appendonly yes
# fsync once per second: a crash can lose ~1s of acknowledged XADDs
# (use 'always' to fsync every write, at a large throughput cost)
appendfsync everysec
EOF

# Verify what production is actually running, not what you assume
redis-cli CONFIG GET appendonly
redis-cli CONFIG GET save

# After a critical XADD: wait up to 100 ms for 1 replica to confirm receipt.
# Narrows the failover-loss window; it is not consensus replication.
redis-cli XADD orders '*' order_id 1042 status created
redis-cli WAIT 1 100

# --- Kafka: topic replicated to 3 brokers, 2 must be in sync to accept writes ---
kafka-topics.sh --bootstrap-server broker1:9092 --create \
  --topic orders --partitions 6 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config unclean.leader.election.enable=false

# --- Kafka: producer.properties ---
cat > producer.properties <<'EOF'
# Leader acknowledges only after all in-sync replicas have the record
acks=all
# Retries cannot create duplicates or reorder records within a partition
enable.idempotence=true
EOF

Redis AOF and WAIT settings next to a Kafka topic with replication factor 3, min.insync.replicas=2 and an acks=all idempotent producer. Each block shows the settings that decide how much acknowledged data survives a node failure.

Retention, memory and a sizing example

Redis does not expire stream entries on its own. You cap a stream with MAXLEN or MINID on XADD, or with a separate XTRIM. The approximate form (MAXLEN ~ 1000000) lets Redis trim only whole internal nodes, which is much cheaper than exact trimming. The cost is that the stream may briefly hold somewhat more entries than the cap.

Trimming ignores consumer groups. If a slow consumer has not acknowledged an entry and the trim removes it, the ID stays in the pending list, but there is nothing left to read or claim. No error is raised. A stalled consumer also creates a second problem. If you trim loosely to avoid that data loss, the backlog grows in RAM alongside every other key. Under noeviction, writes fail once maxmemory is reached. Under an allkeys-* policy, Redis can evict the stream key itself.

Kafka retention is set per topic with retention.ms or retention.bytes and is enforced on disk, whether or not consumers have read the data. Replaying days or weeks of events is routine. cleanup.policy=compact keeps the latest value per key indefinitely.

Sizing example (illustrative estimate)

Assumptions: 500-byte events at 2,000 events/s, which is about 86.4 GB/day of raw data. Redis per-entry overhead is assumed at 1.5× with one replica. Kafka overhead is assumed at 1.1× with no compression and replication factor 3.

The totals are similar for a one-day window. The difference is the medium: RAM costs far more per gigabyte than disk. A multi-day replay window is usually impractical in Redis but ordinary in Kafka.

Copy#!/usr/bin/env bash
# Redis: retention is something you enforce on every write or with a periodic trim.

# Cap the stream at roughly 1M entries on each append.
# '~' lets Redis drop whole internal nodes, so the length can briefly exceed the cap.
redis-cli XADD events MAXLEN '~' 1000000 '*' type order_created id 42

# Alternatively, trim by age. Drop entries older than 24h; stream IDs start with a ms timestamp.
CUTOFF_MS=$(( ($(date +%s) - 86400) * 1000 ))
redis-cli XTRIM events MINID '~' "${CUTOFF_MS}-0"

# Trimming ignores consumer groups. Check for unacknowledged entries,
# because any that were trimmed are gone even though their IDs stay in this list.
redis-cli XPENDING events billing-group - + 10

# Watch memory, because the retained stream must fit in RAM.
redis-cli MEMORY USAGE events
redis-cli INFO memory | grep -E 'used_memory_human|maxmemory_human|maxmemory_policy'

# Prefer failed writes over silently evicting the stream key.
redis-cli CONFIG SET maxmemory-policy noeviction

# Kafka: retention is per-topic config, enforced on disk regardless of consumers.
# Keep 7 days (604800000 ms) with 3 replicas.
kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic events --partitions 12 --replication-factor 3 \
  --config retention.ms=604800000

# Add a size cap as well. retention.bytes applies per partition, not per topic.
kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --entity-type topics --entity-name events \
  --add-config retention.bytes=107374182400

# Compacted topic: keep the latest value per key instead of expiring by time.
kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic customer-state --partitions 6 --replication-factor 3 \
  --config cleanup.policy=compact

Redis trims explicitly with XADD MAXLEN ~ or XTRIM MINID ~, so you have to watch pending entries and memory policy, while Kafka enforces time- and size-based retention or compaction through topic configuration on disk.

Scaling out: sharded stream keys vs partitions

A single Redis stream key lives on one shard, so one hot stream is limited by one primary's CPU and network. Adding nodes to Redis Cluster doesn't help that key. To scale, you split the logical stream into N keys, such as orders:0 through orders:15, and route each event in the application. A typical scheme hashes the entity ID with a stable function like CRC32, takes it modulo N, and calls XADD on the resulting key. Every event for one order lands on the same key, which keeps per-entity ordering, while load spreads across shards. Redis gives no ordering across keys. If a consumer needs a global sequence, the application has to build it. Also avoid putting a shared hash tag in braces in these key names, because that forces all of them into one slot.

Kafka builds this routing into the broker model. A topic has a fixed number of partitions, and the default partitioner hashes the record key modulo the partition count. Within a consumer group, each partition is assigned to at most one consumer. Twelve partitions therefore cap a group at twelve active consumers, and any extra consumers sit idle.

Both designs make the shard count hard to change for the same reason: the key-to-shard mapping depends on N. Kafka lets you add partitions, but it cannot remove them. After you add partitions, new records for an existing key may go to a different partition than that key's older records, so per-key ordering breaks across the transition. Resharding Redis stream keys has the same problem. Choose N for peak throughput up front, with headroom.

Operational cost, ecosystem and engineering time

The bigger cost difference is usually in the people and the monitoring, not the hardware. If you already run Redis with replicas and failover, a stream is one more key type on infrastructure your team knows. Kafka is a separate distributed system. It needs its own capacity planning, upgrade procedure, alerting and on-call knowledge, and that cost exists before you send the first event.

Kafka's exactly-once semantics come from idempotent producers and transactions. They hold only for read-process-write flows that stay inside Kafka. A database write or HTTP call made by a consumer can still happen twice, so those side effects need idempotency keys or deduplication either way.

Managed offerings take over broker operations and upgrades for both systems, but you pay for that. You still own consumer lag, schema evolution and partition design.

When to use Redis Streams, when to use Kafka, and how to migrate

…and the operational habits around them. Those costs persist whichever broker you pick, so base the decision on what your workload actually demands.

Decision checklist

Choose Redis Streams if you need a short buffer between services, already run Redis and can tolerate losing a few writes during failover. Avoid it for long retention, because retained events occupy memory. Avoid it for strict durability, because replication is asynchronous. Avoid it for many consumers, because each new consumer group adds pending-entry tracking on a single shard.

Choose Kafka if the log is the system of record, or if other teams will build consumers you can't predict. Avoid it if you are a small team with low volume and no replay needs. In that case you would be operating a cluster to solve problems you don't have. If you only need work distribution without replay, a plain job queue is simpler than either.

You can start on Redis Streams and plan to move to Kafka later. Account for the cost of that migration up front. You will have to rewrite stream IDs and per-message acks as partition offsets and commits. During cutover you will dual-write, and consumers must deduplicate across both systems.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to redis streams vs kafka, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.