What does it mean to rebuild a read model by replaying Kafka topics?

A read model is a query-optimized store, such as a Postgres table, a search index or a cache, that a service builds from events instead of writing to it directly. The code that consumes events and updates that store is a projection. Event replay means running a projection over a topic's history again, from the earliest retained offset, to rebuild the store from scratch.

Teams rebuild for three reasons:

  • A projection bug wrote wrong data.
  • A new query shape needs fields or groupings that the current store never captured.
  • A schema or store migration moves the data to a new layout or a different database.

This guide is for backend and platform engineers who run event-driven services with Kafka-fed read stores.

Before you plan a rebuild, check that the source topics still hold the history the projection needs. Three settings decide this: retention.ms, cleanup.policy, and whether older segments were moved to tiered or archived storage that your consumers can still read.

Topic setup What survives Rebuildable projections
delete, finite retention Events inside the retention window Only projections that need nothing older than the window
delete, infinite retention (-1) Every event All projections
compact Latest record per key Current-state lookups only

Compaction deletes intermediate values. Counts, audit trails and time-series aggregates depend on every event, so you cannot rebuild them from a compacted topic.

Long or infinite retention costs broker storage, or tiered storage if you offload older segments. Short retention saves that money but means a full rebuild is not possible. Decide which projections must stay rebuildable, and set retention on those topics to match.

#!/usr/bin/env bash
# Pre-rebuild check: does the source topic still hold the history the projection needs?
set -euo pipefail

BOOTSTRAP="kafka-1.internal:9092"   # placeholder broker address
TOPIC="orders.events"               # source topic for the read model

# 1. Effective retention, cleanup policy and tiered-storage settings.
#    --all includes broker defaults, not only per-topic overrides.
#    cleanup.policy=compact (or compact,delete) means intermediate values are gone,
#    so counts, audit trails and time-series aggregates cannot be rebuilt.
#    retention.ms=-1 means time-based deletion is disabled.
#    retention.bytes can also delete old segments, so check it too.
kafka-configs.sh --bootstrap-server "$BOOTSTRAP" \
  --entity-type topics --entity-name "$TOPIC" --describe --all \
  | grep -E 'cleanup.policy|retention.ms|retention.bytes|remote.storage.enable|local.retention'

# 2. Earliest available offset per partition (--time -2 = earliest).
#    A value above 0 means records were already deleted from that partition.
#    Compare these against the offsets your projection originally started from.
kafka-get-offsets.sh --bootstrap-server "$BOOTSTRAP" \
  --topic "$TOPIC" --time -2

# 3. Timestamp of the oldest record still readable in partition 0.
#    The rebuild can only start from here, so make sure it is old enough.
#    Repeat for each partition, because retention runs per partition.
kafka-console-consumer.sh --bootstrap-server "$BOOTSTRAP" \
  --topic "$TOPIC" --partition 0 --offset earliest \
  --max-messages 1 --property print.timestamp=true --property print.value=false

A shell check that reads a topic's effective cleanup policy, retention and tiered-storage settings, finds the earliest offset still held in each partition, and shows the oldest readable timestamp, so you can confirm the full history exists before you start a replay.

Why is resetting offsets on the live consumer group risky?

The obvious way to rebuild is to stop the projection's consumer group, reset its offsets to the earliest available position, clear or keep the existing tables, and restart. It needs no extra infrastructure, and the store keeps serving traffic the whole time. That last part is the problem.

If you truncate first, readers see an empty store that fills in over minutes or hours. If you don't truncate, it is less obvious but just as bad. Replayed events overwrite current rows with older versions, so a customer's balance or an order's status moves backward until the replay catches up. Different entities sit at different points in history at the same moment. Any query that joins or aggregates across them returns combinations that never existed.

Idempotent handlers don't fix this. Idempotency guarantees that the final state is correct once replay finishes. It says nothing about what readers see while replay is in progress.

A failure halfway through is worse. The consumer has committed offsets somewhere in the middle of the log, and the old read model has already been overwritten. There is nothing to roll back to. The only way forward is to finish the replay, which may be the very step that is failing.

The in-place reset is cheaper and faster: one store, one consumer group, no cutover. If a projection has no live readers, or stale reads are acceptable for a while, it can be the right call. Otherwise you are giving up rollback and consistent reads to save storage. The blue/green approach makes the opposite trade.

How does a blue/green projection rebuild work?

…keeps the running projection untouched and builds its replacement beside it.

  1. Deploy the new projection under a new consumer group ID, for example orders-view-v2, with auto.offset.reset=earliest. A group with no committed offsets starts from the oldest retained record, so you get a full replay without touching the offsets of the live group.
  2. Write into a new versioned store, such as a new table, index or keyspace (orders_view_v2). The old projection keeps serving reads from v1, and rollback means not switching.
  3. Watch consumer lag for the v2 group. When lag stays near zero, v2 is processing the same live events as v1.
  4. Switch reads to v2 with a config flag or an alias or view swap. Keep v1 running for a while so you can switch back.
  5. Retire v1: stop its consumers, delete the consumer group, and drop the old store.
                 ┌── orders-view-v1 ──► orders_view_v1 ──┐
 orders.events ──┤                                       ├──► [read switch] ──► API
 (shared topics) └── orders-view-v2 ──► orders_view_v2 ──┘

The cost is that storage and write load on the read store roughly double during the backfill. Both groups write the live traffic, and v2 also writes the entire history as fast as it can. On a shared database or search cluster, that extra load can slow down v1's reads. For very large datasets, a second full copy may not fit, or the replay may take longer than you can run two copies side by side. In those cases, rebuild one partition or tenant range at a time, or build v2 on separate infrastructure and move to it afterward.

#!/usr/bin/env bash
# Run the v2 projection beside v1, wait for its lag to drain, then retire v1.
set -euo pipefail
BOOTSTRAP="kafka-1.internal:9092"

# Step 1: v2 consumer settings. A new group.id has no committed offsets,
# so auto.offset.reset=earliest replays from the oldest retained record.
cat > orders-view-v2.properties <<'EOF'
group.id=orders-view-v2
auto.offset.reset=earliest
enable.auto.commit=false
EOF
# (Start the v2 projection with these settings; it writes to orders_view_v2.)

# Step 3: total lag for v2, read from the LAG column found by header name.
# "-" means a partition has no committed offset yet, so v2 is not ready.
v2_lag() {
  kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
    --describe --group orders-view-v2 2>/dev/null |
  awk '$0 ~ /LAG/ { for (i = 1; i <= NF; i++) if ($i == "LAG") c = i; next }
       c && NF >= c { if ($c == "-") unknown = 1; else sum += $c }
       END { print (unknown ? "unknown" : sum + 0) }'
}

until lag=$(v2_lag) && [ "$lag" != "unknown" ] && [ "$lag" -lt 1000 ]; do
  echo "orders-view-v2 lag: ${lag:-unknown}, still backfilling"
  sleep 60
done
echo "v2 has caught up. Switch reads to orders_view_v2 now."

# Step 5, run later, after reads have used v2 long enough that you no longer
# need to roll back: stop the v1 consumers first, because a group with active
# members cannot be deleted. Then drop orders_view_v1 in the read store.
# kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
#   --delete --group orders-view-v1

A shell sketch of the blue/green rebuild. It configures a new consumer group that replays from earliest, polls kafka-consumer-groups.sh --describe until the v2 group's total lag falls below a threshold, and deletes the old v1 group only after reads have moved and v1 is stopped.

How do you reset Kafka offsets safely when you need to?

Sometimes a reset is justified: a "new" v2 group that already committed offsets during a failed first attempt, or an in-place rebuild where you have explicitly accepted that the read model will be stale or empty while it catches up. The tool for this is kafka-consumer-groups.sh --reset-offsets. It rewrites the group's committed offsets directly, so you choose the starting point before any consumer reconnects.

Option Starting point Typical use
--to-earliest Oldest offset still retained on each partition Full rebuild, if retention covers the whole history
--to-datetime First offset whose record timestamp is at or after the given time Replaying from just before a known bad deploy
--to-offset An exact offset you specify Precise recovery of a single partition

--to-datetime depends on record timestamps. If producers set unreliable timestamps, the resolved offsets can surprise you. --to-offset applies the same number to every partition in scope, so combine it with a partition-qualified --topic (for example, orders:2).

Always scope the reset. Use --topic to name exactly what you intend. Use --all-topics only if you have confirmed every topic the group has committed offsets for. It catches topics you forgot the group subscribed to.

The group must have no active members. The tool refuses to reset a group with live consumers, because a running consumer would overwrite the new offsets on its next commit. Stop every instance first.

Run --dry-run before --execute. The dry run prints the planned new offset for each topic-partition without committing anything. Check that every expected partition appears, that no unexpected topics are listed, and that the offsets match what you expect, such as near the log start for --to-earliest. Then rerun the same command with --execute.

BOOTSTRAP=kafka-1.internal:9092
GROUP=orders-read-model-v2

# 1. Confirm no consumers are connected; the reset is refused while members are active.
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
  --describe --group "$GROUP" --members

# 2. Plan a full rebuild of one topic. --dry-run prints offsets but commits nothing.
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --group "$GROUP" \
  --reset-offsets --topic orders --to-earliest --dry-run

# Example output shape (values are placeholders):
# GROUP                  TOPIC   PARTITION  NEW-OFFSET
# orders-read-model-v2   orders  0          0
# orders-read-model-v2   orders  1          0
# orders-read-model-v2   orders  2          18452   <- log start moved by retention
# Check that every partition is listed, that no other topics appear, and that
# NEW-OFFSET matches the log start rather than the current end.

# Alternative: replay from just before a bad deploy (format YYYY-MM-DDTHH:mm:SS.sss).
# kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --group "$GROUP" \
#   --reset-offsets --topic orders --to-datetime 2026-11-10T14:30:00.000 --dry-run

# Alternative: set an exact offset on a single partition only.
# kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --group "$GROUP" \
#   --reset-offsets --topic orders:2 --to-offset 20000 --dry-run

# 3. Apply the same command with --execute after the plan looks right.
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --group "$GROUP" \
  --reset-offsets --topic orders --to-earliest --execute

This example checks that the consumer group has no active members, previews a scoped offset reset with --dry-run, shows how to read the planned NEW-OFFSET values, and then applies the same reset with --execute.

How do you make projections idempotent, ordered and side-effect free during event replay?

Replay means every event your projection has ever handled will arrive again, sometimes more than once. The projection code has to produce the same state whether it sees an event once, twice, or after a crash mid-batch.

Make writes idempotent. Use upserts keyed by entity ID, not inserts or increments. "Set balance to X" survives redelivery. "Add X to balance" does not.

Track what you have applied. Store the last applied offset or event version per entity or per partition, in the same transaction as the state change. Skip anything at or below it. This turns at-least-once delivery into effectively-once state.

Respect ordering limits. Kafka orders events only within a partition. Nothing is guaranteed across partitions or topics. A projection that joins two topics may see them interleaved differently on replay than it did live. The final state can differ and still be valid, so decide in advance which differences are acceptable.

Isolate side effects. Emails, external API calls and downstream publishes must not fire during a backfill. The best option is to keep them out of state-building code entirely. Otherwise, gate them behind an explicit replay flag.

Handle old schemas. Upcasters or tolerant readers must be deployed before the backfill starts, because the oldest events will arrive first.

Define a path for poison messages. Send them to a dead-letter topic, or skip and log them. Infinite retries stall a partition indefinitely.

Know the cost. Idempotency adds complexity and per-write overhead. When it is subtly wrong, it does not fail loudly. It corrupts data silently.

-- Read model row. last_event_version stores the per-entity version of the
-- last event applied. The topic is keyed by account_id, so all events for one
-- account sit in one partition and arrive in order.
CREATE TABLE account_balance (
  account_id         text        PRIMARY KEY,
  balance_cents      bigint      NOT NULL,
  last_event_version bigint      NOT NULL,
  updated_at         timestamptz NOT NULL DEFAULT now()
);

-- Applied once per event. The event carries the resulting balance
-- ("set to X"), not a delta ("add X"), so redelivery cannot double-count.
-- State and version change in one statement, so they commit atomically.
INSERT INTO account_balance (account_id, balance_cents, last_event_version)
VALUES ('acct-1042', 125000, 17)
ON CONFLICT (account_id) DO UPDATE
SET balance_cents      = EXCLUDED.balance_cents,
    last_event_version = EXCLUDED.last_event_version,
    updated_at         = now()
-- Skip any event at or below the stored version: duplicates and stale
-- replays update 0 rows instead of overwriting newer state.
WHERE account_balance.last_event_version < EXCLUDED.last_event_version;

-- Replaying the same event (version 17) again is a no-op: 0 rows updated.
-- An older event (version 12) arriving late is also a no-op.
INSERT INTO account_balance (account_id, balance_cents, last_event_version)
VALUES ('acct-1042', 98000, 12)
ON CONFLICT (account_id) DO UPDATE
SET balance_cents      = EXCLUDED.balance_cents,
    last_event_version = EXCLUDED.last_event_version,
    updated_at         = now()
WHERE account_balance.last_event_version < EXCLUDED.last_event_version;

A PostgreSQL upsert keyed by entity ID with a version guard, so a duplicate or out-of-date event updates zero rows during replay instead of corrupting the read model.

How do you control backfill throughput and cut over to the new read model?

Throughput

Parallelism is capped by partition count: a consumer group with more instances than partitions leaves the extras idle. Within that limit, the target store is usually the bottleneck, so batch writes instead of writing one row per event. Drop or defer secondary indexes and non-essential constraints until catch-up finishes, then build them once. Incremental index maintenance on every insert is far more expensive.

Throttle the backfill. Old segments are usually no longer in the page cache, so a fast replay pushes brokers into disk reads that compete with live consumers.

Illustrative estimate, assumed inputs: 2 billion retained events, 24 partitions, and 2,000 events/s per instance after batching. That gives 48,000 events/s, or about 11.6 hours. If you throttle to half that rate to protect the brokers, plan for roughly a day.

Cutover

Define "caught up" as lag below a threshold, such as a few hundred events per partition, held for a sustained period like 30 minutes. A single low reading is not enough. Then verify the new model against the old one with row counts, per-key checksums, and sampled record comparisons. Flip reads through a configuration flag or a table or index alias so the switch is instant and reversible. Keep the old store for a defined rollback window.

If you find a projection bug mid-run, fix it and restart from scratch. That is wasted compute, but it is predictable. Debugging an in-place rebuild that is half-corrupted costs engineer-days, and you have no clean baseline to compare against.

When should you avoid Kafka replay, and what are the alternatives?

Replay only works if the log still contains everything the projection needs. Three conditions rule it out:

  • Retention or compaction removed history. Time-based retention may already have deleted early events, so a replay quietly builds a partial view. Compaction keeps only the latest record per key. That works for state-shaped projections but loses the transitions a counting or aggregating projection depends on.
  • Events never carried the field. If the new read model needs data that was never published, replaying the log cannot produce it.
  • Replay is too slow or expensive. Even with complete history, reprocessing years of data across many partitions can take longer than the business can wait.
Approach Rollback safety Read consistency Infrastructure cost Development effort History required
Blue/green replay High: old model untouched Stable until cutover High: duplicate store and consumers Medium Full log
In-place offset reset Low Broken during rebuild Low Low Full log
Source-database backfill High if written to a new store Point-in-time, then must catch up from the log Medium: read load on the primary High: second code path must match projection logic None in Kafka
Snapshot plus partial replay Medium to high Consistent if the snapshot offset is stored atomically Medium High From the snapshot onward

Teams that run many projections over shared topics, or change their read models often, should invest in replay readiness. That means a retention policy that matches rebuild needs, idempotent projections, and a replay mode that suppresses side effects. This adds upfront cost to every projection, but it turns later rebuilds into routine operations. A team with one stable read model and a reliable source database can usually rely on a database backfill instead.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to rebuild read model kafka replay, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.