Microservice Event Replay Backfill: Rebuild State Safely

What an event replay backfill is and when it is the right data backfill strategy

A microservice event replay backfill means pointing a consumer back at the start of a service's event log, or at a chosen point in it, and re-consuming those events to rebuild state. Treat it as a production operation, not a one-off script. It reads large volumes from shared infrastructure, writes to real datastores, and competes with live traffic. It needs the same planning, monitoring, and rollback path as a deploy.

Teams reach for this data backfill strategy in three situations:

• New projections. A new read model or query service needs history it never saw.
• Corrupted read models. A bug wrote bad derived state, and the events underneath are still correct.
• Schema or storage migrations. You are moving a projection to a new table layout or a different database, and rebuilding is cleaner than transforming in place.

Ownership sits with the backend engineers who own the projection logic and the platform engineers who run the broker and its capacity. Neither group can run a replay safely alone.

Preconditions

Replay only works if the log is the source of truth. If some state was written directly to the database and never emitted as an event, replay cannot recover it.

The log must also still hold the history you need. Time- or size-based retention may have deleted old segments. Compaction keeps only the latest record per key, so intermediate transitions are gone. Tiered storage can keep history available, but it changes read throughput and cost.

Replay reproduces history exactly, including its bugs. If events were wrong when they were produced, fix them by appending compensating events. Do not edit the log. Other consumers have already acted on the original records.

When to avoid replay and what the alternatives are

Replay only works if the log still holds everything the projection needs. Four situations break that assumption.

• Compacted topics. Compaction keeps the latest record per key and discards earlier ones. A projection that counts transitions or walks a state machine will silently compute a different result from the compacted log.
• Retention shorter than the window. If the topic keeps only recent data and the read model needs a longer history, the oldest events no longer exist. The rebuild starts from an incomplete history.
• Logs too large to replay in time. Throughput sets a hard floor on how long a rebuild takes. If a full rebuild would take longer than you can run a stale read model, replay is not an option.
• Events were never the system of record. If a service writes to its database and then publishes separately, without an outbox, some events were lost or reordered. The database is authoritative, and replaying the topic reproduces those gaps.

Snapshots shorten replay, but you then have to trust that the snapshot itself was correct. If a projection bug was present when the snapshot was taken, the bug is preserved in every rebuild that starts from it. Version your snapshots and keep enough log behind them to fall back to a full replay.

Designing projections that rebuild state deterministically and idempotently

A replay-safe projection is a pure function of the event stream: given the same events in the same order, it must produce the same state every time. That rules out reading the wall clock, generating random IDs, or calling other services during the projection. Each of those returns a different answer on replay than it did originally. Derive timestamps from the event's own time field, and derive identifiers from the event payload or a deterministic hash of it.

Writes should be idempotent and keyed by entity ID. Store the last-applied offset or event version alongside each entity's state, and skip any event at or below it. This lets a crashed replay resume without double-counting, and lets the rebuild overlap safely with live traffic.

Keep effectful reactions out of the projection. Emails, webhooks, payments, and downstream publishing belong in separate consumers that are disabled or pointed at a sink during a rebuild. Replaying a "payment captured" event should update a balance, not charge a card again.

Kafka guarantees order only within a partition. If one entity's events are keyed inconsistently and land on different partitions, replay can apply them out of order. A version check then either rejects valid events or accepts stale ones.

Old events will not match current schemas. Use upcasters or version-aware handlers, and test them against real historical samples, not synthetic fixtures.

Retrofitting determinism into existing handlers is hard. The rebuilt state may legitimately differ from production, so expect to explain those differences rather than assume they are bugs.

Copyimport uuid

# Fixed namespace so derived IDs are identical on every replay (never uuid4()).
LEDGER_NS = uuid.UUID("6f1c2a9e-3b4d-4e8f-9a01-2c3d4e5f6a7b")

UPSERT = """
INSERT INTO account_balance (account_id, balance_cents, last_event_version, updated_at)
VALUES (%(account_id)s, %(amount)s, %(version)s, %(occurred_at)s)
ON CONFLICT (account_id) DO UPDATE
SET balance_cents      = account_balance.balance_cents + EXCLUDED.balance_cents,
    last_event_version = EXCLUDED.last_event_version,
    updated_at         = EXCLUDED.updated_at
-- Idempotency guard: an event at or below the stored version is a no-op.
WHERE account_balance.last_event_version < EXCLUDED.last_event_version
"""

def project_payment_captured(conn, event: dict) -> None:
    """Pure state projection: no clock reads, no random IDs, no remote calls."""
    params = {
        "account_id": event["account_id"],
        "amount": event["amount_cents"],
        "version": event["version"],          # per-entity sequence number from the event
        "occurred_at": event["occurred_at"],  # event time, not now()
    }
    # Deterministic ID derived from the payload, stable across replays.
    entry_id = uuid.uuid5(LEDGER_NS, f"{event['account_id']}:{event['version']}")

    with conn.transaction():
        conn.execute(UPSERT, params)
        conn.execute(
            "INSERT INTO ledger_entry (id, account_id, amount_cents, occurred_at) "
            "VALUES (%s, %s, %s, %s) ON CONFLICT (id) DO NOTHING",
            (entry_id, params["account_id"], params["amount"], params["occurred_at"]),
        )
    # Deliberately absent: charging the card, sending receipts, publishing events.
    # Those live in a separate consumer that is disabled during a rebuild.

A replay-safe projection handler that derives time and IDs from the event, writes idempotently keyed by account ID with a last-applied version guard, and leaves all side effects to a separate consumer.

How Kafka replay works: consumer groups and offset resets

Kafka replay never moves data. The log is already there. You only change where a consumer group's committed offsets point. Don't rewind the group that serves production, because its handlers would re-apply history to live state while new events keep arriving. Run the rebuild under its own group instead. A fresh group ID with auto.offset.reset=earliest reads from the start of retained data with no tooling. A long-lived dedicated group, such as orders-projection-rebuild, gives you more control, because you can set its offsets explicitly.

Use kafka-consumer-groups.sh with --reset-offsets, and scope it with --group and --topic. Pick the starting point that matches the bug:

• --to-earliest replays everything still retained. Use it for full rebuilds, but first confirm that retention or compaction hasn't already dropped the events you need.
• --to-datetime starts at the first offset at or after a timestamp. This fits when you know roughly when a bad deploy shipped.
• --to-offset sets an exact position. This is useful when you have recorded a known-good checkpoint for each partition.

Run the command first with --dry-run, which prints the target offset for each partition. Check those offsets, then repeat the same command with --execute. The tool rejects the reset unless the group is inactive, meaning no members are connected. This prevents a running consumer from committing its own position over the new offsets. Stop every rebuild consumer before resetting.

During the replay, --describe on the group shows current offset, log-end offset, and lag for each partition. When lag stays near zero, the rebuild has caught up to live traffic.

Copy#!/usr/bin/env bash
# Rewind a dedicated rebuild group. The live production group is never touched.
set -euo pipefail

BOOTSTRAP="kafka-1.internal:9092"
GROUP="orders-projection-rebuild"
TOPIC="orders.events"
# Roughly when the bad deploy shipped. Format: YYYY-MM-DDTHH:mm:SS.sss
START="2026-10-20T14:00:00.000"

# 1. Confirm the group is inactive. Stop every rebuild consumer first,
#    or the reset is rejected. This should list no members.
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
  --describe --group "$GROUP" --members

# 2. Dry run: prints the target offset for each partition and changes nothing.
#    Alternatives: --to-earliest for a full rebuild, or
#    --topic "$TOPIC:0" --to-offset 18250 to start a partition at a recorded checkpoint.
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
  --group "$GROUP" --topic "$TOPIC" \
  --reset-offsets --to-datetime "$START" --dry-run

# 3. After checking those offsets, run the same command with --execute to commit them.
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
  --group "$GROUP" --topic "$TOPIC" \
  --reset-offsets --to-datetime "$START" --execute

# 4. Start the rebuild consumers, then watch progress. The LAG column
#    (log-end offset minus current offset) should trend toward zero.
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
  --describe --group "$GROUP"

Resets a dedicated rebuild consumer group to a timestamp by checking that the group is inactive, previewing the new offsets with --dry-run, applying them with --execute, and then tracking replay progress through per-partition lag.

The blue-green rebuild: replay alongside live, verify, then cut over

Catching up to the head of the log is only half the job. The other half is proving the rebuilt state is correct before anything reads from it. The safest way to do that is to build the new state next to the live store, not on top of it.

Copy                   ┌──────────────┐
                   │  Event log   │
                   └──────┬───────┘
              ┌───────────┴───────────┐
              ▼                       ▼
     ┌─────────────────┐     ┌─────────────────┐
     │  Live consumer  │     │ Rebuild consumer│
     │ (existing group)│     │   (new group)   │
     └────────┬────────┘     └────────┬────────┘
              ▼                       ▼
        ┌───────────┐           ┌───────────┐
        │ Store v1  │           │ Store v2  │
        └─────┬─────┘           └─────┬─────┘
              └──────────┬────────────┘
                         ▼
                 ┌───────────────┐
                 │  Read switch  │◄── service reads
                 └───────────────┘

The rebuild consumer writes into a new table, index, or database. Once its lag is near zero, compare the two stores while both are being written:

• Row counts per entity type or partition.
• Hashes over sampled entities, computed over normalized fields.
• Business invariants, such as balances summing to ledger totals or no orders left without a customer.

Set the discrepancy threshold before you look at the results. If you set it afterward, you will be tempted to explain away the first mismatch you find.

Sampling has a blind spot. It can miss divergence in one field that affects a small slice of entities. Comparing every entity catches that, but it costs a full scan of both stores.

Put cutover behind a reversible switch, such as a config flag or a view alias. Keep v1 updating until you are confident in v2, so rolling back means flipping the switch rather than running another replay.

An in-place rebuild avoids running two stores, but there is no baseline to verify against and no fallback if the projection is wrong. The cost of the parallel approach is real: storage and write load roughly double until you retire v1.

Copy                      ┌──────────────────┐
                      │    Event log     │
                      └─────────┬────────┘
                ┌───────────────┴───────────────┐
                ▼                               ▼
      ┌───────────────────┐           ┌───────────────────┐
      │  Live consumer    │           │ Rebuild consumer  │
      │ (existing group)  │           │ (new group, from  │
      │                   │           │  start of log)    │
      └─────────┬─────────┘           └─────────┬─────────┘
                ▼                               ▼
      ┌───────────────────┐           ┌───────────────────┐
      │     Store v1      │           │     Store v2      │
      │ (live, kept for   │           │ (new table/index) │
      │  rollback)        │           │                   │
      └─────────┬─────────┘           └─────────┬─────────┘
                │     ┌───────────────────┐     │
                ├────►│   Verification    │◄────┤
                │     │ row counts,       │     │
                │     │ sampled hashes,   │     │
                │     │ invariants vs     │     │
                │     │ preset threshold  │     │
                │     └─────────┬─────────┘     │
                │               ▼ pass          │
                │     ┌───────────────────┐     │
                └────►│    Read switch    │◄────┘
                      │ flag or alias,    │
                      │ reversible        │
                      └─────────┬─────────┘
                                ▼ service reads

The same event log feeds a live consumer and a separate rebuild consumer into two stores, which are compared against a threshold set in advance before reads move through a reversible switch, while Store v1 keeps updating so you can roll back.

Operational runbook: throttling, checkpointing, side effects, and rollback

Treat the replay as a production change with its own runbook, not a script someone runs on a Friday.

1. Dry run on a staging copy. Replay a representative slice of the topic into a staging database. This shows projection bugs, schema mismatches and realistic throughput before production load is at stake.
2. Cap throughput and size batches deliberately. Limit records per poll and commit in batches the target database absorbs without lock contention. Unthrottled replay competes with live consumers for broker I/O, network bandwidth and database connections. That can trip lag alerts on services that have nothing to do with the backfill.
3. Watch shared resources, not just the replay. Track broker disk and network, target-database write latency, and consumer lag for live groups. Slow down when any of them move. Throttling has a price: the catch-up window grows. On a high-write topic, a capped replay may barely outpace new arrivals, so estimate the window before you start.
4. Checkpoint progress. Commit offsets only after the projected writes are durable. A crash then resumes from the last checkpoint instead of restarting from the beginning.
5. Suppress side effects. Disable emails, webhooks and outbound events behind a replay flag. Downstream services already consumed those effects from the original run. Re-emitting them creates duplicates, or worse, conflicting state in systems you cannot roll back.
6. Define cutover and rollback up front. Set cutover criteria before you start, such as replay lag near zero, reconciliation checks passing, and error rates flat. Keep the old store writable and routable until the new one has run cleanly for an agreed period. Rollback is then a routing change, not a second backfill.

Copyimport json, time
from confluent_kafka import Consumer, KafkaException

REPLAY_MODE = True          # suppresses emails, webhooks, outbound events
BATCH_SIZE = 500            # sized from the staging dry run
MAX_RECORDS_PER_SEC = 2000  # throughput cap; lower it if live lag or DB latency rises

consumer = Consumer({
    "bootstrap.servers": "kafka-1.internal:9092",
    # Dedicated group: live consumer groups and their offsets are untouched
    "group.id": "orders-projection-replay-2026-10",
    "enable.auto.commit": False,       # we checkpoint manually
    "auto.offset.reset": "earliest",   # first run starts at the beginning
})
consumer.subscribe(["orders.events"])

def write_projection(events): ...   # placeholder: one DB transaction, idempotent upserts
def emit_side_effects(events): ...  # placeholder: notifications, outbound events

while True:
    started = time.monotonic()
    msgs = consumer.consume(num_messages=BATCH_SIZE, timeout=1.0)
    if not msgs:
        continue
    for m in msgs:
        if m.error():
            raise KafkaException(m.error())  # stop; never commit past an unread record
    events = [json.loads(m.value()) for m in msgs]

    write_projection(events)        # durable before the offset moves
    if not REPLAY_MODE:
        emit_side_effects(events)   # downstream already saw these in the original run

    # Checkpoint: a crash after this point resumes here, not from offset 0
    consumer.commit(asynchronous=False)

    # Throttle: keep the average rate at or below MAX_RECORDS_PER_SEC
    min_duration = len(msgs) / MAX_RECORDS_PER_SEC
    elapsed = time.monotonic() - started
    if elapsed < min_duration:
        time.sleep(min_duration - elapsed)

A throttled replay consumer that uses its own consumer group, commits offsets only after projected writes are durable so it resumes on failure, and skips side effects while the replay flag is on.

Impact on development cost and engineering time

Replay is not free, and the costs show up in places teams often don't budget for.

Storage. A full rebuild requires the complete event history. That means longer topic retention or tiered storage, and you pay for those bytes even in months when nobody replays.

Duplicate capacity. During a blue-green rebuild you run two copies of the read store. The new one also absorbs a burst of writes far heavier than its normal load, so it may need temporary extra capacity.

Replay duration. Rebuild time grows with log size, so it lengthens every month the log does. Estimate it before you commit to a cutover window:

Illustrative estimate (assumed numbers, not a benchmark) Replay time ≈ total events ÷ sustained throughput cap Assumed: 2,000,000,000 events, throttled to 20,000 events/s to protect the target store. 2,000,000,000 ÷ 20,000 = 100,000 s ≈ 28 hours, before retries or pauses.

If that number is unacceptable, you have two levers: raise the throughput cap, which needs more partitions and target capacity, or start from a snapshot and replay only the tail.

Engineering effort. Making handlers deterministic and idempotent is real work. You have to remove wall-clock reads, external lookups, and non-idempotent writes.

The payoff is repeatability. A rehearsed rebuild replaces the one-off migration scripts that are written under pressure and then thrown away. You can add new read models by consuming the existing log, without touching source databases or coordinating with their owners. When state is corrupted, you repair it with a procedure you have already tested, not with hand-written SQL during an incident.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice event replay backfill, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.