What is a CQRS read model built from Kafka events?
In a CQRS read model Kafka architecture, the write side owns the business rules and publishes every state change as an event to a Kafka topic. Those events are the contract between services. Downstream, one or more projectors consume the topic and build read models: materialized views shaped for a specific query. A read model is not a replica of the write schema. An order-summary view might flatten customer name, line-item count, and current status into one row, because that is what the dashboard asks for. A second view over the same events might index orders by shipping region for a fulfillment team.
Every read model is disposable. The log is the durable record, so you can drop a view and rebuild it by replaying events, provided the topic's retention or compaction settings still hold the history you need.
This article treats the projector as a stateful consumer with explicit correctness guarantees, not as a simple event-to-table mapper. The view is only as trustworthy as the projector's handling of ordering, duplicates, and offset commits.
Teams adopt event projections for three reasons:
- Query-shaped storage. Each view uses whatever store and layout fits its access pattern.
- Independent scaling. Read replicas and projectors scale separately from the write path.
- Multiple views over one log. New consumers can be added without changing the producer.
The pattern suits backend and platform engineers working in read-heavy or query-diverse domains, where callers can tolerate eventual consistency measured in milliseconds to seconds.
+--------------------------+ +--------------------------+
| Client / API |--->| Write model |
| sends commands | | owns business rules |
+--------------------------+ +--------------------------+
|
| publish state changes
v
+----------------------------------------------------------+
| Kafka topic: order-events (events are the contract) |
| [e1][e2][e3] ... durable log; replay to rebuild views |
+----------------------------------------------------------+
| |
| consume | consume
v v
+--------------------------+ +--------------------------+
| Projector A: summary | | Projector B: by region |
| stateful consumer: | | stateful consumer: |
| ordering, dedup, offsets | | ordering, dedup, offsets |
+--------------------------+ +--------------------------+
| |
v apply event v apply event
+--------------------------+ +--------------------------+
| View: order summary | | View: orders by region |
| disposable, rebuildable | | disposable, rebuildable |
+--------------------------+ +--------------------------+
^ ^
| query | query
+--------------------------+ +--------------------------+
| Dashboard | | Fulfillment team |
+--------------------------+ +--------------------------+
The write model publishes events to a Kafka topic that serves as the contract, and independent stateful projectors build disposable, query-shaped views that can be rebuilt by replaying the log.
How does a projector keep ordering and idempotency correct?
Kafka only guarantees order within a partition, so the producer must use the aggregate ID (for example, orderId) as the message key. The default partitioner hashes the key, which sends every event for one order to the same partition, and a single consumer in the group reads them in the order they were written. There is no global order across partitions. An OrderShipped for order A and a PaymentFailed for order B can arrive in either order, and the projector must not depend on their relative timing.
A projection-friendly envelope
Every event should carry four fields the projector can trust:
- Entity ID: the aggregate the event belongs to, matching the message key.
- Version: a per-entity counter that increases monotonically and is assigned by the write side when it commits the change.
- Event type: tells the projector which handler to run.
- Timestamp: useful for display and debugging. Do not use it for ordering, because clocks drift.
Idempotency under at-least-once delivery
Consumers will see duplicates after rebalances and retries, so applying an event twice must be harmless. There are two ways to get there:
- Version guard. Store
last_applied_versionnext to each row. Skip any event whose version is at or below it. This handles events whose effects stack, such as incrementing an item count. - Converging upserts. Write absolute values, such as "status = SHIPPED" instead of "add one shipment." Replaying the same event then produces the same row.
Projections that join aggregates
A view that combines an order with its customer reads two keys that can live on different partitions, so their relative order is lost. You have three options:
- Buffer events until the dependent data arrives.
- Store the version of each source on the row and only overwrite with newer data.
- Accept brief gaps, such as a missing customer name, and fill them in when the later event lands.
-- Read model row for one order; last_applied_version is the idempotency guard.
CREATE TABLE order_view (
order_id TEXT PRIMARY KEY, -- entity ID, same value as the Kafka message key
status TEXT NOT NULL,
item_count INTEGER NOT NULL DEFAULT 0,
last_applied_version BIGINT NOT NULL, -- highest per-entity version applied so far
updated_at TIMESTAMPTZ NOT NULL -- event timestamp, for display only
);
-- Handler for OrderShipped: converging upsert. It writes an absolute value,
-- so replaying the same event produces the same row.
-- $1 = order_id, $2 = event version, $3 = event timestamp
INSERT INTO order_view (order_id, status, last_applied_version, updated_at)
VALUES ($1, 'SHIPPED', $2, $3)
ON CONFLICT (order_id) DO UPDATE
SET status = EXCLUDED.status,
last_applied_version = EXCLUDED.last_applied_version,
updated_at = EXCLUDED.updated_at
-- The version guard also stops a stale event from overwriting newer state.
WHERE order_view.last_applied_version < EXCLUDED.last_applied_version;
-- Handler for ItemAdded: the effect stacks, so the version guard is required.
-- Without it, a redelivered event would add the item twice.
-- $1 = order_id, $2 = event version, $3 = event timestamp, $4 = quantity added
INSERT INTO order_view (order_id, status, item_count, last_applied_version, updated_at)
VALUES ($1, 'OPEN', $4, $2, $3)
ON CONFLICT (order_id) DO UPDATE
SET item_count = order_view.item_count + EXCLUDED.item_count,
last_applied_version = EXCLUDED.last_applied_version,
updated_at = EXCLUDED.updated_at
WHERE order_view.last_applied_version < EXCLUDED.last_applied_version;
-- If zero rows are affected, the event is a duplicate or older than the stored
-- state. Treat it as already applied and commit the Kafka offset.
PostgreSQL upserts for an order read model: one handler writes an absolute status and converges on replay, the other increments a counter and relies on last_applied_version to skip duplicate and stale events.
Where should a projector commit consumer offsets?
There are two reasonable places to record progress, and they fail differently.
Commit to Kafka after the database write. The projector applies the event, commits the read-store transaction, then commits the consumer offset. If the process crashes between those two commits, or a rebalance revokes the partition first, the next consumer resumes from the last committed offset and replays events that are already applied. With batched or periodic commits, that duplicate window covers every event since the last commit, not just one. This approach is only correct when every handler is idempotent.
Store the offset in the read store. Write the partition and offset into a progress table in the same transaction as the projected rows. On partition assignment, read the stored position and seek to the next offset, ignoring Kafka's committed offset. You can still commit to Kafka so lag monitoring keeps working. The state and the position move together, so a crash replays nothing that was persisted.
This is effectively-once for that one store only. If a handler sends an email or calls another service, that action sits outside the transaction and can still repeat. Write the intent to an outbox table in the same transaction, or pass an idempotency key the downstream system deduplicates on.
| Aspect | Offset in Kafka after write | Offset in read-store transaction |
|---|---|---|
| Duplicates on crash | Yes, back to last commit | None for the store |
| Coupling | Store unaware of Kafka | Store holds consumer position |
| Complexity | Low; needs idempotent handlers | Seek logic on assignment, progress table |
// Config: enable.auto.commit=false. The read store, not Kafka, is the source of truth for position.
consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
@Override public void onPartitionsRevoked(Collection<TopicPartition> parts) {
// Nothing to flush: each applied event already committed its offset with the state.
}
@Override public void onPartitionsAssigned(Collection<TopicPartition> parts) {
for (TopicPartition tp : parts) {
// SELECT last_offset FROM projector_progress WHERE topic = ? AND part = ?
Long last = progress.lastApplied(tp.topic(), tp.partition());
// Resume after the last persisted event. Kafka's committed offset is ignored.
// If no row exists, the consumer falls back to its committed offset or auto.offset.reset.
if (last != null) consumer.seek(tp, last + 1);
}
}
});
while (running) {
for (ConsumerRecord<String, String> rec : consumer.poll(Duration.ofMillis(500))) {
try (Connection c = dataSource.getConnection()) {
c.setAutoCommit(false);
applyEvent(c, rec); // upsert read-model rows; emails/calls go to an outbox table here, not inline
try (PreparedStatement ps = c.prepareStatement(
"INSERT INTO projector_progress (topic, part, last_offset) VALUES (?, ?, ?) " +
"ON CONFLICT (topic, part) DO UPDATE SET last_offset = EXCLUDED.last_offset")) {
ps.setString(1, rec.topic());
ps.setInt(2, rec.partition());
ps.setLong(3, rec.offset());
ps.executeUpdate();
}
c.commit(); // projected state and position become durable together, or not at all
}
}
// Optional: only so consumer-lag monitoring still reflects progress. Not used for recovery.
consumer.commitAsync();
}
A Java Kafka consumer that writes each event's offset to a progress table in the same PostgreSQL transaction as the projected rows, then seeks to that stored position when partitions are assigned.
How do you implement an order-summary read model end to end?
The example uses the second approach from the table above: the store holds the consumer position, so state and offset commit together.
Event schema
All three events share an envelope: event_id, event_type, schema_version, order_id, order_version (a per-order sequence set by the write side), and occurred_at. Payloads differ. OrderPlaced carries customer_id and currency, ItemAdded carries sku, quantity and unit_price, and OrderShipped carries carrier and shipped_at. Keying the topic by order_id keeps each order's events on one partition, in order.
Read store
order_summary has order_id (primary key), customer_id, status, item_count, total_amount, shipped_at, and last_version. A separate projector_offsets table holds consumer_group, topic, partition and next_offset.
Handler
For each record, open a transaction and load the row's last_version. If the event's order_version is less than or equal to it, skip the change, because it is a duplicate or a replay. Otherwise apply the change and set last_version. In the same transaction, upsert next_offset to the record's offset plus one, then commit. On startup or partition assignment, read projector_offsets and seek to those positions.
The view answers SELECT status, item_count, total_amount FROM order_summary WHERE customer_id = ? ORDER BY .... Index it on customer_id.
Schema evolution
- Unknown fields: ignore them, so producers can add fields without breaking the projector.
- Unknown event types: choose a policy and write it down. Skip types this view doesn't care about. Park unrecognized types that might matter. Fail outright if a missing type would corrupt state.
- Changing events: never rename or remove fields. Bump
schema_versionand upcast old shapes to the current one before the handler runs, so a full replay from offset zero still works.
-- PostgreSQL. Read model plus consumer position live in the same database.
CREATE TABLE order_summary (
order_id text PRIMARY KEY,
customer_id text NOT NULL,
status text NOT NULL, -- 'PLACED' | 'SHIPPED'
item_count integer NOT NULL DEFAULT 0,
total_amount numeric(12,2) NOT NULL DEFAULT 0,
shipped_at timestamptz,
last_version bigint NOT NULL -- highest order_version applied
);
CREATE INDEX order_summary_customer_idx ON order_summary (customer_id);
CREATE TABLE projector_offsets (
consumer_group text NOT NULL,
topic text NOT NULL,
partition_id integer NOT NULL,
next_offset bigint NOT NULL, -- record offset + 1; seek here on assignment
PRIMARY KEY (consumer_group, topic, partition_id)
);
-- One ItemAdded record: order ord-7f3a, order_version 3, partition 2, offset 1041.
BEGIN;
-- Version guard: matches 0 rows if this event was already applied (duplicate or replay).
UPDATE order_summary
SET item_count = item_count + 2,
total_amount = total_amount + 2 * 19.99,
last_version = 3
WHERE order_id = 'ord-7f3a' AND last_version < 3;
-- Advance the position in the same transaction, whether or not the state changed.
INSERT INTO projector_offsets (consumer_group, topic, partition_id, next_offset)
VALUES ('order-summary-v1', 'orders', 2, 1042)
ON CONFLICT (consumer_group, topic, partition_id)
DO UPDATE SET next_offset = EXCLUDED.next_offset;
COMMIT;
-- Query served by the view, using the customer_id index.
SELECT order_id, status, item_count, total_amount
FROM order_summary WHERE customer_id = 'cust-0042' ORDER BY order_id;
The example shows PostgreSQL tables for the order-summary view and the per-partition offsets. It applies one ItemAdded event with a version-guarded UPDATE and records the consumer position in the same transaction. It ends with the query the view serves.
How do you handle poison messages and rebuild projections safely?
After the retry limit, a projector facing a poison message has three options. A poison message is an event the handler cannot apply, such as a malformed payload or a reference to an order the view has never seen. First retry a bounded number of times with backoff, because a transient failure like a database failover looks the same at first.
| Strategy | What happens | Cost |
|---|---|---|
| Skip silently | Offset advances, event is gone | View is permanently wrong and nobody knows |
| Block | Consumer retries indefinitely | Every entity on that partition stalls |
| Dead-letter | Event goes to a dead-letter topic, offset advances | One entity is wrong until the event is reprocessed |
Dead-lettering is usually the right trade-off, but only if writes to the dead-letter topic trigger an alert and someone owns replaying them after the fix. Mark the affected entity as stale. Later events applied on top of the missing one can produce a row that looks plausible but is wrong.
Rebuilding a projection
A rebuild can only recover what Kafka still holds. A retention-based topic deletes events older than its retention window. A compacted topic keeps the latest record per key, which is enough if events carry full state but not if the view aggregates intermediate events. If you need full history, use unlimited retention or an archive.
Replay reads as fast as the consumer allows. That loads the brokers and the read store's write path, so throttle consumption and batch writes.
Run rebuilds blue/green. Start the new projector with its own consumer group and its own table, and let it catch up until lag is near zero. Then compare it with the old view, switch reads over, and retire the old version.
How do you make eventual consistency workable and operate projections at scale?
Treat lag as staleness
Consumer lag is how stale the read model is. Offset lag per partition tells you how much work is queued, but users experience time, so also track the age of the last applied event. Expose that freshness to callers, for example as a field holding the version or event timestamp the view reflects, so clients can tell "not found" apart from "not yet projected."
Read-your-writes
Have the write side return the aggregate's new version. The client sends that version with its next read, and the read side compares it to the version stored in the row. If the row is behind, the read side waits briefly with a timeout, or returns a "pending" response. The cost is coupling: clients must carry versions, and the read API now depends on the write side's versioning scheme. The other option is to route reads that come right after a write to the write store. This is simpler for clients, but it puts read traffic back on the store that CQRS was meant to protect, so keep that window short.
Scaling
Give each projection its own consumer group. Each projection then keeps its own offsets, so a slow or failing projection does not block the others, and you can rebuild one without touching the rest. Parallelism is capped by partition count, because in a consumer group each partition is assigned to only one consumer, and extra instances sit idle. A hot key puts all of its load on one partition, and adding consumers will not help. Adding partitions later changes which partition each key hashes to. New events for a key can then be processed before older events still waiting on its previous partition. To avoid this, drain the projector before repartitioning, or rely on per-key version checks to reject out-of-order updates.
Monitor lag (in offsets and in seconds), handler error rate, DLQ depth, and rebuild duration. Rebuild duration tells you your real recovery time.
When should you avoid CQRS read models, and what are the alternatives?
A projection-based read side earns its cost only when reads diverge from writes. If a service has one or two aggregates, its queries map cleanly onto the write tables, and a single database handles the load, a Kafka read model adds moving parts and removes none. Low read diversity means the write schema is already the right shape for queries.
| Option | Consistency | Cost | Flexibility |
|---|---|---|---|
| Query the write store | Strong | Lowest; nothing new to run | Limited to the write schema and indexes |
| Database read replica | Replica lag, usually asynchronous | Low; managed by the database | Same schema as the primary |
| Database materialized view | Stale until refreshed (scheduled or incremental, depending on the engine) | Low to moderate | Any query the engine supports, inside one database |
| Kafka projection | Eventual; lag measured per consumer group | Highest | Any shape, any storage engine |
Be honest about development cost. Each read model adds a consumer group to monitor, often a storage engine to run and back up, a rebuild procedure to test, and a staleness window the product team must accept. Five read models means five of each.
The pattern saves engineering time when views are disposable and can be rebuilt from retained events. To change a view's schema, you build a new projection alongside the old one, cut traffic over, and drop the old one. You avoid running an in-place migration on a live table. A new query becomes a new projection, not a change to the write schema, so read-side teams ship without coordinating changes to the write model.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to cqrs read model kafka, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.