What is event upcasting and why are event schema changes harder than API changes?
Event upcasting is a read-time transformation that converts a stored event from the schema it was written with into the schema your current code expects. The bytes in the event store never change. When a consumer loads an OrderPlaced event written as version 1, an upcaster reshapes it into version 3 in memory, and the consumer sees only version 3. History stays exactly as it was recorded. Only the view of it moves forward.
Changing an event schema is harder than changing an API, for three reasons.
- Events last much longer than requests. An API payload exists for one request and response, and once clients upgrade, the old shape is gone. An event is a fact about the past, kept for as long as the stream's retention allows, which in event sourcing is often forever. Code you deploy today must still read events written years ago.
- You may not know every reader. Projections, analytics jobs, other teams' services and future replays all read the same stream. Any of them can rebuild state from the first event at any time. You cannot coordinate a cutover with readers you cannot list.
- The store is append-only for a reason. Audit trails, regulatory retention and debugging all rely on the log being exactly what was written. Rewriting stored events in place removes that guarantee. It also creates a risky window in which some data is migrated and some is not. That is why "just migrate the data" is not a safe default.
This matters most for teams running event-sourced aggregates, read-model projections that are rebuilt from scratch, or long-retention streams that new consumers start reading from the beginning.
Which event schema changes need an upcaster and which are backward compatible?
Before choosing a strategy, classify every change to an event schema. The question is whether a consumer written against the new schema can still read every historical event correctly, including the oldest one in the stream.
| Change | Example | Backward compatible? | Handling |
|---|---|---|---|
| Add optional field with a default | discountCode added to OrderPlaced |
Yes | None; readers apply the default when it is absent |
| Rename field | custId → customerId |
No | Upcaster copies the value to the new name |
| Change type | quantity string → integer |
No | Upcaster parses and converts |
| Split event | AddressChanged → ShippingAddressChanged + BillingAddressChanged |
No | Upcaster emits several events from one |
| Merge events | ItemAdded + ItemPriced → LineItemAdded |
No | Stateful upcaster or migration |
| Change meaning, same shape | amount in cents → amount in major units |
No | Upcaster rescales, gated on version |
Additive optional fields are safe because old events lack the field, and every reader already treats that absence as "use the default." A field is only optional if no consumer logic depends on it being present. If a projection silently treats a missing value as zero, the change is not really additive.
Schema registry compatibility checks catch the structural rows: removed fields, changed types, new required fields without defaults. They cannot catch the last row. A field that keeps its name and type but changes unit, timezone, or sign convention passes every structural check. Old events then flow into consumers that misread them, and nothing raises an error. Track semantic changes explicitly with a version bump, because a compatibility check will report them as safe.
How does an upcaster chain work at read time?
An upcaster is a pure function that takes an event type and a version N payload and returns the version N+1 payload. Upcasters are chained during deserialization. A stored OrderPlaced v1 passes through the v1→v2 upcaster, then the v2→v3 upcaster, and only the v3 shape reaches domain code. Each upcaster knows about exactly one step. Adding v4 means writing one new function, not editing every consumer.
The chain depends on knowing the stored version, so record it explicitly at write time. One option is to encode it in the event type name (OrderPlaced.v2). The other is a schemaVersion field in the envelope or message headers. The envelope field keeps type names stable for routing and subscriptions. Do not infer the version from payload shape, such as "if currency is missing, it must be v1." That heuristic fails as soon as two versions share a shape or an optional field is legitimately absent.
Upcasters must be deterministic. Do not call other services, read current state, or use the clock. Replay depends on the same stored bytes producing the same event every time. An upcaster that looks up a customer's current region would rewrite history with today's data, so a projection rebuilt next month would silently differ from one rebuilt today.
Define a shared envelope standard across teams: event ID, type, schema version, occurrence time, and payload. With that in place, a single registry implementation can look up upcasters by (type, version) and apply them in order until no further step exists. Every service then evolves events the same way.
from dataclasses import dataclass, replace
@dataclass(frozen=True)
class Envelope:
event_id: str
type: str # stable name, used for routing and subscriptions
schema_version: int # recorded at write time, never inferred from payload
occurred_at: str
payload: dict
def order_placed_v1_to_v2(p: dict) -> dict:
# v2 added currency. All v1 orders were USD, so a constant default is safe.
return {**p, "currency": "USD"}
def order_placed_v2_to_v3(p: dict) -> dict:
# v3 renamed customer -> customer_id. Pure reshaping, no lookups or clock.
rest = {k: v for k, v in p.items() if k != "customer"}
return {**rest, "customer_id": p["customer"]}
# Each entry knows exactly one step: (type, from_version) -> upcaster
UPCASTERS = {
("OrderPlaced", 1): order_placed_v1_to_v2,
("OrderPlaced", 2): order_placed_v2_to_v3,
}
def upcast(env: Envelope) -> Envelope:
# Apply steps in order until no upcaster exists for (type, version)
while (step := UPCASTERS.get((env.type, env.schema_version))) is not None:
env = replace(env, payload=step(env.payload),
schema_version=env.schema_version + 1)
return env
stored = Envelope("evt-7f3a", "OrderPlaced", 1, "2024-03-02T10:15:00Z",
{"order_id": "ord-1001", "customer": "cus-42", "amount": 4999})
print(upcast(stored)) # domain code receives schema_version=3 with customer_id and currency
A shared envelope with an explicit schema_version field, two single-step pure upcasters for OrderPlaced, and a registry that chains them during deserialization so a stored v1 event reaches domain code as v3.
Where should upcasting live: the event store or the consumer boundary?
That raises the question of where the registry runs. Upcasting only helps code that reads events through it. Anything that deserializes raw bytes on its own still sees the original shape.
Event store read path
For aggregates and projections inside one service, put the upcaster chain in the event store's read path. The service owns the store, its repositories and its projection runners, so every reader passes through one deserializer. Domain code then only handles the latest version, and stored events stay immutable. A rehydrated aggregate never needs to know that a three-year-old event once had a different field name.
Consumer boundary
Integration events on a broker are different. Other teams consume them, often in other languages, and they don't share your registry. Upcasting in your producer's read path does nothing for a consumer reading the topic directly. Shipping an upcaster library to every consumer couples their release cycle to your schema history. Replaying a topic from the beginning also shows every historical shape at once.
| Concern | Event store read path | Broker consumer boundary |
|---|---|---|
| Who reads | Your aggregates and projections | Services owned by other teams |
| Shared deserializer | Yes, one per service | No, each consumer has its own |
| Who owns the upcaster | Producing service | Each consumer, or nobody |
| Typical policy | Upcast to latest on read | Stable public contract, versioned explicitly |
For this reason, cross-service events usually get a separate policy. Keep internal domain events private and free to evolve. Translate them into a deliberately designed public contract that changes rarely. When that contract must break, publish the old and new versions in parallel, on separate topics or with a version header. Retire the old version only once every consumer has migrated.
How do you implement upcasters for hard cases like new required fields, splits and merges?
When a new version adds a required field, the upcaster has to supply a value for events recorded before that field existed. A plain default is risky because downstream code cannot tell it apart from a real observation. A revenue report that sums discount_amount will treat a backfilled 0 as "no discount was given" when the truth is "nobody recorded it." Mark the value as derived. A sibling field such as discount_source: "upcast_default", or a per-event derived_fields list, lets analytics filter it out and lets business rules refuse to act on it.
Splitting one stored event into several, for example OrderPlaced into OrderCreated plus PaymentAuthorized, introduces two problems:
- Identity. Each output needs a stable ID. Derive it deterministically from the source event ID and the output's position, such as
{source_id}:0and{source_id}:1. Consumers that deduplicate by ID will then see the same IDs on every replay. - Ordering. The outputs must keep their original stream position, with the split events adjacent and in a fixed order. A new sequence number would interleave them with unrelated events.
Merging several events into one is harder. The upcaster has to buffer events until the group is complete. It also has to decide which source ID and timestamp the merged event inherits, and handle streams where the group was never finished.
Some new meanings cannot be reconstructed from old data. If the old event never captured whether a refund was customer-initiated, the upcaster must not guess. Model the gap explicitly with an unknown variant or a nullable field, and leave the decision to consumers.
from dataclasses import dataclass
from typing import Any
@dataclass(frozen=True)
class Event:
id: str
type: str
version: int
position: tuple[int, int] # (original stream position, index within a split)
data: dict[str, Any]
derived_fields: tuple[str, ...] = () # filled by an upcaster, not observed
def upcast_order_placed_v1(e: Event) -> list[Event]:
"""Split stored OrderPlaced v1 into OrderCreated v2 + PaymentAuthorized v1."""
d, (pos, _) = e.data, e.position
created = Event(
id=f"{e.id}:0", # deterministic, so replays produce the same IDs
type="OrderCreated", version=2, position=(pos, 0),
data={"order_id": d["order_id"], "items": d["items"],
# v1 never recorded discounts: 0 is a placeholder, not a fact
"discount_amount": 0},
derived_fields=("discount_amount",),
)
payment = Event(
id=f"{e.id}:1",
type="PaymentAuthorized", version=1, position=(pos, 1),
data={"order_id": d["order_id"], "amount": d["total"]},
)
# Both outputs keep the source stream position, adjacent and in fixed order
return [created, payment]
def observed_discount_total(events: list[Event]) -> int:
# Analytics skips values the upcaster supplied instead of treating them as history
return sum(
e.data["discount_amount"] for e in events
if e.type == "OrderCreated" and "discount_amount" not in e.derived_fields
)
A one-to-many upcaster splits OrderPlaced v1 into two events. It gives them deterministic IDs and keeps them at the source event's stream position. The backfilled discount is marked as derived, so the report excludes it.
How should you test and roll out upcasters without breaking consumers?
Test against real history, not hand-written samples
Upcasters fail on the payloads nobody remembered: events written by an old producer build, fields serialized as strings before they became numbers, nulls where the schema said "required." Keep golden fixtures: real historical payloads, scrubbed of personal data, stored per version (for example fixtures/OrderPlaced/v1/, v2/, v3/). Add a fixture whenever you find an odd shape in production.
The core test is simple. For each fixture file, load the raw payload with its recorded version, run it through the full chain, and assert three things:
- The result validates against the current schema.
- It reports the current version number.
- Known values survived the transformation unchanged.
Test the whole chain, not just individual steps. A v1 event must reach v4 through every intermediate upcaster, because that is the path it takes in production.
Unit tests miss interactions between events. Before deploying, replay the full event stream, or a representative slice, into fresh projections in CI or staging. Then compare the row counts and key aggregates against the current projections. An upcaster that silently drops a field shows up here as a balance that no longer reconciles.
Roll out readers before writers
Deploy upcasters and tolerant readers to every consumer first. Switch producers to the new version only after that. If you reverse the order, consumers start receiving v3 events they cannot parse. They either crash, dead-letter the events, or worse, deserialize them partially and persist wrong state. The first two cost you a backlog. The third costs you a data-repair incident: finding the affected records, reconstructing the correct values, and rebuilding projections. Fixture tests and replay checks take minutes in CI. The repair takes days of engineering time.
import json
from pathlib import Path
import pytest
from jsonschema import validate
from events.upcasting import upcast, CURRENT_VERSION # project's upcaster chain
from events.schemas import ORDER_PLACED_SCHEMA # JSON Schema for the current version
FIXTURES = Path(__file__).parent / "fixtures" / "OrderPlaced"
# One test case per scrubbed historical payload, across v1/, v2/, v3/ ...
CASES = sorted(FIXTURES.glob("v*/*.json"))
@pytest.mark.parametrize("path", CASES, ids=lambda p: f"{p.parent.name}/{p.name}")
def test_fixture_upcasts_to_current_schema(path):
# Fixture format: {"version": 1, "payload": {...}, "expected": {...}}
fixture = json.loads(path.read_text())
# Run the FULL chain from the recorded version, as production does.
version, event = upcast("OrderPlaced", fixture["version"], fixture["payload"])
assert version == CURRENT_VERSION
validate(instance=event, schema=ORDER_PLACED_SCHEMA) # raises on mismatch
# Values known from the original payload must survive unchanged.
for field, expected in fixture["expected"].items():
assert event[field] == expected, f"{field} changed during upcast"
def test_every_old_version_has_fixtures():
# Fail if an older version has no real-history coverage at all.
present = {p.name for p in FIXTURES.iterdir() if p.is_dir()}
missing = {f"v{n}" for n in range(1, CURRENT_VERSION)} - present
assert not missing, f"no fixtures for {sorted(missing)}"
A pytest suite runs every stored historical OrderPlaced fixture through the full upcaster chain. It asserts that each result reports the current version, validates against the current schema and keeps its known values, and it fails if any older version has no fixtures.
When should you avoid upcasting, and what are the alternatives?
Every upcaster runs on every read of an old event, including every full replay, so its cost grows with history. Illustrative estimate, not a measurement: assume 200 million stored events, 40% of them at old versions, each passing through three upcasters at 5 µs per step. That adds 200M × 0.4 × 3 × 5 µs = 1,200 seconds, about 20 CPU-minutes, to every projection rebuild, and the figure keeps rising as history grows. The code cost is worse. Each upcaster is permanent, and someone must still understand it and keep it tested years after its authors leave.
| Alternative | Beats indefinite upcasting when |
|---|---|
| Weak schema / tolerant reader | Changes are additive and consumers can ignore unknown fields and default missing ones |
| Copy-and-transform migration | Old versions are numerous and the chain is long or semantically tangled |
| Snapshot rebuilding | Reads mostly start from recent state, so old events are rarely replayed |
| Event store compaction | Only the latest state per key matters and full history has no audit value |
Copy-and-transform rewrites history. The original events are what auditors and incident investigators rely on, so keep them archived. Global ordering must be preserved across the copy, and stream IDs or positions may change, which breaks consumer checkpoints. You need either downtime or a dual-read cutover, where consumers read the new stream while you verify it against the old.
Retiring upcasters
Track which schema versions still exist in storage, per event type. Once a version's count reaches zero, through migration or archival, delete its upcaster. The chain should shrink over time.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to event versioning upcasting, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.