Diagnosing a dual-writer entity with a field-by-writer matrix

A dual-writer entity exists when two or more services can change the same row, document or aggregate, and nobody can say with certainty which one is authoritative for a given field. The symptoms are familiar: lost updates, status values flipping back, and long arguments over whose bug it is.

Start by listing every write path, not just the obvious API handlers. Include direct SQL in repositories, ORM saves (which often write every column, not only the ones that changed), batch and reconciliation jobs, admin tools and support consoles, and schema or data migrations. For each path, record the fields it touches and the business event that triggers the write. Then pivot the list into a field-by-writer matrix.

Field Checkout Fulfillment Billing Admin tool
line_items, totals Order placed — — Manual correction
shipment_status — Shipped, delivered — Manual override
payment_state Initial "pending" — Authorized, captured, refunded Refund override

The matrix usually shows that the conflict is not about the data. It is about an overloaded entity. "Order" here is really three concepts from three bounded contexts: a purchase, a shipment and a payment. They share one identifier and one table. The overlapping cells, such as checkout setting payment_state and admin tools touching everything, are where the disputes come from.

Backend and platform engineers who own the shared schema should run this exercise. They can see every connection and job against the database, and they are the ones who pay the cost of incidents caused by ambiguous ownership.

What the single writer principle means and why it fixes the dispute

The single writer principle states that every fact in the system has exactly one service allowed to change it. A fact is a field or an aggregate, such as an order's status, a customer's credit limit, or a shipment's tracking number. Any other service that needs that fact reads it through the owner's API, consumes the owner's events, or queries a replicated read model built from those events. It never writes it.

Ownership is assigned per fact, not per table. A customers table can hold fields owned by three different bounded contexts. The matrix from the previous step shows where those lines fall. The table name is an accident of history, while the business rules that govern each field determine who owns it.

The principle works because invariants need a single point of enforcement. If billing and account management can both change credit_limit, each one enforces its own version of the rules, and neither is accountable when the result is wrong. With one writer, every validation, approval, and side effect for that fact lives in one codebase, owned by one team.

Optimistic locking and last-write-wins do not fix this. They stop lost updates at the row level, but they say nothing about whether the second write was a valid business decision. A version check can reject a stale write and the caller can then retry it blindly. The two decisions still conflict, and the conflict surfaces later as a semantic bug.

Apply the principle when:

  • a field carries business invariants
  • writers live in different deployables or are owned by different teams
  • incidents trace back to "who changed this?"

How it works: three resolution patterns for shared entities

Pattern 1: Split into context-owned aggregates

Replace the single orders row with one aggregate per bounded context: OrderPlacement (Sales), Shipment (Fulfillment), and Payment (Billing). They share the order ID, and each lives in its owner's store. This removes contention on the row and the field because no two services ever update the same record. Each aggregate also carries only the invariants its context needs, so the Fulfillment team can change its rules without coordinating with Sales.

Pattern 2: Convert foreign writes into commands

When a non-owner needs to change state, such as Support cancelling an order, it asks the owner instead of writing directly. It can send a synchronous API call when it needs an immediate answer, or an async command message when it can wait. The owner checks the request against its invariants, for example "cannot cancel after shipment," and either applies it or rejects it. Validation happens in one place, so a rejection is an explicit outcome the caller must handle rather than a silent overwrite.

Pattern 3: Project derived values from events

A customer-facing order status such as "Paid, awaiting shipment" is a function of other services' state. Storing it as a writable field invites every service to update it. Instead, build it in a read-only projection that consumes owner events. If the projection drifts, you replay events to rebuild it. No service ever writes the status directly.

 Support ──CancelOrder (cmd)──► Sales: OrderPlacement ─┐
                                                        │ OrderPlaced
 Billing: Payment ──────────── PaymentCaptured ─────────┤ OrderCancelled
                                                        │
 Fulfillment: Shipment ─────── ShipmentDispatched ──────┤
                                                        ▼
                                          OrderStatus projection
                                          (read-only, rebuildable)
 +-------------+
 |   Support   |
 | (non-owner) |
 +-------------+
    |        ^
    |        |  CancelOrder: sync API call or async command message
    |        |  reply: accepted, or rejected ("cannot cancel after shipment")
    v        |
 +--------------------+ +--------------------+ +--------------------+
 | Sales              | | Billing            | | Fulfillment        |
 | OrderPlacement     | | Payment            | | Shipment           |
 | validates commands | | own invariants     | | own invariants     |
 +--------------------+ +--------------------+ +--------------------+
     |      |               |      |               |      |
     v      |               v      |               v      |
 [sales db] |          [billing db]|           [fulfil db]|
            | OrderPlaced          | PaymentCaptured      | ShipmentDispatched
            | OrderCancelled       |                      |
            v                      v                      v
 +------------------------------------------------------------------+
 | event stream: owner events only, consumed by read models         |
 +------------------------------------------------------------------+
                                 |
                                 v
              +----------------------------------------+
              | OrderStatus projection (read-only)     |
              | e.g. "Paid, awaiting shipment"         |
              | rebuilt by replaying events            |
              | no service writes this field           |
              +----------------------------------------+

Each context owns its aggregate and store (Pattern 1), Support changes an order only by sending a command that Sales validates against its invariants (Pattern 2), and the customer-facing status is a read-only projection built from owner events (Pattern 3).

Implementing reliable state propagation: outbox, events and idempotent consumers

The projection only holds together if every owner publishes its changes reliably. The obvious approach is to commit the row and then publish to the broker, but that is a dual write. If the process crashes between the two steps, the database and the event stream disagree, and nothing records that an event was lost.

The transactional outbox

The owner writes the state change and an outbox row in the same local transaction. A typical outbox row holds event_id, aggregate_id, aggregate_version, event_type and payload. Either both writes commit or neither does. A separate relay process then reads unpublished outbox rows, sends them to the broker, and marks them as sent. The relay can crash and retry, so delivery is at-least-once. Duplicates are expected, and lost events are not.

Idempotent, order-tolerant consumers

Consumers must therefore treat redelivery as normal. Each event carries the aggregate's version, which the owner increments on every write. The consumer applies an upsert guarded by that version: insert the row, or update it only WHERE projection.version < incoming.version. A duplicate or a late, older event matches zero rows and is dropped. This guard also handles reordering across partitions. Events with no natural version need a processed-event_id table checked inside the consumer's own transaction.

Operational cost

This pattern adds moving parts you have to run:

  • The relay: a process that must be deployed, scaled and kept running.
  • Dedupe storage: tables that grow and need retention policies.
  • Outbox pruning: cleanup of rows that have already been published.
  • Monitoring: alerts on outbox lag and the age of the oldest unsent row, because a stuck relay fails silently.
-- PostgreSQL 13+ (gen_random_uuid is built in from 13).
-- Owner service: the outbox lives in the same database as the owned table.
CREATE TABLE outbox (
  event_id          uuid PRIMARY KEY,
  aggregate_id      text        NOT NULL,
  aggregate_version bigint      NOT NULL,
  event_type        text        NOT NULL,
  payload           jsonb       NOT NULL,
  created_at        timestamptz NOT NULL DEFAULT now(),
  published_at      timestamptz             -- NULL until the relay sends it
);
-- The relay polls only unsent rows, so index just those.
CREATE INDEX outbox_unsent ON outbox (created_at) WHERE published_at IS NULL;

-- The state change and its event commit together or not at all (no dual write).
BEGIN;
WITH updated AS (
  UPDATE orders
     SET status = 'shipped', version = version + 1  -- owner bumps version on every write
   WHERE id = 'ord_123'
  RETURNING id, version, status
)
INSERT INTO outbox (event_id, aggregate_id, aggregate_version, event_type, payload)
SELECT gen_random_uuid(), id, version, 'OrderShipped',
       jsonb_build_object('status', status, 'version', version)
  FROM updated;
COMMIT;

-- Consumer service: version-guarded upsert into its own projection.
-- $1..$3 come from the event. Duplicates and stale events match zero rows.
INSERT INTO order_projection (order_id, status, version)
VALUES ($1, $2, $3)
ON CONFLICT (order_id) DO UPDATE
   SET status = EXCLUDED.status, version = EXCLUDED.version
 WHERE order_projection.version < EXCLUDED.version;

-- Monitoring: age of the oldest unsent row; alert when it keeps growing.
SELECT now() - min(created_at) AS oldest_unsent_age FROM outbox WHERE published_at IS NULL;

In PostgreSQL, an order update and its outbox event are written in one transaction, the consumer applies a version-guarded upsert that drops duplicate or out-of-order events, and a query reports outbox lag.

Migrating off a shared table without a big-bang rewrite

The order of these steps matters. Each one makes the next safe to take.

  1. Freeze new direct writes. Add no new code paths that write the shared table. Enforce this with code review and a check in CI that flags new SQL against it.
  2. Reroute existing writers one at a time. Move each non-owner's writes behind the owner's API, one service per release. A regression then points to a single change and can be rolled back cleanly.
  3. Confirm with audit logs. Log every write to the table along with the database role that issued it. Leave the logging on for at least one full business cycle, including month-end jobs and rarely run batch scripts. Do not trust a grep of the codebase alone.
  4. Enforce with grants. When the logs stay clean, remove write access from each non-owner with statements like REVOKE INSERT, UPDATE, DELETE ON orders FROM billing_svc; and GRANT SELECT ON orders TO billing_svc;. If the owner should change only certain columns, standard SQL supports column-level grants: GRANT UPDATE (status, total) ON orders TO order_svc;.
  5. Move the owned columns into the owner's schema. Leave a read-only view behind for consumers that have not migrated yet.

Ownership defined field by field on one shared table is only a convention. Any role that can write the row can write every column in it. A forgotten script or a hurried hotfix will eventually break the convention. Grants or separate schemas make the database reject those writes.

Settle the organizational side the same way. Assign ownership to the team accountable for the entity's invariants, meaning the team that gets paged when those invariants break. Record the decision in an ADR, and publish the owner in the data catalog entry so future disputes can be resolved by looking it up.

-- Step 4: enforce ownership once audit logs show no non-owner writes.
-- Non-owners keep read access only.
REVOKE INSERT, UPDATE, DELETE ON orders FROM billing_svc;
REVOKE INSERT, UPDATE, DELETE ON orders FROM shipping_svc;
GRANT SELECT ON orders TO billing_svc, shipping_svc;

-- The owner may change only the columns it is accountable for.
REVOKE UPDATE ON orders FROM order_svc;
GRANT INSERT, SELECT ON orders TO order_svc;
GRANT UPDATE (status, total) ON orders TO order_svc;

-- Step 5: move the owned columns into the owner's schema.
CREATE SCHEMA order_svc;

CREATE TABLE order_svc.order_state (
  order_id BIGINT PRIMARY KEY REFERENCES orders (id),
  status   VARCHAR(32)   NOT NULL,
  total    DECIMAL(12,2) NOT NULL
);

INSERT INTO order_svc.order_state (order_id, status, total)
SELECT id, status, total FROM orders;

-- Only the owner writes the new table.
GRANT SELECT, INSERT, UPDATE, DELETE ON order_svc.order_state TO order_svc;

-- Drop the old columns only after the owner's code writes to the new table.
ALTER TABLE orders DROP COLUMN status;
ALTER TABLE orders DROP COLUMN total;

-- Read-only view for consumers that have not migrated yet.
CREATE VIEW orders_with_state AS
SELECT o.*, s.status, s.total
FROM orders o JOIN order_svc.order_state s ON s.order_id = o.id;

GRANT SELECT ON orders_with_state TO billing_svc, shipping_svc;

Standard SQL that revokes write access from non-owner roles and limits the owner to specific columns. It then moves the owned columns into the owner's schema and leaves a read-only view for consumers that have not migrated.

When to avoid splitting, and the alternatives

Splitting is the wrong move when both services must enforce a single invariant atomically over the same fields. Take a booking service and a billing service that both write reservation.status, where a reservation may only be confirmed once payment is captured. If the rule must hold at every instant, not eventually, then whoever writes those fields needs both facts in one transaction. Distributing that check means sagas, compensations, or distributed locks. All of these add failure modes to protect a rule that one database transaction would enforce for free. The friction usually means the bounded context boundary is drawn in the wrong place. Merge the two services, or move the invariant into one owner that the other service calls.

Over-splitting shows up in three ways:

  • Chatty interactions. Each user request fans out into several synchronous calls, and the services start failing together.
  • Distributed joins. Queries that used to be one SQL join become application-side stitching or duplicated read models.
  • Coordination overhead. Releases, schema changes and incident response all need multiple teams for what is really one feature.
Option Consistency Coupling Enforcement Operational cost
Shared table with locking Strong, within one database High: shared schema Convention only Low at first; contention grows
Field-level ownership on one table Strong per row Medium: shared schema, disjoint columns DB grants or code review Low
Merged service Strong, single transaction Low externally Natural: one codebase Low; larger deployable
Split aggregates with events Eventual Low Strong: separate stores High: outbox, replay, monitoring

Use split aggregates with events only when the business can tolerate a window in which the two copies disagree.

Impact on cost and engineering time

Moving to a single writer is not free, and the costs show up before the savings do. Every write that used to be a local UPDATE now crosses a network boundary to the owner, which adds a hop and a runtime dependency. If the owner is down, callers either fail or need async commands, a durable queue and retries with backoff to keep working. Eventual consistency is acceptable for display status, but inventory counts and account balances cannot show stale values without risking oversells or overdrafts. That pushes those fields toward synchronous reads from the owner or reservation patterns. Workflows that were one local transaction become sagas with compensating actions, and every compensation path needs its own tests. The outbox relay, dead-letter handling and lag monitoring are infrastructure that someone has to run.

The organisational cost is easy to underestimate. When a team takes ownership, it also takes the pager for that entity and gets feature requests from teams that can no longer change the data themselves.

The savings come from removing ambiguity:

Concern Dual writers Single owner
Conflicting-write incidents Recurring, span teams Confined to one service
Root-causing a bad value Search every writer's logs One write path to inspect
Schema changes Coordinated across teams Owner decides, publishes contract
Ownership arguments Reopened per incident Settled once, documented
Runtime coupling Shared database Network calls, queues

The trade is worth making when the cost of debugging contested data is higher than the cost of running the messaging infrastructure it takes to avoid it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservices data ownership, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.