What a distributed monolith is and how to diagnose one
A distributed monolith is a set of services that cannot be deployed, scaled, or changed independently. It pays the costs of a distributed system: network hops, serialization, partial failures, and more pipelines to run. It keeps the coupling of a monolith. You lose the monolith's in-process calls and single transaction boundary, and you gain none of the autonomy that justified the split.
Service count tells you nothing here. Forty services can be well decoupled, and four can be a distributed monolith. The diagnosis rests on how services change together and fail together. If a change in one routinely forces a change in another, they share a boundary in practice, whatever the repository layout says.
Backend and platform engineers who own multi-service systems should run this diagnosis. Three triggers justify it: releases that need a coordination meeting, recurring cross-team negotiation for ordinary features, and incident reviews where one service's failure takes down unrelated paths.
| Symptom | Evidence to collect | Where to look |
|---|---|---|
| Lockstep or coordinated releases | Deploys of different services clustered together, with required ordering | Deploy history |
| One feature touches several repos | Count of repositories changed per ticket or feature | Commit logs linked to issue IDs |
| Tables written by multiple services | Write access to the same table from more than one service | Schema ownership, database grants |
| Shared domain-model library | Library version bumps that force consumers to upgrade together | Dependency manifests, commit logs |
| Cascading outages | Incidents spanning services with no direct user-facing link | Incident timelines, traces |
Tracing a request to find chatty services
Pick one slow or frequently failing user request and pull its distributed trace. You are looking for three shapes.
Deep synchronous chains. A single request that passes through several services in sequence, each waiting on the next. Every hop adds network latency and one more process that has to be up.
Wide fan-out. One service calls many others to assemble a response. Parallel fan-out keeps latency close to the slowest call. Availability still drops, because every required dependency must succeed.
Deep chain Fan-out
gateway gateway
└─> orders └─> orders ──┬─> pricing
└─> pricing ├─> inventory
└─> inventory ├─> customers
└─> customers └─> shipping
└─> shipping
N+1 remote calls. Look for a repeating sawtooth of identical spans. It means code is calling a service once per item inside a loop. The fix is one batched call that takes a list of IDs, as the code example shows. With 50 items at 20 ms each, the loop costs about a second of waiting. The batched call costs roughly one round trip.
Illustrative estimate
Assume every service has 99.9% availability and adds 20 ms per hop. These are inputs I picked for the example, not measurements.
| Chain depth (n) | Combined availability (0.999^n) | Added latency |
|---|---|---|
| 1 | 99.9% | 20 ms |
| 5 | ~99.5% | 100 ms |
| 10 | ~99.0% | 200 ms |
At ten hops, the request fails roughly ten times as often as any single service does. When a trace shows this pattern, the call graph is behaving like a function call stack that crosses the network.
import requests
PRICING_URL = "" # placeholder service address
# N+1: one remote call per item. In a trace this shows up as a
# sawtooth of identical GET /prices/{id} spans, one after another.
def get_prices_n_plus_one(item_ids):
prices = {}
for item_id in item_ids:
resp = requests.get(f"{PRICING_URL}/prices/{item_id}", timeout=2)
resp.raise_for_status()
prices[item_id] = resp.json()["price"]
return prices # 50 items x ~20 ms per round trip = ~1 s of waiting
# Batched: send all IDs in one request. The trace shows a single span,
# and the cost is roughly one round trip regardless of list length.
def get_prices_batched(item_ids):
resp = requests.post(
f"{PRICING_URL}/prices:batchGet",
json={"ids": list(item_ids)},
timeout=2,
)
resp.raise_for_status()
# Assumed response shape: {"prices": [{"id": "...", "price": 12.5}, ...]}
return {p["id"]: p["price"] for p in resp.json()["prices"]}
# Illustrative estimate (assumed inputs, not measurements):
# chain depth compounds availability multiplicatively and latency additively.
for n in (1, 5, 10):
print(f"depth={n:2d} availability={0.999 ** n:.4f} latency={20 * n} ms")
This example contrasts an N+1 loop that calls a pricing service once per item with a single batched request that takes a list of IDs, and it computes the illustrative 0.999^n availability and 20 ms-per-hop latency figures for chain depths of 1, 5 and 10.
Forms of service coupling and which ones block independent change
That call stack is temporal coupling, but it is only one of four forms. They are not equally harmful. The test is whether a change in one service forces a coordinated change, test cycle, or release in another.
| Coupling type | How it shows up | Blocks independent deploys? | Typical mitigation |
|---|---|---|---|
| Temporal | Synchronous call chains; one service down means many fail | Not directly, but it couples availability and latency | Async messaging, local caches of reference data, timeouts with fallbacks |
| Data/schema | Services read or write the same tables | Yes. A column change breaks every reader | One owning service per dataset, exposed through an API or events |
| Deployment | Services must ship together or in a fixed order | Yes, by definition | Backward-compatible contracts, versioned APIs, expand-then-contract changes |
| Semantic/domain model | A shared Order or Customer type that every service imports |
Yes, once a field change requires synchronized upgrades | Each service owns its own model and translates at the boundary |
Shared libraries need a closer look. Logging, auth clients, and observability libraries are reasonable to share. They encode cross-cutting mechanics, not business meaning, and services can usually upgrade them on their own schedule. A shared domain model library is different. When adding a field means every consumer must bump the dependency and redeploy, you have rebuilt the monolith's compile step across a network.
Do not overcorrect by duplicating everything. If every service writes its own retry logic, token validation, or currency rounding, the copies drift. Behavior then diverges in ways that surface as production inconsistencies, which are harder to find than a version mismatch. Duplicate domain models. Share infrastructure plumbing.
Why decomposition goes wrong: layers, nouns, and split transactions
Most distributed monoliths come from splitting along the wrong axis. Teams cut by technical layer, with a gateway service, a business-logic service, and a persistence service, or by entity, producing a CRUD-only User service, Order service, and Inventory service. Neither cut matches how the business changes. Placing an order touches orders, inventory, and payment in one logical unit, so a rule like "reserve stock only if payment authorizes" now spans three services. The consistency that one database transaction used to guarantee has to be rebuilt with sagas, compensating calls, or distributed locks.
These splits get chosen because they look clean on a diagram, because they suit a scaling plan written before anyone measured load, and because they match the org chart. That last reason matters most. Conway's law says systems mirror the communication structure of the teams that build them. If a frontend team, a backend team, and a database team each own a layer, the services will couple along those team seams. Redrawing service boundaries without changing team ownership usually fails, because the old coordination patterns return within a few release cycles.
Splitting is often the wrong move. It costs more than it returns for a small team, a domain that is still being discovered, or a product without independent scaling or deployment needs. Even careful boundaries reflect only what you understand about the domain today. Moving a boundary between modules is a refactor inside one codebase. Moving it between services means migrating data, versioning APIs, and coordinating deploys. Start coarse, and split only once a boundary has proven stable.
Redrawing boundaries around data ownership
Start from write paths rather than from entities. For each table, list every code path that writes to it and note which other tables it writes in the same transaction. Columns that are always updated together belong together. Then mark the invariants that must hold at commit time, such as "available stock never goes negative" or "a payment is never captured twice." Each invariant needs strong consistency, so all the data it touches must sit inside one boundary. Everything else can tolerate a short delay.
Assign each resulting group exactly one owning service. That service is the only writer to its tables. Every other service gets the data through a contract the owner defines and versions:
- A published event. An
OrderPlacedevent might carryevent_id,schema_version,occurred_at,order_id,customer_id,line_items(SKU, quantity, unit price), andtotal. It describes a business fact, not a row change. - A coarse-grained API. A billing service that used to join directly against the
customersandaddressestables callsGET /customers/{id}/billing-profileinstead. That call returns the name, billing address, and credit status in one response.
Consumers keep derived copies shaped for their own queries. They update those copies from events and accept that the copies may be slightly stale. Because the owner is the single writer, it can change its schema freely as long as the contract holds. A consumer that loses or corrupts its copy can rebuild it by replaying events or re-reading the API. Shared tables give you neither property: any writer can break an invariant, and any reader blocks a schema change.
// Owned and versioned by the Orders service. It describes a business fact, not a row change.
interface OrderPlaced {
event_id: string; // unique per event; consumers dedupe on it
schema_version: number; // owner bumps this on breaking changes
occurred_at: string; // ISO 8601 timestamp
order_id: string;
customer_id: string;
line_items: { sku: string; quantity: number; unit_price: string }[];
total: string; // decimal string avoids float rounding
}
// Owned by the Customers service. Replaces billing's direct joins on customers + addresses.
// GET /customers/{id}/billing-profile
interface BillingProfileV1 {
customer_id: string;
name: string;
billing_address: { line1: string; city: string; postal_code: string; country: string };
credit_status: "good" | "on_hold";
}
// Billing's derived copy, shaped for its own queries. It never writes to Orders tables.
// In-memory here for brevity; in practice this is billing's own database.
type OrderSummary = { order_id: string; customer_id: string; total: string; placed_at: string };
const appliedEventIds = new Set<string>();
const orderSummaries = new Map<string, OrderSummary>();
export function onOrderPlaced(evt: OrderPlaced): void {
if (appliedEventIds.has(evt.event_id)) return; // at-least-once delivery: skip duplicates
if (evt.schema_version !== 1) throw new Error(`unsupported schema_version ${evt.schema_version}`);
orderSummaries.set(evt.order_id, {
order_id: evt.order_id, customer_id: evt.customer_id,
total: evt.total, placed_at: evt.occurred_at,
});
appliedEventIds.add(evt.event_id);
}
// A lost or corrupted copy is rebuilt by replaying the owner's events, not by reading its tables.
export function rebuild(history: Iterable<OrderPlaced>): void {
appliedEventIds.clear();
orderSummaries.clear();
for (const evt of history) onOrderPlaced(evt);
}
Owner-defined contracts (an OrderPlaced event and a billing-profile API response) and a billing-side projection that keeps an idempotent, rebuildable derived copy instead of reading the owner's tables.
Remediation techniques, alternatives, and their trade-offs
Once boundaries follow data ownership, the remaining work is removing the coupling that still crosses them. The default response to a slow, fragile call graph is often more infrastructure, such as a service mesh, retries, or another cache tier. Another common response is splitting services further. Both treat symptoms. The techniques below remove the dependency itself, and each one has a cost you should accept deliberately.
| Technique | Coupling it removes | Cost it introduces |
|---|---|---|
| Merge services into one service or a modular monolith | Deployment and temporal coupling between services that always change together | Teams contend over a shared codebase and pipeline |
| Asynchronous events | Temporal coupling: the caller no longer waits on the callee | Eventual consistency, duplicated data, ordering and idempotency concerns, harder debugging |
| Local read models or caches | Runtime dependency for reads | Stale data, plus the need to rebuild or reconcile |
| Coarse-grained APIs | Chatty round trips and leaked internal structure | Larger payloads and some fields that callers ignore |
| Contract tests with versioned interfaces | Lockstep releases across services | Tooling overhead and the burden of supporting old versions |
Merging is a legitimate outcome. If two services are always changed and deployed together, the network boundary between them adds latency and failure modes without buying any independence. Module boundaries inside one process can enforce the same separation at lower cost.
Events need the most care. Brokers commonly deliver at least once, so consumers will see duplicates. A consumer should record each processed event ID in the same transaction as its state change and skip any event ID it has already seen. The example below does this.
import sqlite3
SCHEMA = """
CREATE TABLE IF NOT EXISTS processed_events (event_id TEXT PRIMARY KEY);
CREATE TABLE IF NOT EXISTS customer_totals (
customer_id TEXT PRIMARY KEY,
total_cents INTEGER NOT NULL
);
"""
def handle_order_placed(conn, event):
# One transaction covers the dedup record and the state change.
# Either both commit or neither does, so a crash cannot leave a
# half-processed event behind.
with conn: # commits on success, rolls back on exception
cur = conn.execute(
"INSERT OR IGNORE INTO processed_events (event_id) VALUES (?)",
(event["event_id"],),
)
if cur.rowcount == 0:
return False # already processed: a redelivery, so skip it
conn.execute(
"""INSERT INTO customer_totals (customer_id, total_cents) VALUES (?, ?)
ON CONFLICT(customer_id)
DO UPDATE SET total_cents = total_cents + excluded.total_cents""",
(event["customer_id"], event["amount_cents"]),
)
return True
if __name__ == "__main__":
conn = sqlite3.connect(":memory:")
conn.executescript(SCHEMA)
evt = {"event_id": "evt-7f3a", "customer_id": "cust-42", "amount_cents": 1999}
print(handle_order_placed(conn, evt)) # True: applied
print(handle_order_placed(conn, evt)) # False: duplicate ignored
print(conn.execute("SELECT total_cents FROM customer_totals").fetchone()) # (1999,)
An event consumer stores each event ID in the same transaction as its state change, so a duplicate delivery from an at-least-once broker is detected and skipped and the total is not counted twice.
Migrating incrementally and measuring the impact on cost and engineering time
Pick the coupling point that hurt most in your diagnosis, such as the service pair that always deploys together or the table three services write to, and fix that one first. Avoid a big-bang rewrite. A rewrite delays all feedback until the end and stacks every risk into a single cutover.
Untangle data before code. First, enforce one writer per table: route every other service's writes through the owner's API, even if that adds a hop for now. Then split the schema. Either dual-write to the old and new stores while you compare results, or backfill the new store and switch reads over gradually. This is slow and risky work. For a while you will run two schemas, sync jobs and extra network calls, so complexity rises before it falls.
Track progress with numbers you can count:
- coordinated deploys per month
- repos touched per feature
- call-chain depth on key requests
- services affected per incident
These numbers drive cost. Less release coordination, less waiting on other teams and fewer cascading incidents all reduce ongoing engineering time. Those savings have to pay back the upfront migration effort.
Illustrative estimate (assumed numbers, not a benchmark): Monthly savings = (coordinated releases before − after) × engineers per release × hours each. Assume (4 − 1) × 6 × 3 = 54 hours/month. With an assumed migration cost of 600 hours, payback = 600 ÷ 54 ≈ 11 months.
New boundaries only hold if the organization matches them. Give each service a single owning team, make the owner approve schema changes, and publish versioned contracts that other teams can depend on.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to distributed monolith, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.