Incidental vs essential duplication: what you are actually trying to remove
Most attempts to reduce duplicated code across services fail at the first step: they treat every pair of similar-looking blocks as the same problem. They are not. Two kinds of duplication look the same in a diff but behave very differently over time.
Incidental duplication is code that matches today but encodes separate business rules. Take a discount calculation in billing and one in promotions. Each has a different owner, a different reason to change, and different stakeholders asking for changes. They match only because both teams started from the same assumption. Sooner or later one of them changes and the other doesn't.
Essential duplication is code that has to stay identical, because if it drifts something breaks. Examples are the signing scheme on an internal API, the format of a correlation ID, or a retry policy that downstream systems rely on. If two copies disagree, you get a bug at the boundary between services.
Only essential duplication is worth sharing. If you abstract incidental duplication, you join two rules that want to move apart. When the first divergence shows up, the shared function gets a boolean parameter. The second divergence adds a mode flag. Before long the function is a pile of conditionals that nobody can safely change, because every caller depends on a different path through it. With only two examples to go on, you usually can't tell which kind of duplication you have, so the abstraction is a guess.
That's why copying on purpose is a reasonable default. A copy costs you some repeated edits. A wrong abstraction costs you coordination across teams on every change. Teams pursue code reuse to get consistency where it matters and to fix things in one place. Both of those payoffs only exist for essential duplication.
// --- Essential duplication: one shared definition ---
// shared/correlation-id.ts (published as an internal package)
// Every service must emit and parse the same format, or tracing
// breaks at service boundaries. Drift here is a bug, so share it.
import { randomUUID } from "node:crypto";
export const CORRELATION_HEADER = "x-correlation-id";
const PREFIX = "req_";
export function newCorrelationId(): string {
return PREFIX + randomUUID();
}
export function isCorrelationId(value: string): boolean {
return value.startsWith(PREFIX) && value.length === PREFIX.length + 36;
}
// --- Incidental duplication: deliberate copies ---
// billing/discount.ts (owned by the billing team)
// Contract discount for invoiced customers. Changes when finance
// renegotiates terms.
export function billingDiscount(subtotal: number, rate: number): number {
return Math.round(subtotal * rate * 100) / 100;
}
// promotions/discount.ts (owned by the growth team)
// Campaign discount. Identical body today, but the reason to change
// is different: caps, stacking rules and seasonal campaigns.
// Merging the two into applyDiscount(subtotal, rate, isPromo,
// capMode) would be the first step toward a flag-driven mess.
export function promoDiscount(subtotal: number, rate: number): number {
return Math.round(subtotal * rate * 100) / 100;
}
The correlation ID format is shared because every service has to agree on it, while the two discount functions are copied on purpose because they look the same today but belong to different owners and change for different reasons.
Duplication vs coupling: the four kinds of coupling reuse creates
Each reuse mechanism removes duplication by adding a dependency between services. That dependency is the real cost, so name it before you adopt the mechanism. Four kinds show up repeatedly.
Build-time coupling
Consumers compile against your code and pick up its transitive dependencies. If a shared library pins an HTTP client or serializer, every service that imports it is held to that pin, and conflicts appear in builds the library authors never see.
Release coupling
A fix only takes effect after each consumer upgrades and redeploys. When old and new versions can't coexist, for example because they encode a message differently, teams have to upgrade in lockstep. A coordinated release like that is the main symptom of a distributed monolith.
Runtime coupling
Shared services, sidecars and gateways ship no source to consumers. Instead they sit on the request path, so their latency, outages and deploy schedule become part of your service's behaviour.
Semantic coupling
A shared domain model imposes one meaning of "Customer" or "Order" on contexts that disagree about it. This is the hardest kind to undo because the assumptions spread into business logic, not just imports.
| Mechanism | Build-time | Release | Runtime | Semantic |
|---|---|---|---|---|
| Copy | None | None | None | Only if you copy domain types |
| Shared library | Yes | Yes, on breaking changes | None | Depends on contents |
| Generated code | Stubs only | Low; each service regenerates | None | Limited to the wire contract |
| Platform service | None | Through API versioning | Yes, network hop | Through its API |
| Sidecar | None | Owned by the platform team | Yes, local process | Low |
What belongs in shared libraries and what does not
The test for a shared library is not "do two services contain similar code?" but "would both services want this code to change at the same moment, for the same reason?" Infrastructure and cross-cutting concerns usually pass that test. Logging setup, trace context propagation, retry and backoff policies, auth token validation, and HTTP client defaults like timeouts and connection limits encode organizational policy rather than any one team's product decisions. When the security team tightens token validation or the observability team changes a propagation header, every service should pick up the change. Divergence here causes real damage, such as broken traces or inconsistent auth checks.
Shared domain models and business logic usually fail the test. A common Order or Customer type looks like reuse, but it ties every consuming team's data model to one definition. Billing needs fields that fulfillment never reads, and each addition forces a coordinated release across teams that would otherwise ship independently.
| Candidate | Changes driven by | Share as a library? |
|---|---|---|
| Logging and trace propagation | Platform or observability policy | Yes |
| Retry policy, HTTP client defaults | Reliability standards | Yes |
| Auth token validation | Security requirements | Yes, or move it to a gateway or sidecar |
| Domain entities | Each team's product needs | No; share the wire contract instead |
| Business rules | A single owning team | No; expose them as a service |
Polyglot organizations face an extra cost. A library written in one language must be reimplemented for every other stack, and those parallel versions drift apart. In that case a contract-first schema or a platform component is often cheaper than maintaining several libraries that are meant to behave the same way.
Designing a loosely coupled shared library
Once you have decided something genuinely belongs in shared code, the design of the library decides whether it stays cheap. A few rules carry most of the weight.
Split by concern, not by team
A single common module couples every consumer to every change in it. A service that needs only retry logic ends up pulling a logging setup and a database helper it never calls. Compare the two layouts:
- One module:
common/containingretry,http,logging,db,auth. Every service upgrades together, for every reason. - Focused packages:
retry,httpdefaults,logfields, each versioned and released independently. A service takes only what it imports.
Keep the dependency footprint near zero
Each transitive dependency is a constraint you impose on every caller. When two libraries require incompatible versions of the same package, you get a diamond conflict, and the service cannot upgrade either one. These conflicts rarely appear in design review. They show up as a release blocker weeks later. Prefer the standard library, and treat every new dependency as an API change.
Don't assume the caller's framework
A retry package should accept a plain config (max attempts, base delay, a retryable-error predicate) and wrap a function. It should not import a web framework or a specific HTTP client. An HTTP-defaults package should return timeouts and settings the caller applies, not construct the client itself.
Version for lag
Use semantic versioning strictly. Announce deprecations with an explicit window, measured in releases or time, and support at least one previous major version. Services should be able to stay behind deliberately. A library that forces lockstep upgrades has recreated the coupling it was meant to avoid.
# retry/__init__.py: standard library only, no framework or HTTP client imports.
import random
import time
from dataclasses import dataclass
from typing import Callable, TypeVar
T = TypeVar("T")
@dataclass(frozen=True)
class RetryConfig:
max_attempts: int = 3
base_delay: float = 0.2 # seconds
max_delay: float = 5.0
# The caller decides what is retryable, so the library never imports
# the caller's client to inspect its exception types.
is_retryable: Callable[[Exception], bool] = lambda exc: True
def retry(fn: Callable[[], T], config: RetryConfig = RetryConfig(),
sleep: Callable[[float], None] = time.sleep) -> T:
"""Call fn, retrying failures with capped exponential backoff and full jitter."""
for attempt in range(1, config.max_attempts + 1):
try:
return fn()
except Exception as exc:
if attempt == config.max_attempts or not config.is_retryable(exc):
raise
delay = min(config.max_delay, config.base_delay * 2 ** (attempt - 1))
sleep(random.uniform(0, delay)) # sleep is injectable for tests
raise AssertionError("unreachable")
# Caller side: the service brings its own client and its own idea of a transient error.
# order = retry(
# lambda: orders_client.get("ord-12345"),
# RetryConfig(max_attempts=5, is_retryable=lambda e: isinstance(e, TimeoutError)),
# )
A retry package that uses only the standard library, takes plain config and a caller-supplied retryable-error check, and wraps any function without importing a framework or HTTP client.
Alternatives to shared code: contract-first generation and platform components
Sometimes the best way to reduce duplicated code is to stop sharing code and share a contract or a running component instead.
Contract-first generation
With contract-first generation, the schema is the shared artifact. Teams agree on an OpenAPI document or Protocol Buffers definitions, and each service generates its own client and server types at build time. No binary is shared between services, so a consumer controls when it regenerates and never inherits another team's transitive dependencies. Shared code becomes a shared interface, which is the coupling you already had.
This approach has three costs:
- Build tooling. Every service needs code generation in its build pipeline.
- Harder debugging. Generated code can be difficult to read and step through.
- Schema versioning discipline. Generation does not catch breaking changes. A renamed field still breaks consumers unless schemas follow strict compatibility rules and those rules are checked in CI.
Platform components
Some cross-cutting behavior doesn't need to live in application code. Gateways, sidecars and service meshes can handle mTLS, retries and rate limiting outside the process. Base container images and service templates cover boilerplate such as logging setup and health endpoints.
In-process library Sidecar / gateway
+-------------------+ +-----------+ +-----------+
| service code | | service |-->| proxy |--> network
| + shared lib |--> net | code | | (mTLS, |
| (retries, mTLS) | +-----------+ | retries) |
+-------------------+ one process +-----------+
upgrade = rebuild every upgrade = roll the proxy;
consuming service adds a hop and a failure point
Moving logic to the platform has costs. Each request takes an extra network hop, so latency goes up. There are more components to deploy, monitor and debug. A misconfigured proxy can now fail every service at once. The platform team owns that runtime around the clock, so it needs the staffing and on-call coverage to support it.
// proto/orders/v1/order.proto
// The schema is the shared artifact. Each service keeps this file
// and generates its own copy of the types at build time, for example:
// protoc -I proto --python_out=gen proto/orders/v1/order.proto
// Generated output in gen/ belongs to the consuming service and is never
// published as a shared library.
syntax = "proto3";
// The version lives in the package name. A breaking redesign goes into a
// new orders.v2 package instead of changing v1 in place.
package orders.v1;
message Order {
// Field numbers identify fields on the wire. Never reuse or renumber them.
string order_id = 1;
string customer_id = 2;
int64 total_cents = 3;
// Field 4 (string coupon) was removed. Reserving its number and name
// prevents a later edit from reusing them with a different meaning,
// which would silently corrupt data for consumers on older code.
reserved 4;
reserved "coupon";
// New fields are additive and get new numbers. Old consumers ignore
// fields they don't recognize, so adding this one is not a breaking change.
string currency_code = 5;
}
service OrderService {
rpc GetOrder(GetOrderRequest) returns (Order);
}
message GetOrderRequest {
string order_id = 1;
}
A versioned Protocol Buffers contract that each service compiles into its own code, using reserved fields and additive changes to stay backward compatible.
A decision checklist and the ownership it requires
Before choosing between a shared library, a generated client, a platform component or a deliberate copy, answer five questions. How many services use the code? How often does it change? Do all consumers need a change at the same moment? Who will own it? What happens when one consumer needs different behavior? That last answer matters most. If the only option is a flag inside the shared code, every special case adds to what all consumers carry.
| Signal | Deliberate copy | Shared library | Platform component |
|---|---|---|---|
| Consumers | Two or three | Many, same language | Many, mixed languages |
| Change rate | Rare | Moderate | Frequent, policy-driven |
| Lockstep needed | No | No | Yes |
| Divergent behavior | Edit the copy | Extension point | Configuration |
Ownership and process
Every shared artifact needs a named owning team. "Everyone owns it" usually means nobody reviews breaking changes. The owners keep a changelog, so a consumer can judge an upgrade without reading diffs. On every release, they run automated tests against a few representative consumers. This catches breakage before it ships, not after each team upgrades on its own schedule.
Dependency update bots should open pull requests in each consuming repository. This makes version drift visible and reviewable, but each team still decides when to merge. If you force lockstep upgrades, you bring back the deployment coupling the library was supposed to avoid.
Deliberate copies need a provenance note: a comment recording the source repository, the path and the date copied. Without it, nobody can find the other copies when a security fix lands in one of them.
Impact on development cost, engineering time, and migrating off a bloated common library
A shared library does not remove the cost of a fix; it changes its shape. With copies, a security patch is written, reviewed and tested once per copy, often by engineers who did not write the original. With a library, the patch is written once, but every consumer still has to bump the dependency, run its pipeline and deploy. You save authoring and review effort, not rollout effort. Version drift makes this worse: a service pinned to an old release keeps the old behavior, bug included, while everyone assumes it was fixed because the code "is shared."
Illustrative estimate (assumed numbers, not a benchmark)
- Copies:
N × (F + D) - Library:
L + N × U
Assume N = 12 services, F = 3 h to fix and test one copy, D = 1 h to deploy, L = 6 h to fix and release the library, U = 1.5 h for each consumer to upgrade and deploy. Copies cost 12 × 4 = 48 h. The library costs 6 + 18 = 24 h. If a breaking API change pushes U above F + D, the library loses.
A wrong abstraction also costs more to undo than a copy, because every caller has to change at once.
Migrating off an overgrown common module
- Inventory every module and the services that import it.
- Inline or copy back domain-specific code to its single real owner, with provenance notes.
- Split stable infrastructure, such as logging and auth clients, into small, separately versioned packages.
- Deprecate the monolith gradually. Publish a removal date and keep a window long enough for the slowest team to migrate.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to reduce duplicated code, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- The dependency graph answers "what does this touch?" in seconds instead of a manual audit.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.