What does it take to scale a monolith horizontally, and when is it the right move?
To scale a monolith horizontally, you run N identical copies of the same application behind a load balancer. Each replica has the same code, the same configuration and the same database connection. The load balancer is free to send any request to any of them. You don't split the codebase, and you don't change how features talk to each other. You change only how many processes serve traffic at once.
Teams usually want three things from this:
- Throughput. More replicas give you more CPU and more worker threads for request handling.
- Availability. A crashed process or a lost node removes one replica, not the whole service.
- Zero-downtime deploys. Rolling updates replace replicas one at a time while the others keep serving.
This fits a team with a working monolith that is limited by CPU or request handling. Typical signs are app servers at high CPU, request queues that grow under load, or a single process that can't keep up with concurrent connections.
It does not fit when the database is the constraint. If most of your request time goes to queries, or to lock contention on a single hot table, extra replicas only add connections and concurrent writers to the component that is already struggling. Latency can get worse. Before adding replicas, measure where request time actually goes: application code, database, or external calls.
The core idea behind the rest of this checklist is that a single process can quietly hold state that N processes cannot share. Every piece of process-local state must be handled in one of three ways:
- Externalized to a shared store.
- Partitioned so each replica owns a distinct slice.
- Knowingly accepted as lossy.
Any state you don't handle in one of these ways will become a bug.
+-----------+
| Clients |
+-----+-----+
|
v
+-------------------+
| Load balancer |
+---------+---------+
|
+----------------------+----------------------+
v v v
+------------------+ +------------------+ +------------------+
| Replica 1 | | Replica 2 | | Replica N |
| same code+config | | same code+config | | same code+config |
| local: lossy | | local: lossy | | local: lossy |
| owns slice A | | owns slice B | | owns slice N |
+--------+---------+ +--------+---------+ +--------+---------+
| | |
+------+---------------+-----------------+----+
| |
v v
+---------------------------+ +---------------------------+
| Shared state store | | Database |
| sessions, locks, job queue| | shared by all replicas |
+---------------------------+ +---------------------------+
Each piece of process-local state: externalize, partition, or accept loss
N identical monolith replicas sit behind one load balancer, and each replica's process-local state is either moved to a shared store, partitioned so one replica owns each slice, or kept locally and accepted as lossy, while every replica still uses the same database.
How do you inventory the state hiding in your monolith?
Hidden state rarely announces itself, so search for it deliberately. Read the code for module-level or static mutable variables, singletons registered at startup, anything that writes outside a database connection, and anything started by a timer at boot. Check for each of these:
- In-process session maps: login state held in a dictionary keyed by session ID.
- In-memory caches: computed results or database rows held in a process-local map.
- Singletons holding counters or rate limits: per-user quotas, request counters, circuit-breaker state.
- Local file uploads and temp files: user uploads, generated exports, multi-step processing scratch files.
- WebSocket connection registries: maps of user ID to open socket, used for pushing messages.
- In-process job queues: work handed to a background thread or channel.
- Cron-style schedulers: timers that fire nightly jobs from inside the app.
| State type | Symptom on many replicas | Usual decision |
|---|---|---|
| Session map | Random logouts as requests hit other replicas | Externalize |
| Business-data cache | Stale or inconsistent reads between replicas | Externalize or accept loss with short TTL |
| Counters / rate limits | Effective limit multiplies by replica count | Externalize |
| Uploaded files | Intermittent 404s on download | Externalize |
| WebSocket registry | Messages silently fail to reach some users | Partition or externalize routing |
| In-process queue | Jobs lost on restart or deploy | Externalize |
| Scheduler | Each job runs once per replica | Partition via a lock or leader |
Some state is legitimately local. Memoized results of expensive pure computations, compiled templates, and warm per-node caches of immutable reference data are correct on every replica because they cannot disagree with each other. Classify these as "accept loss" and move on. The goal is classification, not elimination. Externalizing state that never needed sharing adds network hops and new failure modes, and it spends engineering time that the riskier items on this list need more.
Stateless sessions, a shared session store, or sticky sessions: which should you use?
Sessions are usually the first piece of state that breaks when you scale a monolith horizontally, and there are three ways to handle them.
Stateless sessions put the session in a signed token or cookie, so any replica can verify it without a lookup. Keep claims minimal, such as a user ID, an expiry, and a key ID. The token travels on every request, and cookies have size limits. Sign with a key identified by that key ID. Replicas verify against the current key plus recently retired ones, so rotation never logs everyone out. Revocation is the weak point, because a valid signature stays valid until it expires. You can use short expiries with a refresh step, or a denylist. A denylist is shared state again.
A shared session store keeps sessions server-side, so revocation is a simple delete. Every request adds a network hop, though, and the store becomes a dependency that needs its own availability and failover plan.
Sticky sessions have the load balancer pin a client to one replica, usually through a cookie or a hash of the client IP. This is a stopgap. Sessions are lost when a replica dies, during deploys, and on scale-in. Long-lived clients also skew load. Worst of all, affinity hides state bugs until the day you remove it.
| Option | Dev cost | Ops cost | Revocation | Failure mode |
|---|---|---|---|---|
| Stateless tokens | Medium | Low (key management) | Delayed or needs denylist | Leaked key forges sessions |
| Shared store | Low | Medium–high | Immediate | Store outage blocks logins |
| Sticky sessions | Very low | Low | Local only | Lost sessions, uneven load |
import base64, binascii, hashlib, hmac, json, time
# kid -> key. New tokens use CURRENT_KID. Retired keys remain here until
# every token they signed has expired. Placeholders: load from a secret store.
SIGNING_KEYS = {
"2026-11": b"placeholder-current-key-not-a-real-secret",
"2026-10": b"placeholder-retired-key-not-a-real-secret",
}
CURRENT_KID = "2026-11"
def _b64e(b: bytes) -> str:
return base64.urlsafe_b64encode(b).rstrip(b"=").decode()
def _b64d(s: str) -> bytes:
return base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))
def issue(user_id: str, ttl: int = 900) -> str:
# Minimal claims and a short expiry, so revocation waits at most ttl seconds
payload = _b64e(json.dumps({"sub": user_id, "exp": int(time.time()) + ttl}).encode())
signed = f"{CURRENT_KID}.{payload}".encode()
sig = hmac.new(SIGNING_KEYS[CURRENT_KID], signed, hashlib.sha256).digest()
return f"{CURRENT_KID}.{payload}.{_b64e(sig)}"
def verify(token: str) -> dict | None:
try:
kid, payload, sig = token.split(".")
key = SIGNING_KEYS[kid] # unknown or fully retired kid -> reject
expected = hmac.new(key, f"{kid}.{payload}".encode(), hashlib.sha256).digest()
# Constant-time compare. The kid is covered by the signature.
if not hmac.compare_digest(expected, _b64d(sig)):
return None
claims = json.loads(_b64d(payload))
except (ValueError, KeyError, binascii.Error):
return None
if not isinstance(claims, dict) or claims.get("exp", 0) < time.time():
return None # expired: client must use the refresh flow
return claims
A stdlib-only Python example that signs session tokens with the current key and verifies them against the current and retired keys by key ID, so rotating keys does not log users out.
How should caches and shared state behave across replicas?
Sort every in-process cache into one of two groups before adding replicas.
- Performance-only caches: rendered fragments, computed reports, lookups of slow-changing reference data. If replica A serves a value a few seconds older than replica B, nothing breaks. These can stay local, with a short TTL that limits how long a stale value can survive.
- Caches that act as a source of truth: feature flags, permission sets, rate-limit counters, prices, anything a write path reads back. If replicas disagree here, users get different answers depending on which replica they hit. Move these to a shared cache, or broadcast an invalidation event on every write so that each replica evicts its copy. A TTL alone does not make this data correct. It only puts a bound on how long the data stays wrong.
More replicas also make stampedes more likely. After a deploy, every replica starts cold. When a popular key expires, every replica misses at the same moment and sends the same query to the database. Three mitigations help:
- Request coalescing: one caller per key fetches the value, and concurrent callers wait for that result.
- Jittered TTLs: add random spread to expiry times so keys written together do not expire together.
- Stale-while-revalidate: keep serving the old value while a single background refresh runs.
A shared cache comes with a trade-off. It gives you consistency, but it also becomes a dependency for every replica. If it goes down, all replicas fall through to the database at once, which is the same stampede at a larger scale. Set timeouts on cache calls, and decide in advance whether a cache failure should degrade the response or fail the request.
// Per-replica cache for PERFORMANCE-ONLY data (reports, reference lookups).
// Do not use this for flags, permissions or prices. Those need a shared cache or invalidation.
type Entry<T> = { value: T; freshUntil: number; staleUntil: number };
const cache = new Map<string, Entry<unknown>>();
const inFlight = new Map<string, Promise<unknown>>();
// Spread expiry by +/-10% so keys written together do not expire together.
const jitter = (ms: number) => ms * (0.9 + Math.random() * 0.2);
// Request coalescing: concurrent callers for one key share a single load() call.
function refresh<T>(key: string, load: () => Promise<T>, ttlMs: number, staleMs: number): Promise<T> {
const pending = inFlight.get(key);
if (pending) return pending as Promise<T>;
const p = load()
.then((value) => {
const freshUntil = Date.now() + jitter(ttlMs);
cache.set(key, { value, freshUntil, staleUntil: freshUntil + staleMs });
return value;
})
.finally(() => inFlight.delete(key));
inFlight.set(key, p);
return p;
}
export async function getCached<T>(key: string, load: () => Promise<T>, ttlMs = 30_000, staleMs = 60_000): Promise<T> {
const entry = cache.get(key) as Entry<T> | undefined;
const now = Date.now();
if (entry && now < entry.freshUntil) return entry.value;
if (entry && now < entry.staleUntil) {
// Stale-while-revalidate: return the old value and refresh once in the background.
// If the refresh fails, keep serving the stale value until staleUntil.
refresh(key, load, ttlMs, staleMs).catch(() => {});
return entry.value;
}
// Cold start or fully expired: every caller waits on the same single fetch.
return refresh(key, load, ttlMs, staleMs);
}
A dependency-free in-process cache wrapper that combines request coalescing, jittered TTLs and stale-while-revalidate, so each replica sends at most one database query per key when that key expires.
How do you stop scheduled jobs from running on every replica?
An in-process scheduler, such as a cron library or a timer started at boot, has no awareness of other copies of the application. Every replica loads the same schedule and fires on time, so six replicas send the nightly invoice email six times. There are three common fixes.
A dedicated scheduler deployment disables the scheduler in web replicas through configuration and runs one separate instance that owns all schedules. Leader election keeps the scheduler in every replica, but only the elected leader acts. A database lease stores a locked_until column on each job row. A replica claims the job with a conditional UPDATE that sets locked_until to the database's now() plus an interval, but only where the existing value has already expired. Only the replica whose update affects a row runs that job.
All three depend on time, and time is unreliable. A garbage-collection pause longer than the lease lets another replica take the job while the original holder resumes believing it still owns it. Clock skew breaks expiry when replicas calculate it from their own clocks, so always use the database clock. During a network partition, the holder cannot renew its lease but keeps working. Fencing tokens narrow this gap, but only where the downstream system checks them. Make handlers idempotent: give each run a unique key and write results as upserts.
| Approach | Complexity | Failure behavior |
|---|---|---|
| Dedicated scheduler | Low | Missed runs until restart; deploys can briefly overlap |
| Leader election | High | Split brain during partitions or pauses |
| Database lease | Medium | Duplicate runs after pauses longer than the lease |
-- PostgreSQL. One row per scheduled job; the lease is just a timestamp.
CREATE TABLE scheduled_jobs (
name text PRIMARY KEY,
locked_until timestamptz NOT NULL DEFAULT '-infinity', -- never NULL, so expiry checks stay simple
locked_by text,
fence bigint NOT NULL DEFAULT 0 -- increases on every claim
);
-- Claim: succeeds only if the previous lease has expired.
-- Expiry is calculated from the database clock (now()), not the replica's clock.
UPDATE scheduled_jobs
SET locked_until = now() + interval '5 minutes',
locked_by = 'web-3',
fence = fence + 1
WHERE name = 'nightly_invoice_email'
AND locked_until < now()
RETURNING fence; -- zero rows: another replica holds the lease, so skip this run
-- Renew while working. This succeeds only if no other replica has claimed the job since.
UPDATE scheduled_jobs
SET locked_until = now() + interval '5 minutes'
WHERE name = 'nightly_invoice_email'
AND locked_by = 'web-3'
AND fence = 42; -- the token returned by the claim; zero rows means stop
-- The lease can still fail after a long pause, so the work itself must be idempotent.
CREATE TABLE job_runs (
run_key text PRIMARY KEY, -- one key per logical run, e.g. job name plus date
fence bigint NOT NULL,
finished_at timestamptz NOT NULL DEFAULT now()
);
INSERT INTO job_runs (run_key, fence)
VALUES ('nightly_invoice_email:2026-11-07', 42)
ON CONFLICT (run_key) DO NOTHING; -- a duplicate run records nothing new
A PostgreSQL lease claim that uses the database clock for expiry and a fencing token for renewal, plus an idempotent run record keyed per logical run so a duplicate execution has no extra effect.
Moving concurrency, files, and real-time connections out of the process
Concurrency: let the database arbitrate
An in-process mutex only coordinates threads inside one replica. With three replicas, three requests can each take "the" lock and update the same row. Move that coordination into the database, which every replica already shares:
- Row locks (
SELECT ... FOR UPDATEinside a transaction) for short critical sections where waiting is acceptable. - Optimistic concurrency with a version column, for cases where conflicts are rare. Read the row and its
version, then write withUPDATE orders SET status = 'paid', version = version + 1 WHERE id = 42 AND version = 7. If zero rows are affected, another replica got there first, and the caller re-reads or reports a conflict. - Unique constraints for "only one of these may exist" rules. Check-then-insert races across replicas. A constraint cannot race.
- Idempotency keys on write endpoints. Store the key with the result so that when a client or load balancer retries a request and it lands on a different replica, that replica returns the stored response and does not repeat the side effect.
Local disk
Any replica can be replaced, and each one sees a different filesystem. User uploads, generated reports, and SQLite-style local databases must move to shared object or block storage. If you keep anything on local disk, treat it as disposable. Also audit log paths and temp directories for host assumptions, such as a file written by one request and read by the next.
Websockets and SSE
A connection is pinned to one replica, but the event that should reach it can originate on any replica. Publish events through a pub/sub layer. Each replica subscribes and forwards messages to its own connected clients. Deploys will drop connections, so clients need to reconnect with backoff and resume from a last-seen event ID.
How do you prove a monolith is ready for many replicas before production?
Operational readiness checklist
- Separate liveness from readiness. Liveness asks "is this process wedged?" and should not check the database. If it does, one database blip restarts every replica at once. Readiness asks "should this replica receive traffic?" and can check its dependencies.
- Drain on shutdown. On SIGTERM, fail readiness first, stop accepting new requests, and let in-flight requests finish within a timeout. Then release distributed locks and close pools. A replica that dies while holding a lock blocks the others until the lock's TTL expires.
- Read configuration from the environment. Every replica must get identical settings from environment variables or a config service. A file edited by hand on one host makes that replica behave differently from the rest.
- Size connection pools against the database limit. Illustrative estimate, assumed inputs: the database allows 500 connections, you reserve 50 for admin and migrations, and each replica's pool holds 20. That gives 450 / 20 = 22 replicas. A rolling deploy with 25% surge briefly runs about 28 replicas, which needs 560 connections. To stay under the limit, cap the steady-state count around 18 or shrink the pools. Another option is a connection pooler in front of the database, which multiplexes many client connections onto fewer server connections.
Verification plan
Run at least two replicas in staging with session affinity off and random routing. Any state that still lives in process memory will show up as intermittent, flapping failures. Kill replicas mid-request and confirm that clients retry and locks are released. Run a rolling deploy under sustained load while watching error rates and database connection counts.
This work adds dependencies and network hops before you see any benefit. Finding hidden state in staging still costs far less than finding it during a production incident.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to scale monolith horizontally, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.