How to Scale a Monolith Horizontally: State, Locks and Cron Jobs

What changes when a monolith runs as multiple instances

To scale a monolith horizontally means running several identical copies of the same application behind a load balancer, each able to serve any request. Teams choose this over splitting into services because it keeps one codebase, one deploy pipeline and one data model. More capacity comes from adding instances, not from redrawing service boundaries.

The catch is that the second instance turns your app into a distributed system. Code that was correct on one process starts to fail in quiet ways. A session stored in memory disappears when the next request lands elsewhere. A file written to local disk is missing on the other node. A mutex guards only its own process. A cron job scheduled inside the app now fires once per instance. None of these break in development, because development runs one copy.

Who should do this

It fits teams whose single server is hitting real limits. CPU may be saturated by request handling, or one box may be an unacceptable availability risk where a reboot or bad deploy takes everything down. If the application tier is the constraint and the code is mostly request/response, multiple instances are usually the cheapest next step.

When to avoid it

Skip it if the database is the bottleneck. More app instances add connections and load to the thing that is already struggling. Also reconsider if the app holds a lot of in-process state and traffic is modest. A larger machine buys time without the migration work, and vertical scaling is often the honest answer.

Copy                         +------------------+
                         |     Clients      |
                         +--------+---------+
                                  |
                                  v
                         +------------------+
                         |  Load balancer   |
                         +---+----------+---+
                             |          |
              +--------------+          +-------+
              | req 1                           | req 2
              v                                 v
   +----------------------+          +----------------------+
   | Instance A           |          | Instance B           |
   | same code + config   |          | same code + config   |
   |                      |          |                      |
   | session S1 in memory |          | session S1: missing  |
   | upload.pdf on disk   |          | upload.pdf: missing  |
   | mutex guards A only  |          | mutex guards B only  |
   | cron job fires here  |          | cron job fires again |
   +----------+-----------+          +-----------+----------+
              |                                  |
              +----------------+-----------------+
                               | connections = instances x pool size
                               v
                    +----------------------+
                    | Shared database      |
                    | often the bottleneck |
                    +----------------------+

Two identical instances behind a load balancer: in-memory sessions, local files, mutexes and in-app cron jobs stop being shared, and every added instance opens more connections to the one database.

Audit checklist: finding hidden single-instance assumptions

Before changing any code, build an inventory. Single-instance assumptions rarely appear in design docs. They sit in a module-level dictionary someone added three years ago, or a timer started at boot. Search the codebase and the build artifacts, not just the architecture diagram.

Work through these searches and log every hit in a spreadsheet with file, line and owner:

• Module-level mutable state: globals, class-level fields, singletons holding maps or counters. Each instance gets its own copy, so values diverge silently.
• In-memory caches: memoization decorators, hand-rolled LRU maps, framework cache defaults. Invalidation on one instance never reaches the others.
• Local file writes: uploads, generated reports, temp files read back on a later request. The follow-up request may land on a different instance.
• In-process mutexes: language-level locks, synchronized blocks, semaphores guarding "only one at a time" logic. They serialize threads within one process and nothing beyond it.
• Timers and background threads: anything spawned at startup that loops or sleeps.
• Framework scheduler annotations: for example, Spring's @Scheduled or similar decorators. Every instance runs them.
• Cron entries in the image: check the Dockerfile, entrypoint scripts and any crontab copied into the container.

Classify each finding by what goes wrong and which fix applies:

Fix the high-risk rows before you add the second instance.

Copy#!/usr/bin/env bash
# First-pass audit for single-instance assumptions. Expect false positives:
# every hit is triaged by hand into the inventory spreadsheet.
set -euo pipefail
SRC="${1:-.}"
OUT="${2:-single-instance-findings.tsv}"
printf 'category\tfile:line:match\n' > "$OUT"

# scan <category> <extended-regex>: tag each hit with its category
scan() {
  grep -rInE --exclude-dir=.git --exclude-dir=node_modules \
    --exclude-dir=vendor --exclude-dir=build --exclude-dir=target \
    --include='*.py' --include='*.java' --include='*.kt' \
    --include='*.js' --include='*.ts' --include='*.go' --include='*.rb' \
    -e "$2" "$SRC" | sed "s/^/$1\t/" >> "$OUT" || true
}

# Module-level mutable state: top-level dict/list/set literals, static collections
scan module-state '^[A-Za-z_][A-Za-z0-9_]* *= *(\{|\[|dict\(|set\()|static +(final +)?(Map|HashMap|List|Set|AtomicInteger|AtomicLong)'
# In-memory caches: memoization and hand-rolled LRU maps
scan cache 'lru_cache|@cache\b|@Cacheable|LinkedHashMap|new +LRU'
# Local file writes (open() also matches reads; triage manually)
scan file-write 'FileOutputStream|FileWriter|Files\.write|fs\.writeFile|tempfile\.|open\('
# In-process locks: only serialize threads inside one process
scan in-process-lock 'threading\.(Lock|RLock|Semaphore)|synchronized|ReentrantLock|sync\.Mutex|Mutex\.new'
# Timers and background threads started at boot
scan timer-thread 'setInterval|threading\.(Thread|Timer)|ScheduledExecutorService|new +Timer\(|time\.NewTicker'
# Framework schedulers: run on every instance
scan scheduler '@Scheduled|@EnableScheduling|BackgroundScheduler|add_job\(|cron\.schedule'

# Cron baked into the image: Dockerfiles, entrypoints, copied crontabs
grep -rInE --include='Dockerfile*' --include='*.sh' -e 'cron|crontab' "$SRC" \
  | sed 's/^/image-cron\t/' >> "$OUT" || true
find "$SRC" -type f \( -name 'crontab*' -o -path '*/cron.d/*' \) \
  | sed 's/^/image-cron\t/' >> "$OUT"

echo "$(($(wc -l < "$OUT") - 1)) findings written to $OUT"

A grep-based bash script that sweeps source files, Dockerfiles and entrypoint scripts for each category of single-instance assumption and writes tagged hits to a TSV for triage and risk classification.

Sessions, local disk and caches: making requests stateless

Sticky sessions (load-balancer affinity that pins a client to one instance) let a stateful monolith run behind several instances without code changes. They are a stopgap, though. Long-lived or heavy clients pile onto one instance, so load spreads unevenly. Any state held in memory is lost when that instance is redeployed or crashes. Most importantly, the stateful code stays in place, and every later fix still has to deal with it.

Sessions

Move sessions to a shared store keyed by session ID, or to signed client-side tokens. A shared store allows server-side revocation but adds a lookup to every request. Signed tokens need no lookup, but revoking one before it expires is hard, and anything inside the token is visible to the client unless you also encrypt it.

Files

Write uploads and long-lived generated files, such as reports and exports, to shared object storage. The next request may land on a different instance, and that instance cannot read another instance's local disk. Temp files are fine on local disk if they are created and deleted within a single request.

Caches

Choose a cache strategy based on how much staleness the data can tolerate:

Any shared session store or cache is a new dependency. Its latency adds to each request. If it goes down, decide in advance whether the app fails open or fails closed.

Replacing in-process locks with database guarantees and leases

A synchronized block or in-process mutex only coordinates threads inside one process. Once a second instance exists, the shared database is usually the cheapest coordinator. Use the weakest mechanism that gives the guarantee you need.

Start with constraints. If two instances insert the same payment, a unique index decides the winner atomically, and no lock can leak. Row locks suit read-modify-write on existing rows. They release on commit or rollback.

Session-scoped advisory locks belong to the database connection, not to your request. With a connection pool, the connection can return to the pool still holding the lock. The lock can then be released when the pool resets or closes that connection, or a different request can inherit it. Prefer transaction-scoped variants.

A lease records an owner, an expiry, and a monotonically increasing token. The holder renews before the TTL ends. A garbage-collection pause or network stall can outlast the TTL, so the old holder may wake up and write anyway. Fencing prevents this. Each write carries the token, and the store rejects any token lower than the latest one, often through a version check in the WHERE clause. Compute expiry from the store's clock, not each instance's clock, because clocks on different hosts drift. For this reason, a cache key with a TTL is fine for reducing duplicate work, but it is not enough when correctness depends on the lock.

Copy-- PostgreSQL. All expiry math uses the database clock (now()), never the app host's clock.
CREATE TABLE leases (
  name       text PRIMARY KEY,
  owner      text        NOT NULL,
  token      bigint      NOT NULL,          -- fencing token, only ever increases
  expires_at timestamptz NOT NULL
);

-- Acquire: succeeds if no lease exists or the current one has expired.
-- A takeover bumps the token. No row returned means another holder is live.
INSERT INTO leases (name, owner, token, expires_at)
VALUES ('nightly-billing', 'worker-a', 1, now() + interval '30 seconds')
ON CONFLICT (name) DO UPDATE
  SET owner      = EXCLUDED.owner,
      token      = leases.token + 1,
      expires_at = EXCLUDED.expires_at
  WHERE leases.expires_at < now()
RETURNING token;                              -- suppose this returns 42

-- Renew well before the TTL ends. Zero rows updated means the lease was lost: stop work.
UPDATE leases
   SET expires_at = now() + interval '30 seconds'
 WHERE name = 'nightly-billing' AND owner = 'worker-a' AND token = 42
   AND expires_at > now();

-- Fenced write: the protected row remembers the highest token it has accepted.
-- A paused former holder still carrying token 41 updates zero rows.
UPDATE billing_runs
   SET status = 'done', fence_token = 42
 WHERE run_date = DATE '2026-10-22'
   AND fence_token <= 42;

-- For short critical sections, prefer a transaction-scoped advisory lock:
-- it is released at COMMIT/ROLLBACK, so a pooled connection cannot carry it to another request.
BEGIN;
SELECT pg_try_advisory_xact_lock(hashtext('nightly-billing'));  -- false: someone else holds it
-- ... do the work ...
COMMIT;

A PostgreSQL lease that takes expiry from the database clock, increments a fencing token on takeover so a stale holder's writes are rejected by a version check, and a transaction-scoped advisory lock that cannot leak through a connection pool.

Stopping duplicate cron jobs: options ranked by simplicity

When every instance runs the same scheduler, a nightly job runs once per replica. There are three common ways to get one run per cluster. They are listed here from simplest to most robust.

1. Dedicated scheduler role

Run the same build with a flag or environment variable that enables the scheduler on exactly one instance. It takes the least code and is easy to reason about. The cost is that this instance is a single point of failure. If it dies, nothing runs until someone notices. Its configuration also tends to drift from the web fleet, because it gets deployed and scaled differently.

2. Database-backed leader election or lease

Each instance tries to acquire a lease row with an expiry. The holder runs jobs and renews the lease, and the others wait. Failover is automatic, but it is not instant, because a successor must wait out the TTL. A short TTL speeds failover but risks flapping during GC pauses or slow renewals. A long TTL leaves a gap after a crash. Renewal must also check that the instance still holds the lease before doing work.

3. External scheduler plus queue

A scheduler outside the application only enqueues a message, and any instance consumes it. Scheduling and execution become separate concerns, so a crashed worker means a retry, not a missed run. Delivery is usually at-least-once, so jobs must tolerate duplicates.

Copy-- PostgreSQL: one lease row per scheduler, shared by all instances.
CREATE TABLE IF NOT EXISTS scheduler_lease (
  name       text PRIMARY KEY,
  holder     text        NOT NULL,
  expires_at timestamptz NOT NULL
);

-- Each instance runs this every few seconds (well inside the TTL).
-- It takes the lease if it is free or expired, or renews it if this
-- instance already holds it. It does nothing if a live peer holds it.
-- Server time (now()) is used so instance clock skew does not matter.
INSERT INTO scheduler_lease (name, holder, expires_at)
VALUES ('nightly-jobs', 'web-7f3a', now() + interval '30 seconds')
ON CONFLICT (name) DO UPDATE
  SET holder     = EXCLUDED.holder,
      expires_at = EXCLUDED.expires_at
  WHERE scheduler_lease.expires_at < now()
     OR scheduler_lease.holder = EXCLUDED.holder
RETURNING holder;
-- One row returned: this instance is leader until expires_at.
-- Zero rows returned: another instance holds a live lease, so stay idle.

-- Right before starting a job, confirm the lease is still ours and not
-- about to lapse. A long GC pause can expire it after the last renewal.
SELECT 1
FROM scheduler_lease
WHERE name = 'nightly-jobs'
  AND holder = 'web-7f3a'
  AND expires_at > now() + interval '5 seconds';

-- On graceful shutdown, release the lease so a peer can take over
-- without waiting out the full TTL.
DELETE FROM scheduler_lease
WHERE name = 'nightly-jobs' AND holder = 'web-7f3a';

A PostgreSQL lease row that instances atomically acquire or renew with INSERT ... ON CONFLICT. The holder re-checks the lease before running a job and releases it on shutdown to shorten failover.

Designing jobs and real-time features for at-least-once execution

Every row in that table ends the same way: under some failure, a job runs twice. Lease expiry during a long GC pause, a worker crashing after doing the work but before acknowledging it, a network timeout that makes the scheduler retry. Queues and schedulers that advertise exactly-once delivery usually mean exactly-once within their own boundary. The moment your job writes to a database, calls a payment API or sends an email, there is a window where the side effect happened but the acknowledgement did not. Design for at-least-once and make the second run harmless.

A job-run table as the gate

Give each scheduled execution an identity: job name plus the scheduled time it is meant to cover (not the time it actually started). Insert a row into a job_runs table with a unique constraint on (job_name, scheduled_for) before doing the work. The instance whose insert succeeds proceeds; any other instance gets a constraint violation and exits. Record status and completion time on the same row so a crashed run is visible and can be retried deliberately.

Idempotency keys for side effects

The gate stops duplicate starts, not duplicate effects inside a retried run. Derive a deterministic key per effect, such as invoice-reminder:{customer_id}:{scheduled_for}, and pass it to downstream APIs that accept one, or store it under a unique constraint alongside the write.

WebSockets and server-sent events

A WebSocket or SSE connection lives on one instance. When an event is produced on instance A, clients connected to instance B never see it unless A publishes to a shared pub/sub channel that every instance subscribes to and fans out locally. Treat delivery as best-effort. Clients should reconnect with a last-seen event ID and refetch anything they missed.

Verification, cost and engineering time

Before and after launch

• Run two instances in staging, never one. Many single-instance assumptions only fail when a second process exists, so put both behind the same load balancer you use in production.
• Kill an instance mid-request and mid-job with a hard kill, not a polite stop. The request should fail cleanly or be retried safely. The job's lease should expire, and another instance should complete the work without duplicating its side effects.
• Tag every log line and metric with an instance ID. A duplicate cron run then shows up as one job with two IDs, which is easier to spot than doubled counts.
• Check that health checks and graceful shutdown work together. An instance should leave rotation first, then finish in-flight requests and either complete or release leased jobs within the orchestrator's termination grace period.

Where the cost goes

Time saved by auditing first

Illustrative estimate (assumed numbers, not a benchmark). Assume the audit costs A = 2 days. It catches S = 6 stopgaps, such as sticky sessions or a "cron runs here" flag. Fixing each one properly up front takes D = 1 day. Leaving it in place costs R = 3 days to rewrite later, plus I = 1 day of incident handling.

Net savings = S × (R + I − D) − A = 6 × (3 + 1 − 1) − 2 = 16 engineer-days.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to scale monolith horizontally, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.