Why do scheduled jobs run twice when a monolith scales to multiple replicas?
An in-process scheduler has no idea it has siblings. When your monolith starts, the scheduler library reads its job definitions, computes the next fire time from the local clock, and runs each job in the same process as your web handlers. Run one instance and that works. Scale to three replicas behind a load balancer and you have three independent schedulers, each sure it is the only one. The nightly invoice job runs three times, and customers get three emails.
A distributed cron lock fixes this. Before a job runs, each replica tries to acquire a shared lock for that job, usually in a database or coordination store that every replica can reach. One replica wins and does the work. The others skip that run.
Replica count is the obvious cause. These are less obvious:
- Rolling deploys. During a rollout, old and new pods run together for seconds or minutes. A job that fires inside that window runs in both versions, even if you normally run a single replica.
- Clock skew. Hosts drift apart. If one replica's clock is a few seconds ahead, it can finish a job before another replica's scheduler fires for the same window. A lock that has already been released will not block the second run.
- Crash retries. A process dies mid-job, restarts, and the scheduler's misfire handling runs the job again. To your logs this looks like a fresh run, not a retry.
This guide is for teams running a monolith with an in-app scheduler on more than one instance.
Who should use a distributed cron lock?
A distributed cron lock fits when all of the following are true:
- Scheduling lives inside the application. Jobs are defined in code and fired by a scheduler library running in the same process as the web or worker code, rather than by a separate platform scheduler.
- More than one instance can be alive at once. This covers permanent horizontal scaling, and it also covers a single-replica service that uses rolling deploys, because old and new instances overlap during a rollout.
- A duplicate run causes harm or waste. Typical cases are sending emails or notifications, charging or invoicing, calling rate-limited or paid third-party APIs, and generating reports or exports that downstream systems consume.
- Every replica can reach a shared, strongly consistent store. In practice this is usually the primary Postgres database the monolith already uses.
The typical user is a small to mid-sized team that has outgrown "run it on one box" but does not want to operate a separate job platform yet. Adding a lock keeps the scheduler where it is, keeps all replicas identical, and adds no new infrastructure.
How does a distributed cron lock work?
The mechanism has the same shape whatever store backs it:
- Every replica fires. Each replica's scheduler fires the job at its configured time, independently. Nothing tries to stop the schedulers from firing. Coordination happens at the next step.
- Each replica derives the same key. The key is computed deterministically from the job's identity, such as a hash of the job name. Optionally the key also includes the schedule slot. Every replica and every release must compute the identical value.
- Each replica attempts an atomic acquire. The shared store guarantees that only one caller can succeed for a given key at a time. With Postgres this is
pg_try_advisory_lockor anINSERTguarded by a unique constraint. With a lease it is a conditional write that succeeds only if no unexpired owner exists. - The winner runs and the losers skip. The acquire call is non-blocking, so losers return immediately and do nothing for that tick. They should log the skip.
- The lock is released. The release happens in one of four ways: an explicit unlock, the end of a transaction, the closing of the holding connection, or the expiry of a lease TTL. Which mechanism applies determines what happens when the holder crashes or stalls.
A lock by itself only prevents concurrent runs. To prevent a second run within the same schedule window, which can happen when clock skew separates the replicas' fire times, you also record the slot itself. The sections below cover each piece: advisory locks for mutual exclusion, a job_runs table for once-per-window, leases with fencing tokens for long or stall-prone jobs, and idempotency so that any remaining duplicate is harmless.
What are the options for running cron jobs exactly once across replicas?
No option here gives true exactly-once execution. A process can die after doing the work but before recording it, or a network partition can leave two nodes each believing they own the job. Every design is either at-most-once (a run may be skipped) or at-least-once (a run may repeat). Choose which failure you can tolerate, then pick the mechanism.
| Option | Assumes | Failure mode | Operational cost |
|---|---|---|---|
| Single designated cron instance | One replica configured differently (env flag, separate deployment) | That instance dies and jobs stop silently: at-most-once | Low to build, but deploys are no longer uniform |
| Lock per job run | A shared database every replica can reach | Holder crashes mid-run, or the lock is released before a slow replica fires | Low: no new infrastructure |
| Leader election | A coordination service or lease table with heartbeats | Split-brain during partitions or GC pauses lets two leaders overlap | Medium: lease tuning and fencing |
| External scheduler or orchestrator job | A platform that launches one job process per schedule | Retries and misfire policies can double-run: at-least-once | Medium to high: a second runtime to deploy and monitor |
The default should be a lock held in the database you already run. Your replicas already depend on it, so the lock adds no new component. Its failure modes are also the same as your application's failure modes. Per-run locking also keeps every replica identical, which removes the designated-instance problem.
The heavier options earn their cost in specific situations. Use leader election when many jobs need one long-lived owner with in-memory state. Use an external scheduler when jobs run long enough to block deploys, or when they need their own resource limits and retry history.
How do Postgres advisory locks work?
Postgres advisory locks are application-defined locks the database tracks but never takes on its own. You choose a key, either one 64-bit integer or two 32-bit integers, and Postgres enforces mutual exclusion on that key. It does not care which table or row the key refers to. All applications connected to the same database share one key namespace. If two unrelated services both pick key 42, they block each other, so derive keys deliberately, for example by hashing the job name.
There are two scopes, and each has a blocking and a non-blocking form:
| Function | Scope | Waits if held? | Released when |
|---|---|---|---|
pg_advisory_lock |
Session | Yes | pg_advisory_unlock is called or the session ends |
pg_try_advisory_lock |
Session | No, returns false |
pg_advisory_unlock is called or the session ends |
pg_advisory_xact_lock |
Transaction | Yes | Commit or rollback |
pg_try_advisory_xact_lock |
Transaction | No, returns false |
Commit or rollback |
For cron, the try variants are usually the right choice. A replica that loses the race should skip the run, not queue up behind the winner and execute the same job a second time once it finishes.
Automatic release is the main appeal. If the process holding the lock crashes or is killed, its connection closes and Postgres drops the lock. You don't need a stale-lock reaper or a TTL to tune.
That same property is the trade-off. The lock reflects the health of the connection, not the health of the job. Advisory locks are not replicated, so after a failover the promoted primary holds none of them. A network blip that drops the connection has the same effect. In both cases another replica can acquire the lock while the original worker is still running.
-- Session A (replica 1): derive a key from the job name and try to take it.
-- hashtext() returns int4, so use the (int, int) form with a fixed namespace
-- value (1001 here) to keep cron keys apart from other advisory-lock users.
SELECT pg_try_advisory_lock(1001, hashtext('nightly_invoice_export'));
-- pg_try_advisory_lock
-- t <- replica 1 wins and runs the job
-- Session B (replica 2): same key, a few milliseconds later.
SELECT pg_try_advisory_lock(1001, hashtext('nightly_invoice_export'));
-- f <- returns immediately; replica 2 skips this run
-- Session A: release explicitly when the job finishes.
SELECT pg_advisory_unlock(1001, hashtext('nightly_invoice_export'));
-- If session A's process had crashed instead, closing its connection would
-- have released the lock with no reaper or TTL. Check who holds what with:
SELECT pid, classid, objid, granted
FROM pg_locks
WHERE locktype = 'advisory';
-- Transaction-scoped variant: the lock lives only until COMMIT or ROLLBACK,
-- so there is no unlock call to forget.
BEGIN;
SELECT pg_try_advisory_xact_lock(1001, hashtext('nightly_invoice_export'));
-- ... run the job's work inside this transaction if the result was true ...
COMMIT; -- lock released here
Two replicas race for the same job key with the non-blocking try functions. The loser gets false and skips the run, and pg_locks shows which session holds the advisory lock in both the session and transaction scopes.
How to implement a distributed cron lock with Postgres advisory locks
Each job needs a stable lock key, derived the same way on every replica and in every release. Hash the job name (for example billing.invoice_rollup) in application code to a 64-bit integer and pass it to pg_try_advisory_lock(bigint). You can also use the two-int32 form, pg_try_advisory_lock(int, int), with the first value as a namespace for "scheduled jobs" and the second as the job ID. Use the non-blocking try variants. A replica that waits on the lock would run the job late instead of skipping it.
On each tick, the replicas race like this:
- Replica A:
try_lock(key)returns true. A runs the job, then unlocks. - Replica B:
try_lock(key)returns false. B skips this tick. - Replica C:
try_lock(key)returns false. C skips this tick.
Collisions fail silently. If two job names hash to the same key, or another library uses advisory locks in the same key space, one job sees the lock held and skips without raising an error. Keep a registry of the keys you use, and log every skip.
Session locks belong to a backend connection. A transaction-mode pooler can give your unlock call to a different backend. The unlock then fails, and the lock leaks until that backend's connection closes. You have two ways around this:
- Hold one dedicated, unpooled connection for the whole job.
- Use
pg_try_advisory_xact_lock. Postgres releases this lock at commit or rollback.
The transaction-scoped option has costs if the job runs for a long time. The open transaction occupies a pool slot for the whole run. It also holds back the xmin horizon, so vacuum cannot remove dead row versions anywhere in the database, and tables bloat until the transaction ends.
import hashlib
import logging
import psycopg
log = logging.getLogger("cron")
def job_lock_key(job_name: str) -> int:
# Python's built-in hash() is salted per process, so it can't be used here.
# blake2b gives the same 64-bit value on every replica and every release.
digest = hashlib.blake2b(job_name.encode("utf-8"), digest_size=8).digest()
# Signed conversion keeps the value inside Postgres bigint range.
return int.from_bytes(digest, "big", signed=True)
def run_if_winner(dsn: str, job_name: str, job) -> bool:
key = job_lock_key(job_name)
with psycopg.connect(dsn) as conn:
with conn.transaction():
# Non-blocking and transaction-scoped. Losers skip instead of waiting.
# The lock follows the transaction, so it works through a transaction-mode pooler.
won = conn.execute(
"SELECT pg_try_advisory_xact_lock(%s::bigint)", (key,)
).fetchone()[0]
if not won:
# Log every skip. A hash collision or a clash with another
# advisory-lock user looks exactly like this line.
log.info("skip %s: advisory lock %d is held", job_name, key)
return False
# Run the job inside the same transaction. Keep it short, because an open
# transaction holds a pool slot and holds back vacuum.
job(conn)
# Postgres releases the lock at commit, or at rollback if job() raised.
return True
Each replica derives the same 64-bit key from the job name, calls pg_try_advisory_xact_lock inside a transaction, and either runs the job or logs a skip, with Postgres releasing the lock when the transaction ends.
Why doesn't a lock guarantee one run per schedule window?
An advisory lock gives you mutual exclusion: no two replicas run the job at the same moment. It says nothing about how many times the job runs per schedule window. Suppose a job is scheduled for 02:00 and finishes in 300 ms. Replica A fires at 02:00:00.1, takes the lock, finishes, and releases it. Replica B's clock is 800 ms behind, or its scheduler thread was paused, so it fires at 02:00:00.9. The lock is free, so the job runs a second time in the same minute.
The fix is to record the slot, not just hold a lock. Create a job_runs table with job_name, scheduled_for, status, started_at, and finished_at, plus a unique constraint on (job_name, scheduled_for). Before executing, each replica inserts a row for the slot. Compute scheduled_for by truncating to the schedule boundary, not by using the replica's current time. The first insert succeeds and that replica runs the job. Every later insert hits the constraint, and the replica skips. Clock skew no longer matters, because every replica computes the same slot value.
The same table covers the failure that try-lock-and-skip hides. If the holder crashes partway through, the other replicas already skipped, and nothing records that the run never finished. With job_runs, that run shows up as a row stuck in running with an old started_at. A slot with no row at all means every replica missed it. A periodic sweeper can query for both cases and alert or re-enqueue a catch-up run. Re-running is only safe if the job tolerates a partial previous attempt.
-- PostgreSQL. One row per (job, slot); the unique constraint enforces run-once-per-window.
CREATE TABLE job_runs (
job_name text NOT NULL,
scheduled_for timestamptz NOT NULL,
status text NOT NULL DEFAULT 'running', -- running | done | failed
started_at timestamptz NOT NULL DEFAULT now(),
finished_at timestamptz,
UNIQUE (job_name, scheduled_for)
);
-- Claim the slot BEFORE executing. $1 is the fire time from the cron expression
-- (e.g. 2026-11-08 02:00:00+00), never the replica's wall clock, so skewed replicas agree.
-- One row returned: this replica runs the job. Zero rows: the slot is already claimed, so skip.
INSERT INTO job_runs (job_name, scheduled_for)
VALUES ('nightly_invoice_rollup', $1)
ON CONFLICT (job_name, scheduled_for) DO NOTHING
RETURNING scheduled_for;
-- After the job succeeds, mark the slot finished.
UPDATE job_runs SET status = 'done', finished_at = now()
WHERE job_name = 'nightly_invoice_rollup' AND scheduled_for = $1;
-- Sweeper, case 1: the slot was claimed but the holder probably died partway through.
SELECT job_name, scheduled_for, started_at
FROM job_runs
WHERE status = 'running' AND started_at < now() - interval '30 minutes';
-- Sweeper, case 2: daily 02:00 UTC slots from the last 7 days that no replica claimed.
SELECT slot
FROM generate_series(
date_trunc('day', now(), 'UTC') - interval '7 days' + interval '2 hours',
now() - interval '5 minutes', -- grace period so a slot that is firing now isn't flagged
interval '1 day') AS slot
WHERE NOT EXISTS (
SELECT 1 FROM job_runs r
WHERE r.job_name = 'nightly_invoice_rollup' AND r.scheduled_for = slot
);
A PostgreSQL job_runs table that each replica claims with an insert before running the job, plus two sweeper queries that find runs stuck in 'running' and schedule slots with no row at all.
When should you use lease-based locks or leader election instead?
Advisory locks fit poorly in three cases. Your stack may have no Postgres. The lock may need to outlive a database connection, because a session lock is released when a pooler recycles the connection or a failover drops it. Or the job may run for hours, which keeps a connection pinned for the whole run.
A lease is a row or key that records an owner, an expiry time and a heartbeat. The holder renews the expiry on a timer, and another replica may take the lease only after the expiry has passed. The TTL must be longer than the worst pause the holder can suffer, such as a GC stall, a VM suspension or a network partition. If the TTL is shorter, a paused holder can wake up still believing it owns the lease. Expiry alone cannot prevent that, so each acquisition also issues an increasing fencing token. Downstream writes include the token, and the storage layer rejects any token lower than the highest it has already seen.
Leader election applies the same lease idea to the scheduler itself. It can run through a database lease or the orchestrator's built-in mechanism, such as a Kubernetes Lease object. One replica then runs every schedule, which is cleaner than taking a lock per job. The costs are that all scheduled load lands on one replica, failover takes up to a full TTL, and correctness depends on the election working.
| Approach | Removes | Adds |
|---|---|---|
| Lease with fencing | Connection coupling | Renewal logic, token checks downstream |
| Leader election | Per-job locking | Load on one replica, failover delay |
| External scheduler | In-app coordination | Another system to run and secure; job definitions stored apart from the code |
-- Lease row: owner, expiry, heartbeat time, and a fencing token that only increases.
CREATE TABLE job_leases (
job_name text PRIMARY KEY,
owner text NOT NULL,
expires_at timestamptz NOT NULL,
heartbeat_at timestamptz NOT NULL,
fencing_token bigint NOT NULL
);
-- Acquire: insert, or take over only if the current lease has expired.
-- Each takeover bumps the token. No row returned = someone else holds it.
INSERT INTO job_leases (job_name, owner, expires_at, heartbeat_at, fencing_token)
VALUES ('nightly-report', 'replica-a', now() + interval '120 seconds', now(), 1)
ON CONFLICT (job_name) DO UPDATE
SET owner = EXCLUDED.owner,
expires_at = EXCLUDED.expires_at,
heartbeat_at = EXCLUDED.heartbeat_at,
fencing_token = job_leases.fencing_token + 1
WHERE job_leases.expires_at < now()
RETURNING fencing_token;
-- Heartbeat: renew only while we still own this exact lease and it has not lapsed.
-- Zero rows updated means the lease was lost; stop the job.
UPDATE job_leases
SET expires_at = now() + interval '120 seconds', heartbeat_at = now()
WHERE job_name = 'nightly-report' AND owner = 'replica-a'
AND fencing_token = $1 AND expires_at > now();
-- Fenced write: storage rejects a token lower than the highest it has seen,
-- so a holder that wakes from a pause with an old token cannot overwrite newer output.
UPDATE report_output
SET body = $2, written_token = $1
WHERE report_date = current_date
AND written_token <= $1;
A PostgreSQL lease table with conditional acquisition, an owner-checked heartbeat, and a downstream write guarded by a fencing token so a stale holder's writes are rejected.
When should you avoid a distributed cron lock?
A lock is the wrong tool, or an unnecessary one, in these cases:
- The job must run on every replica. Some jobs act on per-instance state, such as clearing a local cache, rotating local temp files or refreshing in-memory configuration. A shared lock would make every replica except one skip work it actually needs.
- The work should be split across replicas, not serialized. If a job processes a large backlog, one lock holder doing all of it wastes the other replicas. Have the scheduler enqueue work items, and let workers claim them from a queue or with row-level claiming such as
SELECT ... FOR UPDATE SKIP LOCKED. - The job is already idempotent and cheap. If a duplicate run produces the same end state and costs little, such as recomputing a materialized summary with an upsert, the lock adds code and failure modes without changing the outcome. Idempotency alone may be enough.
- Only one instance can ever run. If the service truly runs as one instance and the deploy strategy stops the old instance before starting the new one, schedulers never overlap. Crash-and-restart misfires can still double-run a job, but idempotency or a
job_runsslot record handles that better than a lock. - There is no strongly consistent shared store. A lock in an eventually consistent or asynchronously replicated store can be granted to two replicas at once. That is worse than having no lock, because it looks safe.
- A platform scheduler already owns the schedule. If the orchestrator already launches one process per schedule, adding an in-app lock on top duplicates coordination. Rely on the platform's concurrency settings and make the job idempotent against its retries.
- The job runs for hours or must survive database failover. Connection-bound advisory locks are a poor fit here. Use a lease with fencing tokens, or move the job to its own workload, as described in the previous section.
How do idempotent jobs and monitoring save engineering time?
A lock lowers the odds of a duplicate run. Idempotency is what makes a duplicate harmless. Design each job so that running it twice for the same window leaves the same end state as running it once:
- Natural keys: identify work by business identity, such as
invoice_idplusbilling_period, rather than by a generated row ID. - Upserts: write with
INSERT ... ON CONFLICTagainst a unique constraint. The database then rejects the second write even when two replicas race. - Processed markers: record each completed unit of work in a table keyed by job name and window. Check the marker in the same transaction that does the work.
With these in place, at-least-once execution is acceptable, and a lock failure no longer causes a data-repair incident.
Observe the lock, not only the job
Log the hostname or pod name of the instance that acquired the lock, along with the window it ran for. When two instances log the same window, you have direct evidence of a duplicate run.
Alert on missed runs as well as failures. A broken lock or a crashed scheduler thread usually produces no error at all. The job simply stops running. A check such as "no successful run of nightly-settlement in 26 hours" catches that silence. An error-rate alert never will.
Development cost
Advisory locks plus idempotency run on the database and deploy pipeline the team already operates. Leader election and external schedulers each add a component that someone has to build, upgrade, secure and debug during incidents.
Decision checklist
- Can the job be made idempotent? Do that first, whatever else you choose.
- Do jobs finish well within their interval, and does the app already use Postgres? Use advisory locks.
- Do jobs run long, or do you need fencing against stalled holders? Use lease-based locks with fencing tokens.
- Do you have many jobs, need cross-service scheduling, or need audit history? Consider leader election or an external scheduler, and budget for operating it.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to distributed cron lock, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.