Why do scheduled tasks run multiple times when a monolith scales out?

A monolith usually defines its scheduled tasks in application code, using an in-process scheduler that starts when the app boots. On one server, a job defined once runs once. When you scale out to N replicas behind a load balancer, every replica boots the same scheduler with the same job definitions. Each one fires independently, so the job runs N times per tick. Nothing in the scheduler knows that other copies exist.

Rolling deploys add a second source of duplicates. During a rollout, old and new instances run side by side. A job scheduled near the deploy can fire on both generations, even if you normally run a single replica.

Without a lock                      With a distributed cron lock

 app-1 [sched] --run-->+             app-1 [sched] --acquire OK--> run
 app-2 [sched] --run-->+--> DB       app-2 [sched] --acquire FAIL--> skip
 app-3 [sched] --run-->+             app-3 [sched] --acquire FAIL--> skip
   3 executions per tick               1 execution per tick

A distributed cron lock is a shared lock, usually held in the database or another store all replicas can reach. Each instance must acquire it before running a job. One instance wins and the others skip that run.

The lock answers only part of the problem. Every scheduled job needs answers to three questions:

  1. Who runs it? This is the lock's job: exactly one instance per scheduled run.
  2. What happens if it runs twice? Locks can fail, expire early, or be released during a crash. The job itself must tolerate a duplicate.
  3. What happens if it doesn't run? If the lock holder dies mid-run, or no instance acquires the lock, you need a way to notice the gap and recover.
WITHOUT A LOCK: every replica's in-process scheduler fires on each tick

+---------------+
| app-1 [sched] |--run job--+
+---------------+           |
+---------------+           v        +-------------+
| app-2 [sched] |--run job--+------->|  shared DB  |  3 executions / tick
+---------------+           ^        +-------------+
+---------------+           |
| app-3 [sched] |--run job--+
+---------------+

WITH A DISTRIBUTED CRON LOCK: acquire first, run only if acquired

+---------------+                    +-------------+
| app-1 [sched] |--acquire: OK------>|  shared DB  |  1 execution / tick
|               |--run job---------->| (lock table)|
+---------------+                    |             |
+---------------+                    |             |
| app-2 [sched] |--acquire: FAIL---->|             |
|               |  skip this run     |             |
+---------------+                    |             |
+---------------+                    |             |
| app-3 [sched] |--acquire: FAIL---->|             |
|               |  skip this run     |             |
+---------------+                    +-------------+

Rolling deploy: old and new replicas overlap, so even more schedulers fire.

Without a lock, each replica's scheduler runs the job against the shared database on every tick. With a distributed cron lock, only the instance that acquires the lock runs the job and the others skip it.

How does a distributed cron lock work?

The scheduler itself does not change. Every replica still loads every job definition and fires on every tick. The lock is a thin gate between "the schedule fired" and "the job body runs." Each run goes through the same steps:

  1. Fire. The in-process scheduler on each replica triggers the job at its scheduled time. Clocks differ slightly between hosts, so the triggers arrive at slightly different moments.
  2. Derive a key. Each instance computes the same lock key for the job. It may be derived from the job name alone (one lock per job) or from the job name plus the scheduled fire time (one lock per schedule slot). The key must be identical on every instance and across deploys.
  3. Try to acquire, without waiting. The instance makes one atomic request to the shared store: a PostgreSQL advisory lock, a conditional update on a lease row, a unique insert into a run table, or a set-if-not-exists in a key-value store. The store guarantees that only one request succeeds for a given key at a time.
  4. Run or skip. The instance that acquired the lock runs the job. Every other instance logs a skip and returns immediately. Instances must not block waiting for the lock, because a waiter would run the job again as soon as the winner released it.
  5. Hold, renew, or record. While the job runs, the holder keeps the lock alive. Depending on the mechanism, it holds a transaction open, renews a lease with a heartbeat, or updates a run row.
  6. Release or expire. On success the holder releases the lock or marks the slot complete. If the holder crashes, the lock is freed by the session ending, the transaction rolling back, or the lease expiring, so that a later run can proceed.

The store does the coordination. The application only needs atomic "acquire if free" and "release only if I own it" operations. The sections below cover three ways to build those on a database: advisory locks, lease rows, and per-slot run rows.

What are the alternatives to a distributed cron lock?

There are five common ways to stop a scaled monolith from running the same job on every instance. They differ in where the coordination happens.

Dedicated scheduler process. One instance or process type runs the scheduler, and web instances don't. This is simple and needs no locking code. The cost is a single point of failure. A deploy restart that overlaps a schedule slot silently skips that run.

External trigger. Platform cron calls an endpoint or enqueues a job. This keeps scheduling out of the app. It does not remove duplicates, though. Triggers can retry or fire twice, so deduplication moves to request or enqueue time.

Per-job distributed cron lock. Every instance schedules every job, and a lock on the existing database decides which instance runs it. No new infrastructure is needed.

Leader election. One instance is elected leader and runs all jobs. This pays off with many jobs or long-lived scheduling state. For a handful of jobs it adds failover logic you must test.

Job queue. The scheduler only enqueues work, and workers execute it. You get retries and visibility. The scheduler still needs one of the options above.

For a few jobs, a database lock is usually the right default. It costs little engineering time, but only if jobs are idempotent and resumable. Lock expiry or failover will eventually produce a duplicate or partial run.

Option Setup effort Extra infrastructure Duplicate risk Missed-run risk Best fit
Dedicated scheduler Low Process type Low High (restarts, crashes) Non-critical jobs
External trigger Low Platform cron Medium (retries) Low Hosted platforms
Distributed cron lock Low–medium None Low if idempotent Medium Handful of jobs
Leader election High Often a coordinator Low Medium (failover gaps) Many jobs
Job queue Medium Queue/broker Moved to enqueue Low Heavy or retryable work

Who should use a distributed cron lock?

A distributed cron lock fits teams in this position:

  • You run a monolith with an in-process scheduler on more than one replica, or on one replica with rolling deploys that overlap old and new instances.
  • You already have a shared database that every replica can reach, and you are comfortable with the scheduler depending on its availability.
  • You have a small number of jobs, such as nightly reports, cleanup tasks, digest emails, or periodic syncs, and each one is independent of the others.
  • Your jobs finish well within a predictable time, so a lease length or an open transaction is easy to size.
  • Your jobs can be made idempotent, or already are, so an occasional duplicate run is harmless.
  • You do not want to add and operate new infrastructure, such as a coordination service or a message broker, just to schedule a few tasks.

It also suits teams that want job definitions to stay in application code, versioned and reviewed with the rest of the app, instead of being split across platform configuration.

When should you avoid a distributed cron lock?

A lock is the wrong tool in several situations:

  • Side effects cannot be made idempotent and duplicates are unacceptable. A lock reduces duplicates. It does not guarantee exactly-once execution, because leases expire, connections drop, and failover discards advisory locks. If a second run would cause irreversible harm and you cannot add idempotency keys or state guards, fix that first or use a design with fencing tokens checked at the write target.
  • Jobs are long-running or unbounded. Transaction-scoped advisory locks keep a transaction open for the whole job and hold back vacuum. Leases need careful sizing and heartbeats. Long or variable work is usually better split into batches on a job queue.
  • Jobs run very frequently. A job that fires every few seconds turns each tick into a lock round-trip from every replica against the primary. That load and contention may outweigh the benefit. A dedicated scheduler or a queue fits better.
  • You have many jobs with ordering or shared state. Dozens of jobs, or jobs that must run in sequence, are easier to reason about under one elected leader than as many independent locks.
  • The primary database is already the bottleneck or fails over often. The lock adds load to the primary and ties scheduling to its availability. Advisory locks are not replicated, so frequent failovers mean frequent windows of duplicate execution.
  • Your platform already guarantees a single scheduled invocation, or your instances scale to zero. If no instance is running at the scheduled time, an in-process scheduler fires nothing, and the lock does not help. An external trigger is the better starting point.
  • The work is heavy enough to need retries, backoff, and worker isolation. Use a job queue for execution. You may still use a lock to make sure only one instance enqueues.

How can a distributed cron lock save engineering time?

Most of the savings come from not building or operating anything new:

  • No new infrastructure. The lock reuses the database you already provision, back up, monitor, and secure. There is no coordination cluster to run, no broker to upgrade, and no extra on-call surface.
  • A small, local code change. The lock wraps the job at a single hook: try to acquire, run or skip, release. Existing job code and schedule definitions stay where they are, so adopting it is a contained change rather than a migration.
  • Scaling stays a configuration change. Once jobs are gated, adding replicas or changing deploy strategy no longer requires a review of which jobs might double-fire.
  • Fewer duplicate-run incidents. Double-sent emails, double-applied charges, and conflicting writes are expensive to clean up because someone has to find and reverse each side effect. Preventing most of them up front, and making the rest harmless through idempotency, removes that class of cleanup work.
  • Faster debugging. A run table with one row per slot answers "did it run, where, and how many times" with a single query, instead of correlating logs across instances.
  • A path forward without rewrites. Jobs written to be idempotent and checkpointed under a lock carry over to a job queue or leader election largely unchanged if requirements grow.

The savings depend on keeping the lock simple. A lock that needs custom failover logic, clock synchronization, or per-job tuning loses most of its advantage over the alternatives.

How do PostgreSQL advisory locks work for scheduled tasks?

PostgreSQL advisory locks are application-defined mutexes keyed by a 64-bit integer. Each instance calls pg_try_advisory_lock(key) when its schedule fires. The call returns true or false immediately and never waits. The instance that gets true runs the job, and the others skip that tick. Skipping is the right behaviour here. A blocking pg_advisory_lock would queue every instance behind the winner, and each would run the job again once the lock was released.

The key must be the same on every instance and every deploy. Hashing the job name in SQL with hashtext() is convenient, but that function is internal and its output is not guaranteed to stay stable. A safer approach is to compute a fixed hash in application code or keep job keys in a table.

Session-scoped vs transaction-scoped

A session-scoped lock (pg_try_advisory_lock) is held by a backend connection until you unlock it or the session ends. That causes two problems:

  • Transaction-mode poolers. A pooler in transaction mode returns the server connection to the pool after each transaction. The lock stays attached to a backend your app no longer controls. A later unlock may go to a different backend and fail, leaving the original lock held indefinitely.
  • Resets and failover. If the connection drops, the lock is released while your job keeps running. Advisory locks are not replicated, so a promoted standby starts with no locks held, and a second instance can acquire the lock alongside the first.

A transaction-scoped lock (pg_try_advisory_xact_lock) is released at commit or rollback, which makes it work through poolers. The cost is that you must hold one transaction open for the whole job, and long transactions have their own side effects, such as holding back vacuum.

Using advisory locks also ties your scheduler to the primary's availability and load. That is reasonable if you choose it deliberately rather than by default.

import hashlib
import os

import psycopg  # psycopg 3

DSN = os.environ.get("DATABASE_URL", "postgresql://app:changeme@db.internal:5432/app")


def job_lock_key(job_name: str) -> int:
    # Derive the key in app code so it stays the same across instances, deploys
    # and PostgreSQL versions. 8 bytes, signed, fits PostgreSQL's bigint.
    digest = hashlib.blake2b(job_name.encode("utf-8"), digest_size=8).digest()
    return int.from_bytes(digest, "big", signed=True)


def run_if_leader(job_name: str, job) -> bool:
    key = job_lock_key(job_name)
    with psycopg.connect(DSN) as conn:
        # Transaction-scoped lock: released automatically at commit or rollback,
        # so it never outlives the transaction even behind a transaction-mode pooler.
        with conn.transaction():
            got_lock = conn.execute(
                "SELECT pg_try_advisory_xact_lock(%s)", (key,)
            ).fetchone()[0]
            if not got_lock:
                # Another instance holds it: skip this tick instead of waiting.
                return False
            # The transaction stays open for the whole job. Keep jobs short,
            # because long transactions hold back vacuum.
            job(conn)
            return True


if __name__ == "__main__":
    ran = run_if_leader("nightly-invoice-rollup", lambda conn: print("running job"))
    print("ran" if ran else "skipped: lock held elsewhere")

A Python (psycopg 3) scheduler hook that derives a stable bigint key from the job name in application code and uses pg_try_advisory_xact_lock so only one instance runs each tick and the lock is released when the transaction ends.

How do you implement a lease lock without advisory locks?

If you want the lock to work on any SQL database, or to stay independent of session state, use a lease stored in a row. Create a job_locks table with job_name as the primary key, an owner column holding a unique ID per process or run, and a locked_until timestamp.

To acquire the lease, run a conditional UPDATE that sets owner and locked_until = now() + lease only WHERE job_name = ? AND locked_until < now(), then check that exactly one row changed. Alternatively, use INSERT ... ON CONFLICT (job_name) DO UPDATE with the same expiry condition, so the first run doesn't need a seeded row. Upsert syntax varies by database. To release, update or delete only WHERE job_name = ? AND owner = ?.

Base the lease on worst-case runtime, not typical runtime. For long jobs, keep the lease short and renew it with a heartbeat that also filters on owner. If a renewal touches zero rows, you have lost the lock, and the job should stop. Always compute expiry with the database's now(). Application clocks on different hosts drift, and skew silently shortens or extends leases.

Redis-style locks follow the same rules: set-if-not-exists with an expiry and a random token. Release must be an atomic compare-and-delete. A plain delete is a bug: if your lease already expired and another instance acquired the key, you delete their lock.

Expiry is still a guess. A GC pause, a slow query or a stalled host can outlive the lease while the original holder keeps writing, so two holders can run at once. Without fencing tokens checked at the write target, or idempotent writes, a lease only reduces duplicates. It does not prevent them.

-- PostgreSQL syntax; upsert syntax differs on other databases.
CREATE TABLE job_locks (
  job_name     text        PRIMARY KEY,
  owner        text        NOT NULL,  -- unique per process/run, e.g. host:pid:uuid
  locked_until timestamptz NOT NULL
);

-- Acquire: insert if no row exists, or take over only if the old lease expired.
-- now() is the database clock, so app-host clock skew does not matter.
-- Run in autocommit: now() is fixed at transaction start.
INSERT INTO job_locks (job_name, owner, locked_until)
VALUES ('nightly-report', 'worker-b:4242:7f3c9a1e', now() + interval '2 minutes')
ON CONFLICT (job_name) DO UPDATE
  SET owner = EXCLUDED.owner,
      locked_until = EXCLUDED.locked_until
  WHERE job_locks.locked_until < now()
RETURNING owner;
-- One row returned: you hold the lease. Zero rows: someone else does, so skip this run.

-- Heartbeat (e.g. every 30s for long jobs): extend only if we still own a live lease.
UPDATE job_locks
   SET locked_until = now() + interval '2 minutes'
 WHERE job_name = 'nightly-report'
   AND owner = 'worker-b:4242:7f3c9a1e'
   AND locked_until > now();
-- 0 rows updated: the lease is lost. Stop the job instead of continuing to write.

-- Release: the owner filter keeps us from deleting another instance's lease
-- if ours already expired and was taken over.
DELETE FROM job_locks
 WHERE job_name = 'nightly-report'
   AND owner = 'worker-b:4242:7f3c9a1e';

A PostgreSQL row-based lease lock that acquires with an expiry-guarded upsert, renews with an owner-filtered heartbeat and releases only the caller's own lease, using database time throughout.

Should you lock per run or per schedule slot?

A lock held for the duration of a run answers "is anyone running this job right now?" It does not answer "has the 02:00 run happened?" Those questions diverge as soon as schedulers fire at slightly different times. Suppose instance A runs the nightly job, finishes at 02:00:40, and releases the lock. Instance B's scheduler fires late at 02:00:50, finds the lock free, and runs the same job a second time.

Keying on the schedule slot closes that gap. Before doing any work, each instance inserts a row into a runs table with a unique constraint on (job_name, scheduled_time). One insert succeeds and that instance does the work. The others get a unique violation and stop. The row stays after the run completes, so an instance that wakes up late cannot repeat the slot. Store status and timestamps on the same row, and the table becomes a history. On startup, an instance can enumerate the slots expected since the last completed run, then backfill the missing ones if the job tolerates late execution, or alert if it doesn't.

Compute scheduled_time from the cron expression's intended fire time, never from now() truncated at execution. If two instances' clocks differ by a few seconds, wall-clock truncation near a boundary can produce two adjacent keys, and both instances will run.

The slot row also fixes a quiet failure of skip-if-locked. If the lock holder crashes mid-run, the lock is released, but every other instance has already skipped, so the slot is lost with no record of it. A slot row stuck in started with a stale heartbeat is visible, and another instance can reclaim it.

-- PostgreSQL. One row per (job, slot); the row outlives the run.
CREATE TABLE job_runs (
  job_name       text        NOT NULL,
  scheduled_time timestamptz NOT NULL,  -- cron's intended fire time, never now() truncated
  status         text        NOT NULL DEFAULT 'started',  -- started | completed | failed
  owner          text        NOT NULL,
  heartbeat_at   timestamptz NOT NULL DEFAULT now(),       -- DB clock, so instance skew is irrelevant
  completed_at   timestamptz,
  PRIMARY KEY (job_name, scheduled_time)
);

-- Claim the slot: one instance gets a row back, the rest get zero rows and stop.
INSERT INTO job_runs (job_name, scheduled_time, owner)
VALUES ('nightly-report', '2026-11-17 02:00:00+00', 'app-instance-b')
ON CONFLICT (job_name, scheduled_time) DO NOTHING
RETURNING scheduled_time;

-- Owner heartbeats while working, then marks the slot done.
UPDATE job_runs SET heartbeat_at = now()
WHERE job_name = 'nightly-report' AND scheduled_time = '2026-11-17 02:00:00+00' AND owner = 'app-instance-b';
UPDATE job_runs SET status = 'completed', completed_at = now()
WHERE job_name = 'nightly-report' AND scheduled_time = '2026-11-17 02:00:00+00' AND owner = 'app-instance-b';

-- Reclaim a slot whose owner crashed mid-run. The WHERE is re-checked under the
-- row lock, so if two instances race, only one sees the stale heartbeat.
UPDATE job_runs SET owner = 'app-instance-c', heartbeat_at = now()
WHERE job_name = 'nightly-report' AND scheduled_time = '2026-11-17 02:00:00+00'
  AND status = 'started' AND heartbeat_at < now() - interval '10 minutes'
RETURNING scheduled_time;

-- On startup: expected 02:00 UTC slots in the last 7 days with no completed run.
SELECT s.slot
FROM generate_series(date_trunc('day', now() AT TIME ZONE 'UTC') - interval '7 days' + interval '2 hours',
                     now() AT TIME ZONE 'UTC', interval '1 day') AS s(slot)
LEFT JOIN job_runs r
  ON r.job_name = 'nightly-report' AND r.scheduled_time = s.slot AT TIME ZONE 'UTC'
 AND r.status = 'completed'
WHERE r.job_name IS NULL;  -- backfill these if the job tolerates late runs, otherwise alert

A PostgreSQL run table keyed on (job_name, scheduled_time) that claims each slot exactly once, reclaims slots left in 'started' by a crashed instance, and lists missed slots for catch-up after downtime.

How do you make scheduled jobs safe when the lock fails?

Every lock eventually fails: a lease expires during a long garbage-collection pause, a network partition hides the holder, or a deploy restarts an instance mid-run. Treat the lock as an optimization that prevents wasted work, and make the job itself safe to run twice.

Idempotency keys. Derive a key from the job name and the schedule slot, such as invoice-run:2026-11-17, and store it with a unique constraint alongside any side effect. A second run that tries to insert the same key fails at the database instead of sending a second email or charging a card twice.

Bounded batches with checkpoints. Process a fixed number of rows per transaction and record progress, such as the last processed ID, in the same commit. If the instance dies, the next run picks up from the checkpoint. No one has to work out by hand which half of the data was touched.

Row claiming with SELECT ... FOR UPDATE SKIP LOCKED. If two runs overlap, each claims different rows instead of blocking or colliding. Overlap then splits the work between the runs and does not duplicate it.

State-transition guards. Write UPDATE ... SET status = 'sent' WHERE id = $1 AND status = 'pending' and check the affected row count. A zero means another worker already did the work, so skip it.

These patterns lower cost in two ways. During development, you stop debating edge cases in the lock, because a rare double run causes no harm. During incidents, a partial run resumes on its own, and you skip the forensic cleanup. They also cover most of what a job queue needs. If you later move to a queue, which typically delivers at least once, the job logic carries over largely unchanged.

When is leader election worth the extra complexity?

Leader election moves coordination up a level. Instead of each job acquiring its own lock, the instances elect one leader, and only the leader runs the scheduler for every schedule. The other instances stay idle until leadership changes hands.

What it adds

You get a single scheduler with a simple mental model: one process owns the clock. Jobs can depend on one another, run in a defined order, or share in-memory state, because nothing else is scheduling concurrently. You also stop adding a lock to each new job.

What it requires

Leadership is a lease, so the leader has to renew it well before it expires. A leader that stalls during a long GC pause, a network partition, or a frozen VM can wake up still believing it leads while another instance has already taken over. Fencing tokens address this. Each new leadership term gets a monotonically increasing number, and the systems the jobs write to reject writes that carry an older token. When the leader loses its lease mid-job, it should cancel in-flight work, and the new leader must be able to resume or safely rerun that work.

A correct election needs a consensus-backed store. That means running a quorum of nodes, understanding failure modes such as split brain, and adding another dependency your team has to monitor and operate.

When to use it

Leader election fits when you have many jobs, need strict ordering between them, or already operate a coordination service. For a few nightly jobs, per-job locks give you the same safety with far less to run.

How do you monitor scheduled jobs and detect missed runs?

Locking makes duplicate runs rare. Monitoring tells you whether the lock is actually working and whether the job ran at all.

Log every lock decision

Emit a structured log line each time an instance acquires the lock, skips because another holder has it, or releases it. Include the job name, the schedule slot, and an instance identifier such as hostname or pod name. Skips are expected and should be cheap to log. Without them, you cannot tell "another instance ran it" from "nobody tried."

Record each run in the run table

Write the start time, end time, outcome, and instance ID for every run against its slot. This record is the source of truth when logs have rotated or were sampled. A unique constraint on job and slot also shows any duplicate execution as a constraint violation instead of a silent double charge.

Alert on absence, not just failure

A job that never starts produces no error. Alert when no successful run exists for a slot after its expected window closes. A query against the run table, checked by something outside the application, catches crashed schedulers, deadlocked lock holders, and misconfigured cron expressions.

Compare duration to lease length

Track run duration as a fraction of the lease. A job that creeps from 30% to 85% of its lease will eventually outlive it and let a second instance start. Alert on that trend before it happens.

During an incident, these records answer "did it run, where, and how many times" with a query in minutes. Without them, engineers reconstruct the answer from partial logs across several instances.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to distributed cron lock, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.