Cron Jobs on Multiple Instances: Stopping Duplicate Scheduled Runs

Why cron jobs on multiple instances fire more than once

An in-process scheduler has no idea that other copies of the application exist. Each replica reads the same cron expression from the same configuration, keeps its own clock, and fires when that expression matches. Three replicas produce three runs of the nightly invoice job. Ten replicas produce ten. Nothing has failed: each process is doing exactly what it was told, and none of them has a shared view of whether the work has already started.

The count also changes over time, which is why testing rarely catches it. A local environment or a single-replica staging deployment shows one execution and looks correct. In production, an autoscaler adds instances under load, so a job that ran twice at noon may run five times during an evening peak. A rolling deploy keeps old and new pods alive together for a while, and a schedule that fires during that overlap runs on both generations. A blue/green cutover can briefly double the fleet. Each of these is a short window, so the duplicates show up sporadically and are hard to reproduce.

Treat this as a correctness bug. Changing replica counts or shifting the schedule only alters how often it happens. Duplicate runs send repeated emails, charge cards twice, and insert conflicting rows.

Any team that runs a horizontally scaled monolith with scheduled work is affected. Moving from one instance to several makes the problem appear without any change to the code.

Three structural options: external trigger, lock per execution, elected leader

You can make scheduled work run once in three structural ways. They differ in where the decision is made.

Copy(1) External trigger      (2) Lock per run            (3) Elected leader
  [scheduler]              [A] [B] [C]                 [A*] [B] [C]
      |                     \   |   /                   |
      v                      [lock table]               runs all jobs
  [one instance             winner runs,               B, C idle until
   or worker pod]           others skip                lease expires

External trigger. Something outside the app fires each run: an orchestrator-level scheduled job, a dedicated worker deployment, or an external scheduler calling an internal endpoint. Uniqueness becomes the orchestrator's job. Orchestrators do not fully guarantee it, though. A Kubernetes CronJob defaults to concurrencyPolicy: Allow, so a slow run can overlap the next one. Job retries can also re-run work that half-finished.

Lock per execution. Every instance keeps its scheduler. Before each run, an instance tries to take a lock in shared storage. The instance that gets the lock runs the job, and the others skip it.

Elected leader. One instance holds a renewable lease and runs all scheduled work. If it dies, another instance takes over once the lease expires.

Config flag on one instance. Setting SCHEDULER_ENABLED=true on one replica looks simple, but it fails quietly. If that replica is scaled down, renamed, or replaced by a deploy that drops the flag, jobs stop running and nothing raises an error. It also has no failover.

Copy(1) EXTERNAL TRIGGER      (2) LOCK PER RUN          (3) ELECTED LEADER
+--------------------+    +--------------------+    +--------------------+
| orchestrator job / |    | [A]    [B]    [C]  |    | lease store        |
| ext. scheduler     |    | each has scheduler |    | holder=A, ttl      |
+---------+----------+    +--+------+------+---+    +---------+----------+
          | fires            |      |      |                  ^ renew
          v                  v      v      v                  |
+--------------------+    +--------------------+    +---------+----------+
| worker pod or      |    | lock table (DB)    |    | [A*] runs all jobs |
| internal endpoint  |    | report -> A wins   |    +--------------------+
+--------------------+    +---------+----------+    | [B] [C] standby;   |
                                    |               | take over on expiry|
                                    v               +--------------------+
                            A runs; B, C skip

risk: overlap with        risk: lock TTL < run      risk: no runs until
concurrencyPolicy:        time; lock store is       old lease expires;
Allow; retries re-run     a new dependency          clock/TTL errors can
half-done work                                      allow two leaders

CONFIG FLAG (anti-pattern): scheduler on one named replica only
+-----------------------+   +-----------+   +-----------+
| [A] SCHEDULER_ENABLED |   | [B] false |   | [C] false |
+-----------+-----------+   +-----------+   +-----------+
            | scaled down, renamed, or redeployed without the flag
            v
   nothing schedules: jobs stop silently, no error, no failover

Where the run-once decision lives in each topology (external trigger, per-run lock, elected leader) and the weak point of each, compared with the config-flag approach, which stops running jobs without any error when its one flagged replica goes away.

How a ShedLock-style lock table works

The lock-per-execution model needs one shared table. Each job name gets a single row with four columns: name (primary key), lock_until, locked_at and locked_by (usually hostname or pod name). Every instance runs its scheduler normally. When a trigger fires, the instance asks the database for permission before running the job body.

Acquiring the lock

Acquisition is one conditional statement: UPDATE ... SET lock_until = now + lockAtMostFor, locked_at = now, locked_by = ? WHERE name = ? AND lock_until <= now. The database serializes concurrent updates to the same row, so only one instance can match the predicate. The rule is strict: run only if exactly one row was affected. If zero rows are affected, either another holder exists or the row doesn't exist yet. The provider then tries an INSERT, and a primary-key violation means another instance won.

When finished, the holder releases by setting lock_until back to now, or to the lockAtLeastFor floor covered in the next section.

Compute now on the database side where the lock provider supports it. If each instance supplies its own timestamp, clock drift causes two failures:

• A fast clock can treat a live lock as expired, which produces two holders.
• A slow clock can write a lock_until far in the future, which leaves the lock stuck.

Table hygiene

• Renamed jobs leave orphaned rows. They are harmless but misleading, so prune them.
• Shared databases need namespaced lock names. If several environments share one database, prefix names such as staging:invoice-sweep. Otherwise staging can silently suppress production.
• Rolling deploys can cause confusing skips. During a deploy, one job name may map to old logic on some pods and new logic on others. Whichever version grabs the lock runs, and the other version's run is skipped. Rename the lock when job semantics change.

Copy-- PostgreSQL. One row per job name.
CREATE TABLE shedlock (
  name       VARCHAR(64)  PRIMARY KEY,  -- namespaced, e.g. 'prod:invoice-sweep'
  lock_until TIMESTAMP    NOT NULL,
  locked_at  TIMESTAMP    NOT NULL,
  locked_by  VARCHAR(255) NOT NULL       -- hostname or pod name
);

-- Acquire: take an expired lock. Time comes from the DB clock, not the instance.
-- lockAtMostFor = 10 minutes here. Run the job only if this affects exactly 1 row.
UPDATE shedlock
   SET lock_until = CURRENT_TIMESTAMP + INTERVAL '10 minutes',
       locked_at  = CURRENT_TIMESTAMP,
       locked_by  = 'worker-pod-7f9c'
 WHERE name = 'prod:invoice-sweep'
   AND lock_until <= CURRENT_TIMESTAMP;

-- If 0 rows were affected, the row may not exist yet. Insert it.
-- 1 row inserted means we hold the lock. 0 rows means another instance won.
INSERT INTO shedlock (name, lock_until, locked_at, locked_by)
VALUES ('prod:invoice-sweep', CURRENT_TIMESTAMP + INTERVAL '10 minutes',
        CURRENT_TIMESTAMP, 'worker-pod-7f9c')
ON CONFLICT (name) DO NOTHING;

-- Release after the job body finishes. Honor a lockAtLeastFor floor of 30 seconds.
UPDATE shedlock
   SET lock_until = GREATEST(locked_at + INTERVAL '30 seconds', CURRENT_TIMESTAMP)
 WHERE name = 'prod:invoice-sweep'
   AND locked_by = 'worker-pod-7f9c';

-- Hygiene: find rows left behind by renamed or removed jobs.
SELECT name, locked_at, locked_by
  FROM shedlock
 WHERE locked_at < CURRENT_TIMESTAMP - INTERVAL '30 days';

A PostgreSQL lock table with conditional acquire, insert-on-first-use, release with a minimum hold, and a query that finds orphaned rows, all timed by the database clock.

Choosing lockAtMostFor and lockAtLeastFor

lockAtMostFor is the safety ceiling. If the holder crashes, gets OOM-killed, or loses its network mid-run, nothing releases the lock explicitly. The row stays until this timeout expires. Set it well above the worst-case runtime you have actually seen. If it is too short, a slow but healthy run loses its lock while still working, and a second instance starts the same job alongside it.

lockAtLeastFor is a minimum hold. A job that finishes in 300 ms would otherwise release the lock immediately. Another instance whose clock or trigger is a few seconds behind could then acquire the lock and run the same scheduled slot again. Keep this value longer than any plausible clock skew and shorter than the schedule interval.

Illustrative estimate: assume a job runs every 5 minutes, usually takes 2 minutes, and has peaked at 9. A 20-minute ceiling gives comfortable headroom. The cost is that if a run crashes at 12:00, the lock holds until 12:20, and the runs at 12:05, 12:10 and 12:15 are all skipped.

No single setting removes both risks. A shorter ceiling invites overlapping runs, and a longer one causes more missed runs after a crash. Skipped runs leave no record, so a job that must cover every interval should not assume "since the last tick." It should persist a high-water mark, such as the last processed timestamp or ID, and resume from there.

If the lock database is unreachable, no instance can acquire the lock, so jobs skip rather than run. That failure is silent unless you make it visible. Log and count every failed acquisition so the skips show up in monitoring.

Leader election for a single scheduler

Instead of locking each execution, you can elect one instance as leader and start the scheduler only there. Followers keep the scheduler code loaded but dormant. Leadership is a renewable lease, and there are three common ways to hold one:

• A lease row with a heartbeat. The leader updates an expires_at column. Another instance can take over only after that time has passed.
• A session-level advisory lock. The database releases it when the holder's connection drops.
• The coordination primitive of your orchestrator or consensus store. Use this if you already run one.

The loop has three phases:

1. Acquire. Try to take the lease. If you succeed, start the scheduler.
2. Renew. Renew well inside the lease duration. A common choice is every third of the lease, so one slow round trip doesn't cost you leadership.
3. Step down. Do this when renewal fails or when your local deadline passes, whichever comes first.

On step-down, stop scheduling new work immediately and, ideally, cancel in-flight jobs. After that point another instance may legitimately believe it is the leader. Measure the deadline from when you sent the last successful renewal, not when it returned. A paused process can wake up believing it still leads. Most split-brain bugs in this pattern come from step-down code that logs the failure but leaves the scheduler running.

The trade-offs:

• Load concentration. Every scheduled job lands on one instance, where request traffic can starve it, or the jobs can starve the requests.
• Failover gap. When a leader dies, nothing runs until its lease expires. A short lease shrinks that gap but makes spurious step-downs more likely during GC pauses or network blips.

Copyimport time

LEASE_S = 15            # failover gap is at most about this long
RENEW_S = LEASE_S / 3   # one slow round trip doesn't cost leadership

# One statement both acquires and renews. It uses the database clock only.
TRY_LEASE = """
UPDATE leases
   SET holder = %(me)s, expires_at = now() + make_interval(secs => %(ttl)s)
 WHERE name = 'scheduler' AND (holder = %(me)s OR expires_at < now())
"""

def try_lease(conn, me):
    with conn.cursor() as cur:
        cur.execute(TRY_LEASE, {"me": me, "ttl": LEASE_S})
        ok = cur.rowcount == 1
    conn.commit()
    return ok

def run(conn, me, scheduler):  # scheduler: your start/stop/cancel wrapper
    leading, deadline = False, 0.0
    while True:
        sent = time.monotonic()  # deadline counts from send, not reply
        try:
            ok = try_lease(conn, me)
        except Exception:
            conn.rollback()  # may raise too; any error means renewal failed
            ok = False
        if ok:
            deadline = sent + LEASE_S
            if not leading:
                scheduler.start()
                leading = True
        elif leading or time.monotonic() >= deadline:
            # Another instance may already lead. Stopping is mandatory.
            scheduler.stop()            # no new runs
            scheduler.cancel_running()  # abort in-flight jobs
            leading = False
        time.sleep(RENEW_S)

A sketch of a Postgres lease row with acquire, renew and step-down. The leader renews every third of the lease and measures its deadline from when each renewal was sent. On a failed renewal it stops the scheduler and cancels running jobs instead of only logging the error.

Fencing tokens and idempotent jobs as the real safety net

Every scheme so far shares one weakness: the process holding the lock cannot tell when it has lost it. A long GC pause, a frozen VM or a network partition can stall a worker past lockAtMostFor or its lease. Meanwhile a second instance acquires the lock and starts the same job. When the first process resumes, it still believes it owns the lock and keeps writing. Locks reduce duplicate scheduled tasks. They cannot prevent them.

Reject stale holders at the write

A fencing token moves the check from the scheduler to the data. Each lock acquisition returns a monotonically increasing number. The job includes that number with every write, and the target refuses any write carrying a token lower than the highest one it has already seen. A version column works the same way when the job updates rows: UPDATE ... WHERE version = :expected affects zero rows if a newer holder got there first. This only protects resources that check the token. A plain HTTP endpoint will not.

Make the work itself idempotent

• Natural keys. Derive the record's identity from the business fact, such as customer and billing date, rather than generating a fresh ID on each run.
• Processed markers. Write the "done" marker in the same transaction as the work. If the marker commits later, a crash between the two steps leaves the work done but unmarked, and the next run repeats it.
• Unique constraints on outbound records. A constraint on invoice_id plus period makes the database reject a second invoice outright.

The cost

These protections are not free. You need dedupe tables and schema changes to jobs that were never designed for retries. Third-party calls that accept no idempotency key are the hardest case. A duplicate charge or email cannot be blocked by your database, so it needs compensating actions or manual reconciliation.

Copy-- PostgreSQL. :token is the fencing token returned by the lock service (bind parameter).
CREATE TABLE job_fence (
  job_name  text PRIMARY KEY,
  max_token bigint NOT NULL          -- highest token this resource has accepted
);
INSERT INTO job_fence VALUES ('monthly_invoices', 0);

CREATE TABLE invoices (
  invoice_id   text   NOT NULL,      -- natural key, e.g. 'INV-cust-4211'
  period       text   NOT NULL,      -- e.g. '2026-10'
  amount_cents bigint NOT NULL,
  UNIQUE (invoice_id, period)        -- a second invoice for the same period is rejected
);

CREATE TABLE job_runs (
  job_name text NOT NULL,
  period   text NOT NULL,
  PRIMARY KEY (job_name, period)     -- 'processed' marker
);

BEGIN;
-- Fence check. The row lock serializes concurrent holders. A stale holder
-- (lower token) updates 0 rows, and the app must ROLLBACK and stop.
UPDATE job_fence SET max_token = :token
 WHERE job_name = 'monthly_invoices' AND max_token <= :token;

-- The work. A duplicate run hits the unique constraint and inserts nothing.
INSERT INTO invoices (invoice_id, period, amount_cents)
VALUES ('INV-cust-4211', '2026-10', 129900)
ON CONFLICT (invoice_id, period) DO NOTHING;

-- The marker commits in the same transaction as the work: both persist or neither does.
INSERT INTO job_runs (job_name, period) VALUES ('monthly_invoices', '2026-10');
COMMIT;

A single PostgreSQL transaction that rejects stale lock holders with a fencing token, then uses a unique constraint on invoice_id plus period and a processed marker committed with the work so that a duplicate run has no effect.

Chunking long jobs, observability, testing, and the impact on cost and engineering time

Chunk the work instead of holding one long lock

A single lock held for an hour means a crash at minute 50 loses 50 minutes of progress and blocks every other instance until lockAtMostFor expires. Instead, have the scheduled trigger only enqueue work items, or have each run claim a bounded batch of rows with per-item claims. On PostgreSQL and MySQL 8+, SELECT ... FOR UPDATE SKIP LOCKED lets several instances pull disjoint batches without waiting on each other. A crash then costs one batch, and the load spreads across the fleet.

Emit the signals that reveal silent failure

For each job, log and count:

• lock acquired
• lock skipped
• run duration, compared against lockAtMostFor
• timestamp of the last successful run

Alert when a job misses its success window, not only when it errors. After teams add locks, the common failure is that nothing runs at all, for example because of a stale lock row or a misconfigured name. That failure produces no errors.

Test the coordination, not just the job

1. Start several instances against a shared database, trigger them at the same instant, and assert exactly one execution.
2. Kill the lock holder mid-run and verify that another instance picks up the work after expiry.
3. Simulate a rolling deploy where old and new versions overlap.

Cost

An external trigger or a lock table takes days to set up. Leader election with fencing takes longer. Either way, the work costs less than investigating one duplicate-charge incident and cleaning it up by hand.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cron jobs multiple instances, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.