Why does a cron job run multiple times when you scale to multiple replicas?
If you run a monolith with an embedded scheduler, such as a cron-style library started at application boot, this applies to you. That scheduler keeps its timetable in process memory. It reads the cron expression, waits for the next tick, and fires. Nothing in that loop asks whether another copy of the application is about to do the same thing. With one instance, that works. With N replicas, you get N schedulers and N executions of every job.
The symptoms depend on what the job does:
- Notifications: a nightly digest sends three identical emails.
- Billing: a billing run charges a customer twice.
- Data jobs: a reconciliation job has two processes reading the same rows and writing conflicting results, with the last writer winning.
This often goes unnoticed for a long time. Many teams run a single replica until traffic forces a scale-out. Even teams that already run two replicas may have jobs that rarely produce visible damage. Rolling deploys expose the problem without any scaling change. During a rollout, old and new pods run side by side, and each believes it is allowed to fire the 02:00 job. A job that has run correctly for months duplicates only on deploy nights, which makes the cause hard to trace.
The rest of this article rests on one claim. No coordination mechanism gives you exactly-once execution. Locks, leases, leader election and external schedulers can give you at most one runner at a time under normal conditions. Clock skew, network partitions, paused processes and expired leases break those conditions. The job itself therefore has to be idempotent, so that a second execution is harmless.
How do the four approaches to distributed cron work?
1. External scheduler
A Kubernetes CronJob, or system cron on one designated host, owns the schedule and calls a CLI entrypoint or an internal HTTP endpoint, so the replicas stop scheduling. concurrencyPolicy sets what happens when the previous run is still active. Allow starts another run, Forbid skips the new one, and Replace kills the old run and starts the new one. startingDeadlineSeconds limits how late a missed run may still start. The controller only promises about one run per schedule, so it can occasionally start a run twice or skip it.
2. Database lock
The simplest version is a lock table with a unique key on (job_name, run_window). The replica whose insert succeeds runs the job, and the others hit the constraint and skip. For work split into items, SELECT ... FOR UPDATE SKIP LOCKED lets each replica claim different rows like a queue.
PostgreSQL advisory locks need no table, but scope matters:
pg_try_advisory_lockis held by the session until it is explicitly unlocked or the connection closes.pg_try_advisory_xact_lockis released when the transaction ends.
A transaction-mode connection pooler can give each statement a different backend connection. A session lock taken through it can stay attached to a connection that someone else later uses, or the unlock can run on a different connection. Behind such a pooler, use transaction-scoped locks.
3. Key-value lock
Each replica sets the key only if it does not already exist, with a TTL and a unique owner token as the value. To release, the replica deletes the key only if the value still matches its token, and that check-and-delete must be one atomic operation. Without the token check, a replica whose lock already expired could delete the lock now held by another replica.
4. Leader election
Replicas compete for a Kubernetes Lease through a client library, or use etcd or ZooKeeper. There are two ways to use it:
- Lease per job spreads jobs across replicas.
- Leader runs everything is simpler, but puts every job on one process.
replica A ──┐
replica B ──┼──► coordination point ──► winner ──► side-effect target
replica C ──┘ (scheduler / DB / (DB, email, API)
KV / Lease)
-- Option A: lock table. One row per job per run window.
CREATE TABLE IF NOT EXISTS job_runs (
job_name text NOT NULL,
run_window timestamptz NOT NULL,
claimed_by text NOT NULL,
claimed_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (job_name, run_window) -- the unique key makes the claim exclusive
);
-- Every replica runs this at the scheduled time. Only one insert succeeds.
-- A returned row means this replica owns the run. No row means skip.
INSERT INTO job_runs (job_name, run_window, claimed_by)
VALUES ('nightly-report', date_trunc('day', now()), 'replica-7f9c')
ON CONFLICT (job_name, run_window) DO NOTHING
RETURNING job_name;
-- Option B: transaction-scoped advisory lock. This is safe behind a
-- transaction-mode pooler because the lock is released at COMMIT/ROLLBACK
-- on the same backend connection that took it.
BEGIN;
-- 42001 is an app-chosen bigint that identifies the 'nightly-report' job.
SELECT pg_try_advisory_xact_lock(42001) AS got_lock;
-- If got_lock is false, another replica is running the job: ROLLBACK and skip.
-- If it is true, do the job's work inside this same transaction.
COMMIT; -- the lock is released here and no explicit unlock is needed
-- Option C: queue-style claiming. Each replica takes different pending rows.
BEGIN;
SELECT id FROM report_items
WHERE status = 'pending'
ORDER BY id
LIMIT 10
FOR UPDATE SKIP LOCKED; -- rows locked by other replicas are skipped, not waited on
-- Process the returned ids, then mark them done before COMMIT.
COMMIT;
PostgreSQL examples of three database-coordination patterns: a lock table with a unique (job_name, run_window) key, a transaction-scoped advisory lock that is safe behind a transaction-mode pooler, and SKIP LOCKED row claiming.
How do you implement a run key, lock, and idempotent job?
Derive the run key from the schedule
Build the key from the job name and the scheduled fire time, for example invoice-batch:2026-11-01T00:00Z. Do not use now(). If replica A's clock reads 00:00:00.4 and replica B's reads 23:59:59.9, keys built from wall-clock time differ and both replicas win. Rounding to the scheduled slot means every replica competes for the same row or key.
Three ways to claim the slot
- Lock table. Create a table with a unique constraint on
run_keyand runINSERT ... ON CONFLICT DO NOTHING. The replica whose insert reports one affected row owns the run. The row also stays behind as a record of who ran which slot. - PostgreSQL advisory lock. Call
pg_try_advisory_xact_lock(hashtext(run_key))inside a transaction. The lock is released automatically on commit, rollback, or a dropped connection, so a crashed holder never strands it. The lock only holds while that transaction is open. - Key-value lock. Set the key only if it is absent, with an expiry and a random owner token as the value. To release, delete the key only if it still holds your token, and do that compare-and-delete atomically. Otherwise you can delete a lock that expired and was taken by another replica.
Make the write side idempotent
Locks reduce duplicate runs but do not guarantee only one. Enforce correctness at the write. Put a unique constraint on (customer_id, billing_period) so a second invoice insert fails. For updates, pass a fencing token, such as a monotonically increasing run ID, and write only WHERE last_run_id < :run_id. A stale holder's writes then match zero rows.
-- 1. Lock table: one row per scheduled slot, keyed on job name + fire time.
CREATE TABLE job_runs (
run_key text PRIMARY KEY, -- e.g. 'invoice-batch:2026-11-01T00:00Z'
run_id bigserial UNIQUE, -- monotonically increasing fencing token
claimed_by text NOT NULL,
claimed_at timestamptz NOT NULL DEFAULT now()
);
-- The replica that gets a row back owns the run; others get zero rows.
INSERT INTO job_runs (run_key, claimed_by)
VALUES ('invoice-batch:2026-11-01T00:00Z', 'replica-a')
ON CONFLICT (run_key) DO NOTHING
RETURNING run_id;
-- 2. Advisory lock alternative: held only while this transaction is open.
BEGIN;
SELECT pg_try_advisory_xact_lock(hashtext('invoice-batch:2026-11-01T00:00Z'));
-- if false: ROLLBACK and skip; if true: do the work here, then COMMIT
COMMIT;
-- 3. Idempotent write: a natural key makes a duplicate invoice a no-op.
CREATE TABLE invoices (
customer_id bigint NOT NULL,
billing_period date NOT NULL,
amount_cents bigint NOT NULL,
UNIQUE (customer_id, billing_period)
);
INSERT INTO invoices (customer_id, billing_period, amount_cents)
VALUES (42, DATE '2026-11-01', 1999)
ON CONFLICT (customer_id, billing_period) DO NOTHING;
-- 4. Fencing on updates: a stale holder with an older run_id matches zero rows.
UPDATE accounts
SET balance_cents = balance_cents - 1999,
last_run_id = 1017 -- run_id returned by the claim above
WHERE customer_id = 42
AND last_run_id < 1017;
PostgreSQL sketch of claiming a scheduled slot via a unique run_key insert or a transaction-scoped advisory lock, then enforcing exactly-once effects with a natural-key unique constraint and a run_id fencing check on updates.
What happens when a lock or lease is held by two processes at once?
Leases fail in a specific way: the holder does not know it has lost the lease. The TTL has to be longer than the worst-case job duration, or the holder has to renew it while the job runs. Even with renewal, a process that is stopped by a stop-the-world GC pause, CPU throttling, or a VM migration cannot renew. Its lease expires while it is paused, another replica acquires it, and the paused process later resumes and keeps writing because its last check passed.
The store itself can also lose a lock. With a single primary and asynchronous replication, a lock written just before a failover may never reach the new primary, so a second replica can acquire a lock the first one still believes it holds.
Leader election has the same problem in another form. There is a gap between a replica losing leadership and that replica actually stopping work. The lease duration and renew deadline settings trade fast failover against false failovers caused by brief network or scheduling hiccups.
Illustrative estimate (assumed inputs, not measurements): job p50 2 min, p99 8 min, one 120 s pause.
| Configuration | Assumed failure | Window with two holders |
|---|---|---|
| Fixed TTL 5 min, no renewal | p99 run of 8 min | ~3 min |
| TTL 90 s, renew every 30 s | 120 s pause right after a renewal | ~30 s, plus writes issued before the next lease check |
| TTL 15 s, renew every 5 s | Same 120 s pause | ~105 s, and more false takeovers on short stalls |
A shorter TTL speeds up failover, but under a long pause it widens the overlap. No TTL setting closes this window, which is why the fencing token check in the database is required, not optional.
How should long-running jobs, crashes, and missed runs be handled?
Three failure cases matter here: a run that outlasts its interval, a replica that dies mid-run, and ticks that fall inside downtime or a deploy window.
| Approach | Run exceeds interval | Replica crashes mid-run | Runs missed during downtime |
|---|---|---|---|
| Single scheduler instance | Overlaps unless the scheduler skips busy jobs | Run is lost until the instance restarts | Lost unless last-run time is persisted |
| Per-run lock with run key | New run key, so overlap unless you also lock per job | Lock expires; the run key row stays "started" and needs a reclaim rule | Gaps are visible as missing run keys |
| Leader election lease | Leader keeps running; next tick waits or overlaps | New leader after lease TTL; in-flight work is repeated or dropped | New leader decides from stored state |
| External scheduler | Governed by its overlap policy | Retries may re-run partially finished work | Depends on its catch-up setting |
Skipping missed runs keeps load predictable because the system never does more than one run per tick, but it loses work without any signal. A skipped daily invoice run produces no error. Catching up recovers that work, but after a long outage every queued run fires at once, against a database that may still be warming up.
Make the decision per job and store it alongside the job definition: skip, catch up once, or catch up all. Cache refreshes can usually skip. Reports often need one catch-up run. Billing usually needs every period processed.
Rolling deploys need extra care. While old and new code run side by side, a duplicate run can execute against a schema or business rule that the other version doesn't expect. In that case a duplicate does more than repeat work: it can write data in the wrong shape.
Which option should you choose, and when should you avoid each?
Each option removes duplicates by moving the "who runs this" decision somewhere different. Choose based on where that decision can fail.
| Approach | Extra infrastructure | Failure under partition or pause | Visibility | Migration effort from embedded scheduler | Avoid when |
|---|---|---|---|---|---|
| Platform cron | None beyond the orchestrator | A run is skipped or delayed. Duplicates are unlikely unless retries overlap | Strong: the platform records each run | High: schedules move out of code | Job definitions must live in the codebase, or cold-start cost per run is significant |
| Database lock | None if you already have a database | The lock is released when the session dies. A paused holder can resume after losing it | Weak unless you log acquisitions | Low: wrap the existing job | The database is already the bottleneck, or it sits behind a transaction-mode pooler that breaks session-scoped locks |
| Key-value lock | A key-value store | A lease can expire while the holder is paused, which lets two holders write | Moderate: keys and TTLs can be inspected | Low | You cannot add fencing tokens that the write side checks |
| Leader election | A coordination store and library | A changeover can leave a gap with no leader, or an overlap with two | Moderate | Medium | One replica cannot absorb all scheduled work, or changeover pauses are unacceptable |
The lock wrappers usually save the most engineering time, because the job code barely changes. Platform cron saves operational time later, through its run history and retries.
Not coordinating in web replicas at all
A dedicated worker process removes the problem by running only one scheduler. A queue-based system does the same thing differently: a single scheduler enqueues jobs and competing consumers pick them up. Either way you add a deployable unit to build, monitor, and scale. In exchange, scheduled load stops affecting request latency. Pick the lock wrapper when jobs are few and light. Pick the separate unit when scheduled work has its own scaling needs.
How do you monitor scheduled tasks across replicas?
Monitoring has to answer three questions: did this run happen, did it happen only once, and if it did not happen, why not. Logs from individual replicas can't answer the first two, because a missing run leaves no log line and a duplicate is spread across two pods. Keep a shared record instead.
A run-history table
Write one row per attempt to the same store you use for locks:
| Column | Purpose |
|---|---|
job_name |
Which schedule this belongs to |
run_key |
Job name plus scheduled time, e.g. invoice-sweep:2024-06-01T02:00Z |
owner |
Pod name or hostname that claimed the run |
started_at, finished_at |
Duration and detection of stuck runs |
outcome |
running, succeeded, failed, skipped |
Do not put a unique constraint on run_key in this table. You want duplicates to land here so you can see them. Enforce uniqueness in the lock instead.
Detecting gaps and duplicates
To check a schedule window, generate the expected run keys from the cron expression and left-join them against the table. A key with zero rows is a missed run. A key with more than one row in succeeded or running is a double execution, which means your lock failed.
Alert on absence
Alert when an expected run key has no row within its grace period, not only when a run's outcome is failed. Most coordination bugs cause silence, not errors.
Expose the lock holder
Publish the current holder and lease expiry as a metric or status endpoint. If a run is late, on-call can see the difference between "pod-3 holds a lease that expired ten minutes ago" (a stuck or crashed owner) and "nobody holds it" (the scheduler never fired).
-- PostgreSQL. One row per attempt, written to the same database that holds the locks.
CREATE TABLE job_runs (
id bigserial PRIMARY KEY,
job_name text NOT NULL,
run_key text NOT NULL, -- e.g. 'invoice-sweep:2024-06-01T02:00Z'
owner text NOT NULL, -- pod name or hostname that claimed the run
started_at timestamptz NOT NULL DEFAULT now(),
finished_at timestamptz, -- NULL while running; old NULLs suggest stuck runs
outcome text NOT NULL
CHECK (outcome IN ('running', 'succeeded', 'failed', 'skipped'))
);
-- Deliberately NOT unique: duplicates must be recorded so they can be detected.
CREATE INDEX job_runs_key_idx ON job_runs (job_name, run_key);
-- Find missed and doubled runs for an hourly schedule ('0 * * * *') over one day.
-- For more complex cron expressions, generate the expected keys in application code.
WITH expected AS (
SELECT 'invoice-sweep:' || to_char(ts AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI"Z"') AS run_key,
ts AS scheduled_at
FROM generate_series(timestamptz '2024-06-01 00:00+00',
timestamptz '2024-06-01 23:00+00',
interval '1 hour') AS ts
)
SELECT e.run_key,
count(r.id) AS attempts,
string_agg(r.owner, ', ') AS owners,
CASE WHEN count(r.id) = 0 THEN 'missing' ELSE 'duplicate' END AS problem
FROM expected e
LEFT JOIN job_runs r
ON r.job_name = 'invoice-sweep'
AND r.run_key = e.run_key
WHERE e.scheduled_at < now() - interval '15 minutes' -- grace period before calling a run missing
GROUP BY e.run_key
HAVING count(r.id) = 0 -- never started
OR count(r.id) FILTER (WHERE r.outcome IN ('running', 'succeeded')) > 1 -- lock failed
ORDER BY e.run_key;
A PostgreSQL run-history table without a unique constraint on run_key, plus a query that generates the expected hourly run keys and reports any that are missing past a grace period or were executed more than once.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to cron job multiple replicas, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.