Scheduled Jobs on Multiple Replicas: Leader Election and Locks

Why scheduled jobs run N times on multiple replicas

A scheduler embedded in an application process knows nothing about other processes. When a monolith runs as a single instance, "run this every night at 02:00" means one execution. Deploy three replicas behind a load balancer and each one boots its own scheduler, reads the same cron expression, and fires on the same tick. Nothing coordinates them. No shared state records that the job already ran, so every tick executes once per replica.

The symptoms depend on what the job does:

• Duplicate side effects. A reminder-email job sends each customer three copies. A billing run charges cards twice, or creates duplicate invoices that someone has to void by hand.
• Contended batch updates. Several replicas select the same "pending" rows and update them concurrently. You get lock waits, deadlocks, or lost updates, and the job takes longer than it did on one node.
• Silent waste. Cache warmers or report generators do redundant work. Nothing visibly breaks, but database and CPU load scale with replica count instead of with the work.

This usually shows up when a team scales a monolith horizontally for request throughput. Scheduled work was never designed for that. It often surfaces in production after an autoscaler adds instances.

The obvious move is to add a locking library. That comes second. First decide what each job needs. Some jobs must run exactly once, some tolerate overlap, and some should run on every node. That classification determines whether you need a lock, a leader, or nothing at all.

Copy                     +--------------------+
                     |   Load Balancer    |
                     +--------------------+
                               |
          +--------------------+--------------------+
          |                    |                    |
          v                    v                    v
+------------------+ +------------------+ +------------------+
| Replica A        | | Replica B        | | Replica C        |
| +--------------+ | | +--------------+ | | +--------------+ |
| | Scheduler    | | | | Scheduler    | | | | Scheduler    | |
| | fires 02:00  | | | | fires 02:00  | | | | fires 02:00  | |
| +--------------+ | | +--------------+ | | +--------------+ |
|         |        | |         |        | |         |        |
|         v        | |         v        | |         v        |
| +--------------+ | | +--------------+ | | +--------------+ |
| | Job: billing | | | | Job: billing | | | | Job: billing | |
| +--------------+ | | +--------------+ | | +--------------+ |
+---------+--------+ +---------+--------+ +---------+--------+
          |                    |                    |
          v                    v                    v
+------------------------------------------------------------+
| Same DB rows, same payment API, same email provider        |
| Result: 3 charges, 3 emails, contended UPDATEs per tick    |
+------------------------------------------------------------+

Each replica runs its own in-process scheduler with no shared state, so one 02:00 tick triggers the job three times against the same database and external services.

Who should use these patterns

These patterns are for teams that run scheduled work inside an application process and run more than one instance of that process. Typical cases:

• Monoliths scaled horizontally. The app uses an in-process scheduler (Spring's @Scheduled, a cron library, a timer loop), and you now run two or more replicas for availability or throughput.
• Services under an autoscaler. The replica count changes at runtime, so "we only run one instance" is not a guarantee you control.
• Rolling deployments. Even a single-replica service briefly runs old and new pods side by side during a rollout, and a tick that lands in that window can fire twice.
• Teams that want to keep jobs close to the code. Jobs that share the app's domain model, configuration, and database access are often easier to keep in-process than to extract into a separate system.

You probably do not need them if you run exactly one instance with Recreate-style deployments and accept downtime for jobs during restarts, if every job falls in the "safe on every replica" bucket below, or if scheduled work already lives in a dedicated scheduler or queue that owns execution.

Classify each job before choosing a mechanism

Sort every scheduled job into one of three buckets before writing any coordination code. The bucket tells you how much machinery you need, and often the answer is none.

Safe on every replica

Some jobs act only on state that belongs to the node running them. Examples are refreshing an in-process cache, rotating local temp files, or reloading feature flags into memory. Running them on every replica is correct, because each replica has its own copy of that state. A lock here would cause a bug: only one node would refresh its cache, and the others would serve stale data.

Once per tick

Report generation, invoice runs, and reminder emails act on shared state or external systems. If they run twice, customers can be billed twice or receive duplicate emails. These jobs need at-most-once execution per scheduled tick, or exactly-once in effect when you combine a lock with idempotent writes. A per-job lock fits this case. Each tick is a short, bounded unit of work, and different jobs can run on different nodes.

Single long-lived owner

Queue pollers, stream consumers, and sweepers that loop continuously have no clean tick boundary. They need one owner that holds the role until it fails or hands the role over. Leader election fits this case better than a lock that is taken and released on every iteration.

Avoid coordination entirely when you can make the job naturally idempotent. One way is to have it claim rows with a conditional update. Another is to let a queue's own delivery semantics assign the work.

How it works, end to end

Both mechanisms in this article follow the same basic flow. Only the scope of what is being locked differs.

1. Every replica keeps its scheduler. You do not remove the cron trigger from any node. Each replica still wakes up on the tick.
2. Each replica asks a shared store for permission. The store is something all replicas can reach and that supports an atomic conditional write: a database table, a Redis key, or a Kubernetes Lease object.
3. The store grants permission to exactly one caller. A conditional write ("set this only if it is absent or expired") succeeds for one replica and fails for the rest. The store, not the replicas, decides the winner.
4. The permission has an expiry. The grant records an owner and an expiry time. If the owner crashes, the expiry lets another replica take over without manual cleanup.
5. The winner runs; the losers skip. With a per-job lock, the winner runs that one job for that one tick. With leader election, the winner runs every job until it loses the lease.
6. The job's writes are idempotent. Expiry means two holders can overlap in edge cases, so the data layer must tolerate a duplicate run. The lock limits how often that happens; idempotency makes it harmless when it does.

The next sections cover each step in detail: the lock table and its atomic acquisition, leader leases, how to size expiries, and how to make writes safe to repeat.

How a distributed lock for cron works, with ShedLock as an example

When coordination is unavoidable, the simplest lock is a row in a table you already have. A minimal schema has four columns: name (primary key, one row per job), lock_until (when the lock expires), locked_at, and locked_by (hostname or pod name, for debugging).

Acquisition must be a single atomic statement. First, try INSERT with lock_until = now() + interval. If the primary key already exists, run UPDATE ... SET lock_until = now() + interval, locked_at = now(), locked_by = :me WHERE name = :job AND lock_until <= now(). One affected row means you hold the lock. Zero rows means someone else does. Use the database's now() rather than each replica's clock so every replica compares against one time source. The Redis equivalent is SET job:name <owner> NX PX <ms>: the write succeeds only if the key is absent, and the expiry releases it if the holder dies.

ShedLock wraps this pattern around an existing scheduler such as Spring's @Scheduled. It is neither a scheduler nor a queue. Each tick, every replica tries the lock. The replica that wins runs the job, and the others skip that execution entirely instead of waiting for it. lockAtMostFor caps how long a crashed holder can block the job. Set it well above the job's worst-case runtime. lockAtLeastFor keeps the lock held for a minimum time even after a fast job finishes. Without it, a replica whose clock runs slightly behind could fire the same tick after the lock was released.

Skipping has a cost. If the lock is still held when a tick fires, or every replica is down, that tick is lost. Jobs that must cover every interval need explicit catch-up. For example, persist a high-water mark and process everything since that mark, rather than assuming "since the last tick."

Copy-- PostgreSQL. One row per job, same shape as ShedLock's default table.
CREATE TABLE shedlock (
  name       VARCHAR(64)  PRIMARY KEY,  -- job name
  lock_until TIMESTAMP    NOT NULL,     -- lock expires at this time
  locked_at  TIMESTAMP    NOT NULL,     -- when the current holder acquired it
  locked_by  VARCHAR(255) NOT NULL      -- pod or hostname, for debugging
);

-- Step 1: first-ever run for this job. Creates the row and takes the lock.
-- ON CONFLICT DO NOTHING returns 0 rows instead of raising an error if the row exists.
INSERT INTO shedlock (name, lock_until, locked_at, locked_by)
VALUES ('nightly-report', now() + interval '10 minutes', now(), 'worker-7f9c')
ON CONFLICT (name) DO NOTHING;

-- Step 2: if step 1 affected 0 rows, try to take an expired lock.
-- The WHERE clause and the write are one atomic statement, so only one replica can win.
-- now() is the database clock, so every replica compares against the same time source.
UPDATE shedlock
SET lock_until = now() + interval '10 minutes',  -- lockAtMostFor: crash-safety ceiling
    locked_at  = now(),
    locked_by  = 'worker-7f9c'
WHERE name = 'nightly-report'
  AND lock_until <= now();
-- 1 row affected: this replica runs the job.
-- 0 rows affected: another replica holds the lock; skip this tick and do not retry.

-- Release after the job finishes. Keep the lock until at least locked_at + lockAtLeastFor
-- so a replica with a slightly late clock cannot fire the same tick again.
UPDATE shedlock
SET lock_until = GREATEST(now(), locked_at + interval '30 seconds')  -- lockAtLeastFor
WHERE name = 'nightly-report'
  AND locked_by = 'worker-7f9c';

-- Redis equivalent of steps 1-2 (600000 ms = 10 minutes):
--   SET job:nightly-report worker-7f9c NX PX 600000
-- Returns OK if acquired and nil if already held. The expiry releases the key if the holder dies.

A minimal ShedLock-style lock table in PostgreSQL. It acquires the lock atomically with a conditional INSERT followed by a conditional UPDATE against the database clock, releases it with a lockAtLeastFor floor, and shows the equivalent Redis SET NX PX command.

Leader election as the alternative to per-job locks

Instead of one lock per job, the replicas elect a single leader that holds a renewable lease, and only the leader starts the scheduler. Each job runs unguarded inside it. This is simpler because there is one coordination point, one renewal loop, and one question to answer when debugging: "who is leader?"

The advisory lock pitfall is the most dangerous because it fails silently. In transaction-mode pooling, the "session" holding the lock is not your session. An idle-timeout on a proxy can also close the backend connection. In both cases the lock disappears with no error, and a second replica becomes leader.

Other costs:

• Single overloaded leader. All jobs share one process's CPU, memory, and connection pool.
• Failover delay. When a leader crashes, failover takes roughly one lease duration.
• Churn. Every leadership change aborts in-flight jobs on the old leader, and the new leader restarts them.

One leader fits a modest set of short, cheap jobs. Per-job locks fit better when jobs are long or heavy, or when a single job's behaviour should not affect the others.

Copy-- Lease record in a shared store (PostgreSQL), one row per leadership group.
CREATE TABLE IF NOT EXISTS leader_lease (
  name       text PRIMARY KEY,
  holder     text NOT NULL,
  expires_at timestamptz NOT NULL
);

-- Acquire or renew. Each replica runs this every ~5s with its own id.
-- The database clock (now()) decides expiry, so replica clock skew does not.
-- The row is written only if there is no lease, it has expired,
-- or this replica already holds it.
INSERT INTO leader_lease (name, holder, expires_at)
VALUES ('scheduler', 'replica-a', now() + interval '15 seconds')
ON CONFLICT (name) DO UPDATE
  SET holder     = EXCLUDED.holder,
      expires_at = EXCLUDED.expires_at
  WHERE leader_lease.expires_at < now()
     OR leader_lease.holder = EXCLUDED.holder
RETURNING holder;
-- One row returned: this replica is leader, so start or keep running the scheduler.
-- Zero rows returned: another replica holds a live lease, so do not run jobs.
-- If a renewal fails or cannot complete before expires_at, stop the scheduler
-- immediately. Another replica may take over once the 15s lease lapses.
-- A crashed leader is replaced only after expiry, so failover takes up to ~15s.

-- Graceful shutdown: release early so failover does not wait for expiry.
DELETE FROM leader_lease
WHERE name = 'scheduler' AND holder = 'replica-a';

A lease record in PostgreSQL with an owner and expiry that replicas acquire and renew with a single conditional upsert, so only one replica runs the scheduler at a time.

Sizing lease duration and renewal intervals

The lock's lifetime has to outlast the slowest run you will actually see, not the typical one. A job that usually finishes in seconds can stall during a long garbage-collection pause, a slow downstream call, or a retry loop. If the lock expires during that stall, another replica acquires it, and two copies run at once.

Illustrative estimate. These inputs are assumed. An hourly job has a p99 runtime of 4 minutes and a worst observed run of 6 minutes. Add a 30-second pause budget and 1 minute for two 30-second downstream timeouts. That gives 7.5 minutes worst case. Doubling it for margin gives a lockAtMostFor of 15 minutes.

Set lockAtLeastFor to about 1 minute. It holds the lock briefly after a fast run, so a replica whose trigger fires a few seconds late because of clock skew does not run the job a second time.

For leader election, use a short lease that the leader renews. A 60-second lease renewed every 20 seconds survives two missed renewals before it lapses.

The two failure modes pull in opposite directions. A large TTL protects against overlap, but a hung job blocks every later run until the lock expires: up to 15 minutes in this example. A small TTL recovers quickly but risks two runs overlapping.

Clock skew affects both settings. When expiry is computed from each node's own clock, a node whose clock runs fast treats the lock as expired early. Some lock providers can read timestamps from the store's clock instead, so every replica measures time against one source.

If a job's runtime ranges from seconds to hours, no fixed TTL is safe. Renew the lease while the job runs, or split the work into smaller bounded batches.

Idempotency and fencing tokens do the real safety work

A lock reduces how often a job runs twice. It cannot guarantee a job never runs twice. A holder can stall in a long GC pause, lose its network, or outlive its lease while still writing. Design every job so that a duplicate execution is harmless, and treat the lock as a way to avoid wasted work.

Make duplicates harmless

• Natural keys. Key each output by what it represents, such as invoice:{customer}:{month}, not by a generated ID. A rerun then targets the same row.
• Processed-markers. Record each handled item, for example a sent_at column on a notification. Skip items that already have a marker.
• Upserts. Use insert-or-update on the natural key so that a second write converges to the same state.
• Per-period run records. Insert a row such as (job_name, period_start) under a unique constraint before doing the work. The database rejects the second insert, so only one execution claims each period, even if the lock failed.

Fencing tokens

A fencing token is a monotonically increasing number issued each time the lock is granted. The holder sends it with every write. The resource being written stores the highest token it has seen and rejects any write that carries a lower one. If a paused former holder wakes up and writes, its stale token is refused, so it cannot overwrite the newer holder's work. Most simple lock tables, including the basic pattern above, do not issue tokens and do not provide fencing.

When the lock store fails

If the lock store is unreachable, replicas cannot acquire the lock. Coordinated jobs are skipped until the store recovers, unless you configure them to fail open and run on every replica. On a single Redis primary with asynchronous replication, a lock acquired just before failover can be missing on the promoted replica. A second replica can then acquire the same lock while the first still holds it. Idempotency is what limits the damage in both cases.

Copy-- PostgreSQL. Each statement stays safe if a second replica runs it.

-- 1. Per-period run record: the primary key allows one claim per period.
CREATE TABLE job_runs (
  job_name     text        NOT NULL,
  period_start date        NOT NULL,
  claimed_at   timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (job_name, period_start)
);

-- Zero rows returned means another execution already owns this period: exit.
INSERT INTO job_runs (job_name, period_start)
VALUES ('monthly_invoices', '2026-10-01')
ON CONFLICT DO NOTHING
RETURNING job_name;

-- 2. Natural key + upsert: a rerun converges on the same row, never a duplicate.
CREATE TABLE invoices (
  customer_id  bigint NOT NULL,
  month        date   NOT NULL,
  amount_cents bigint NOT NULL,
  PRIMARY KEY (customer_id, month)
);

INSERT INTO invoices (customer_id, month, amount_cents)
VALUES (4821, '2026-10-01', 12900)
ON CONFLICT (customer_id, month)
DO UPDATE SET amount_cents = EXCLUDED.amount_cents;

-- 3. Fencing: the written resource keeps the highest token seen and
--    ignores writes carrying an older one (e.g. from a paused former holder).
CREATE TABLE report_snapshot (
  report_id   text   PRIMARY KEY,
  body        jsonb  NOT NULL,
  fence_token bigint NOT NULL
);

-- 1042 is the token issued with the current lock grant.
INSERT INTO report_snapshot (report_id, body, fence_token)
VALUES ('daily-usage', '{"total": 0}', 1042)
ON CONFLICT (report_id) DO UPDATE
  SET body = EXCLUDED.body, fence_token = EXCLUDED.fence_token
  WHERE report_snapshot.fence_token <= EXCLUDED.fence_token;

PostgreSQL statements that make duplicate job runs harmless: a per-period run record under a primary key, an upsert keyed on a natural key, and a fencing-token check that rejects writes from a stale lock holder.

Observability, moving jobs out of the monolith, and the cost trade-offs

Each run should write one record with the replica or pod that held the lock, the start time, the end time, and the outcome. That record lets you answer "did last night's billing job run, and where?" without searching logs from every replica. Alert when an expected run is missing within its window, not only when a run fails. A job that silently stops acquiring its lock produces no errors, and an error-only alert never fires. Log lock skips at debug level. On N replicas, N−1 skips per tick are normal, and logging them at a higher level hides the signals you need.

Moving a job out of the monolith changes which failures you have to handle:

None of these mechanisms gives exactly-once execution, so the idempotency work from the previous section is still required.

On engineering cost, classifying jobs and making them idempotent costs a few hours per job. Cleaning up double-charged invoices or duplicate emails takes much longer. Use the simplest mechanism that fits each job category so you only operate the infrastructure you need.

How this saves engineering time

The time savings come from avoiding work, not from making work faster.

• Less incident cleanup. Duplicate charges, duplicate emails, and conflicting batch updates each turn into manual work: finding affected records, voiding or reversing them, and answering support tickets. A lock plus idempotent writes prevents most of that work from existing.
• Faster debugging. A lock table with locked_by, or a lease row with holder, plus one run record per execution, answers "which replica ran this and when?" with a single query instead of a search across every replica's logs.
• No new infrastructure for most jobs. A lock table in the database you already run, or a Lease object in the cluster you already operate, avoids standing up and maintaining a separate scheduler service, its deployment pipeline, and its configuration.
• Classification removes unnecessary work. Jobs in the "every replica" bucket need no code changes at all. Knowing that up front stops teams from adding locks that later cause stale-cache bugs.
• Safe scaling decisions. Once scheduled jobs are coordinated and idempotent, changing replica counts or tuning an autoscaler no longer requires checking which jobs will break.

Illustrative estimate. All inputs are assumed. Suppose a team has 10 scheduled jobs, of which 4 need a lock and idempotent writes, at an assumed 4 hours each, for 16 hours of one-time work. Suppose that without coordination, an assumed one duplicate-execution incident per quarter reaches customers, and each takes an assumed 2 engineers 1 day to investigate and clean up, or about 16 engineer-hours. Under those assumptions, the one-time work pays for itself after the first avoided incident, before counting support load or customer impact. Replace each input with your own job count, effort, and incident history before relying on the result.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to scheduled jobs multiple replicas, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.