Cron Jobs on Multiple Instances: Run Scheduled Tasks Once in a Monolith

Why in-process cron fires on every replica

An in-process scheduler (Spring's @Scheduled, a cron library in Node, a background thread in a Rails or Django worker) lives inside the application process. It has no idea other copies of that process exist. When you run one replica, "every night at 02:00" means one run. When you scale to six replicas behind a load balancer, it means six runs starting within milliseconds of each other. Duplicate job execution then shows up as six invoice emails to the same customer, six concurrent updates contending for the same rows, or six calls to a rate-limited third-party API.

The usual first fix is to make one replica special. Teams either run a dedicated "jobs" replica or set an environment variable such as RUN_SCHEDULER=true on exactly one pod. Both approaches tie scheduling to the identity of a process, and orchestrators replace processes freely. During a rolling deploy, the old flagged pod and the new one can overlap, so the job runs twice. If the flagged pod crashes or its node is drained, no job runs until someone notices. Autoscaling clones the flagged template or scales the jobs replica to zero. Sometimes the result is zero runners and sometimes it is two, and neither failure raises an error.

This affects any team running a horizontally scaled monolith that also owns scheduled work. The rest of this article uses the database you already operate to coordinate those jobs, and only then considers whether a separate distributed scheduler is worth adding.

Copy         02:00 fires in every replica (each has its own scheduler)
     |               |               |               |
     v               v               v               v
+-----------+   +-----------+   +-----------+   +-----------+
| replica 1 |   | replica 2 |   | replica 3 |   | replica N |
| scheduler |   | scheduler |   | scheduler |   | scheduler |
+-----------+   +-----------+   +-----------+   +-----------+
     |               |               |               |
     +---------------+-------+-------+---------------+
                             |  N concurrent runs
                             v
    +-------------------------------------------------+
    | shared side effects: emails, DB rows, ext. API  |
    +-------------------------------------------------+

Common workaround: RUN_SCHEDULER=true on exactly one pod

 rolling deploy                        crash / node drained
+-----------+   +-----------+          +-----------+
| old pod   |   | new pod   |          | flagged   |
| flag=true |   | flag=true |          | pod gone  |
+-----------+   +-----------+          +-----------+
     |    overlap    |                      |
     +-------+-------+                      v
             v                          0 runners
      2 runners (duplicate)          (job silently skipped)

Every replica's in-process scheduler fires the same job against shared resources, and pinning the job to one flagged pod still yields two runners during a rolling deploy or none after a crash or node drain.

Who should use this approach

Database-coordinated scheduling fits teams in this situation:

• You run a monolith (or a small number of services) as several identical replicas. Examples include multiple pods in a Kubernetes Deployment, several VMs behind a load balancer, or an autoscaling group.
• Scheduled work already lives in the application. This includes nightly rollups, invoice runs, cleanup tasks, report emails, and syncs with external APIs, and it is written with the framework's scheduler.
• The jobs have side effects that must not repeat. Examples are emails, payments, calls to rate-limited APIs, and writes that would double-count.
• You already operate a relational database that every replica can reach, such as PostgreSQL or another database your lock library supports.
• The number of jobs is modest. There are tens of jobs rather than thousands, they have no dependencies between them, and a small number of engineers own them.

It is a poor fit in other cases:

• You run a single replica and never plan to scale out. In that case the duplicate problem does not exist yet.
• You need workflow features. These include DAGs of dependent jobs, backfills over historical ranges, per-run parameters, or a UI for operators. A dedicated scheduler covers these, as discussed later.
• You need very frequent or high-volume tasks, for example sub-second scheduling or fan-out of millions of independent work items. A queue with workers is the better tool for that.

How it works at a glance

The approach keeps the schedule where it is, inside every replica, and adds a coordination step against shared storage before any work starts:

1. The schedule fires in every replica. Each in-process scheduler triggers at the same time, as before.
2. Each replica tries to claim the job in the database. It does this with one atomic operation, either a conditional upsert on a lock row or a non-blocking advisory lock.
3. Exactly one replica wins. The database serialises the competing attempts, so only one sees success. The others get a "not acquired" result and return immediately without doing anything.
4. The winner runs the job under a lease. The lock carries an expiry, so a crashed holder does not block future runs forever. Optionally, the winner also receives a fencing token to pass to downstream writes.
5. The winner records the run and releases the lock. A run-history row captures start, finish, owner and outcome, which you need for monitoring.
6. Job logic tolerates the rare duplicate. Leases can be outlived, so the jobs themselves use upserts, checkpoints and dedupe keys to make a second run harmless.

The next sections cover each step in detail. Steps 2–4 are the locking mechanism, step 5 is observability, and step 6 is idempotent job design. After that, the article covers the alternative of moving the schedule out of the app entirely.

How lease-based locking works

The mechanism needs one table with a row per job. It is keyed by name, and it holds locked_until, a timestamp, and locked_by, which identifies the host or process. When the schedule fires, every replica tries to claim the row in a single atomic statement. That statement is either an INSERT that does nothing on conflict, or an UPDATE ... SET locked_until = now() + interval, locked_by = $me WHERE name = $job AND locked_until <= now(). Exactly one replica sees "1 row affected" and runs the job. The others see zero rows and skip.

The comparison uses the database's now(), not the application clock. This means every replica measures expiry against the same clock. Skew between replicas then doesn't matter for acquisition. Skew still matters for any client that computes locked_until from its own clock, so don't do that.

To release, the holder runs UPDATE ... SET locked_until = now() WHERE name = $job AND locked_by = $me. The locked_by guard stops a slow runner from releasing a lock that someone else has since acquired.

Sizing the lease

The lease must be longer than the job's worst-case runtime:

• Too short: a slow run outlives its lease and a second replica starts the same job concurrently.
• Too long: if the holder crashes, nobody can run the job until the lease expires. You lose runs for that window.

What the lock does not guarantee

A long GC pause, a frozen container, or a network partition can stall a process past locked_until. When it resumes, it keeps writing without knowing that another replica now holds the lock. The lock therefore gives you at most one concurrent runner, and only as long as runtimes stay within the lease. It does not give exactly-once execution. To protect against a stalled holder, you need one of two things:

• Fencing tokens: a counter that increases with each acquisition and is checked on every write.
• Idempotent job logic: work that produces the same result if it runs twice.

Copy-- PostgreSQL: one row per scheduled job.
CREATE TABLE job_locks (
  name         text        PRIMARY KEY,
  locked_until timestamptz NOT NULL,
  locked_by    text        NOT NULL,
  fence        bigint      NOT NULL DEFAULT 1  -- bumps on every acquisition
);

-- Acquire: every replica runs this when the schedule fires.
-- $1 = job name ('nightly-invoices'), $2 = holder id ('web-3:pid-4127').
-- The lease (15 minutes) must exceed the job's worst-case runtime.
-- Expiry is checked against the database's now(), never the app clock.
INSERT INTO job_locks (name, locked_until, locked_by)
VALUES ($1, now() + interval '15 minutes', $2)
ON CONFLICT (name) DO UPDATE
  SET locked_until = now() + interval '15 minutes',
      locked_by    = EXCLUDED.locked_by,
      fence        = job_locks.fence + 1
  WHERE job_locks.locked_until <= now()   -- only take an expired lease
RETURNING fence;
-- One row returned: you hold the lock. Pass the fence value to the job
-- so downstream writes can reject a stale holder.
-- Zero rows: another replica holds a live lease. Skip this run.

-- Release: only the current holder can end its own lease.
-- If the lease expired and someone else took it, this affects 0 rows.
UPDATE job_locks
   SET locked_until = now()
 WHERE name = $1
   AND locked_by = $2;

A PostgreSQL lease lock that acquires in one atomic upsert checked against the database's now(), returns a fencing token for downstream writes, and releases only when the caller is still the recorded holder.

Implementing the pattern with ShedLock and Postgres advisory locks

ShedLock on the JVM

ShedLock wraps a scheduled method with a check against a shared lock store. You annotate the method with @SchedulerLock, give it a unique name, and register a lock provider pointed at the database the application already uses. Typically that is a JDBC provider backed by a small shedlock table. Before running, each replica tries to insert or update the row for that name. Only the replica that succeeds executes the job.

Two durations control the behavior:

• lockAtMostFor is the safety lease. If the holder crashes, the lock expires after this interval. Set it well above the job's worst-case runtime. If you set it too short, a slow but healthy run gets overlapped.
• lockAtLeastFor keeps the lock held even after a fast job finishes. Without it, a job that completes in 200 ms frees the lock, and a replica whose clock runs slightly behind can fire the same schedule slot moments later and run it again.

Postgres advisory locks

If you don't want a lock table, pg_try_advisory_lock(key) returns immediately with true or false. The lock is held by the database session until you unlock it or the session ends.

Connection pools make session-scoped locks risky:

• Application pools: a connection returned to the pool without an unlock keeps holding the lock indefinitely.
• Transaction-mode poolers: these map consecutive statements to different server connections. The unlock can land on a backend that never held the lock, so the original lock leaks. A recycled server connection can also drop the lock silently while the job is still running.

pg_try_advisory_xact_lock avoids these problems because the lock is released at commit or rollback. The trade-off is that the job's work must fit inside a single transaction. That transaction holds locks and delays vacuum for as long as it is open, so long batch jobs are a poor fit.

Copy-- Pick one stable bigint key per job and share it across all replicas.
-- 74210001 = 'nightly_invoice_rollup' (placeholder value)

-- Option A: session-scoped lock. It must be released on the SAME connection.
-- Do not run this through a transaction-mode pooler, because the unlock may
-- reach a different backend.
SELECT pg_try_advisory_lock(74210001) AS got_lock;
-- If got_lock is false, another replica owns this run. Skip it.
-- If true, do the work. It may span many transactions on this connection.
SELECT pg_advisory_unlock(74210001);
-- Always unlock (e.g. in a finally block) before returning the connection
-- to the pool. Otherwise the pooled connection keeps holding the lock.

-- Option B: transaction-scoped lock. It is released automatically at
-- COMMIT or ROLLBACK, so a pool cannot leak it.
BEGIN;
SELECT pg_try_advisory_xact_lock(74210001) AS got_lock;
-- If got_lock is false, ROLLBACK and skip this run.
-- If true, all of the job's work must happen inside this one transaction.
INSERT INTO invoice_rollups (day, total_cents)
SELECT created_at::date, sum(amount_cents)
FROM invoices
WHERE created_at >= current_date - 1 AND created_at < current_date
GROUP BY 1;
COMMIT;  -- the lock is released here

-- Debugging: list the advisory locks currently held and the backend holding each.
SELECT pid, objid, granted
FROM pg_locks
WHERE locktype = 'advisory';

Session-scoped versus transaction-scoped Postgres advisory locks for a single-run job: the session lock must be explicitly unlocked on the same connection, while the transaction lock is released at commit but confines the job to one transaction.

Moving the schedule out of the app: pg_cron and Kubernetes CronJobs

Both locking approaches keep the schedule inside the application and add coordination on top. The alternative is to have exactly one external scheduler fire each job, so the replicas no longer need to agree on anything.

pg_cron

pg_cron is a Postgres extension that stores job definitions in a table and runs them as SQL on a cron schedule. A single background worker inside the database fires each job, so a job runs once no matter how many application replicas exist. This works well for SQL-only maintenance such as purging expired sessions, refreshing materialized views, or rolling up partitions. It cannot call application code directly. Any job that needs business logic or an HTTP call must go back through the app. Jobs also run on the primary and compete with production traffic there. Installing the extension requires privileges that some managed Postgres offerings restrict or configure differently, so check what your provider allows before you design around it.

Kubernetes CronJobs

A CronJob starts a pod from the monolith's own image with a task entrypoint, so it can run real application code. Setting concurrencyPolicy: Forbid stops a new run from starting while the previous one is still active. Each run pays the full startup cost of the container plus app boot. Runs can also be skipped or doubled. If the controller is down or a run misses its startingDeadlineSeconds window, that run is not made up later. In rare cases a single scheduled run starts twice, so job logic still has to tolerate duplicates. The schedule also moves into deployment manifests and is no longer versioned with the code it triggers.

A dedicated distributed scheduler is worth running when you need dependencies between jobs, backfills, per-run history, or many teams sharing one schedule.

CopyapiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-invoice-rollup
spec:
  # Run at 02:30 every day (controller time zone unless spec.timeZone is set)
  schedule: "30 2 * * *"
  # Do not start a new run while the previous one is still active
  concurrencyPolicy: Forbid
  # If the run cannot start within 10 minutes of its scheduled time, skip it
  startingDeadlineSeconds: 600
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      # Avoid automatic retries; a retry is another chance to run twice
      backoffLimit: 0
      # Kill a run that hangs so the next scheduled run is not blocked by Forbid
      activeDeadlineSeconds: 3600
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: task
              # Same image as the web deployment, so the task uses real app code
              image: registry.example.com/shop-monolith:1.42.0
              # Task entrypoint instead of the web server; still must be idempotent
              command: ["bin/run-task", "invoices:rollup"]
              envFrom:
                - secretRef:
                    name: shop-monolith-env
              resources:
                requests:
                  cpu: 250m
                  memory: 512Mi

A Kubernetes CronJob that runs the monolith's image with a task entrypoint. It uses concurrencyPolicy: Forbid to block overlapping runs and a startingDeadlineSeconds window, after which a late run is skipped rather than made up.

Designing jobs to survive duplicates, retries and partial runs

Every approach above reduces the chance of overlapping runs. None eliminates it. A lease can expire while the holder is stalled in a long GC pause or a slow query. A Kubernetes CronJob can occasionally create two Jobs for one schedule. A deploy can kill a run halfway through. The job's data model has to make a second or interrupted run harmless.

Process in batches with a durable checkpoint. Work through bounded batches and commit a checkpoint, such as the last processed ID, in the same transaction as the batch's writes. A killed run resumes from the checkpoint instead of starting over. A concurrent run either skips committed work or blocks on the same rows.

Upsert instead of blind inserts. Key writes on a natural identifier, such as (account_id, billing_period) for an invoice. A rerun then rewrites the same row rather than creating a duplicate.

Give side effects a dedupe key. Before sending an email, insert a key like invoice-email:{invoice_id} into a table with a unique constraint, and mark it complete after delivery. For payments, pass an idempotency key to providers that accept one, so a retry is recognised rather than charged again.

Query for outstanding work, not elapsed time. "Everything not yet processed" catches up skipped or partial runs automatically. "Rows from the last hour" silently drops them.

With these in place, lease length becomes a performance setting rather than a correctness decision. A lease that overruns wastes some work but does not double-charge anyone.

Observability: knowing a job ran, ran once, and finished

A lock tells you that at most one instance should be running a job. It does not tell you that the job ran, ran once, or finished. For that, record every run in a job_runs table with these columns: job name, started_at, finished_at, the owning instance ID (hostname or pod name), and an outcome such as running, succeeded or failed. Insert the row when the job acquires its lease. Update it when the job exits. A row stuck in running long after the lease expired means the process died mid-run. The instance ID tells you which replica's logs to read.

Two queries cover most failures:

• Gaps: For each job, use the LAG() window function over started_at to compare each start with the previous one. Flag any interval longer than the schedule period plus a tolerance.
• Overlaps: Self-join job_runs on job name. Two runs overlap when each one started before the other finished. Treat a null finished_at as now(). A hit means the lease expired while the job was still working, or the locking was bypassed.

Alert on both overlaps and gaps. Teams tend to build the overlap alert and skip the gap alert, but missed runs cause more trouble in practice. Every database-backed approach makes the database a dependency of the scheduler. During a database outage, ShedLock cannot acquire a lock, pg_cron does not fire, and no job_runs row gets written. Nothing errors and nothing appears in the table, so jobs silently do not run.

The missed-run check therefore has to run outside the scheduler it watches. It must also treat "no recent successful row" or "cannot query" as a failure.

How this saves engineering time

The time savings come from removing categories of work, not from making any single task faster.

• No new system to build, run or learn. A lease table or advisory lock reuses the database the team already backs up, monitors and is on call for. There is no scheduler cluster to provision, upgrade, secure or put on a pager rotation, and no second deployment pipeline for job definitions.
• Fewer duplicate-run incidents to clean up. Without coordination, every scale-out turns each job into N runs. Cleaning up after that means apologising to customers for repeated emails, reversing double charges, and deduplicating rows. Single-runner locking combined with idempotent job logic removes most of that cleanup work.
• No manual failover of a "special" replica. With a flagged pod or a dedicated jobs replica, someone has to notice when it dies, re-flag another instance, and check for overlap after deploys. A lease moves to whichever replica fires next, so crashes, drains, rolling deploys and autoscaling need no human action.
• Schedules stay in the codebase. With ShedLock or advisory locks, a job, its schedule and its lock name change in the same commit and go through the same code review and tests. Nobody has to keep a separate manifest or console in sync with the code.
• Faster diagnosis. The job_runs table and the gap and overlap queries answer "did it run, once, and finish?" with a single query. Without them, an engineer has to search logs across every replica. The instance ID points straight to the right logs.
• Recovery becomes "run it again". Checkpoints, upserts and dedupe keys make a rerun safe. After a failed or partial run, an engineer can trigger the job again instead of reconstructing which rows and side effects already happened.
• Lease tuning stops being a high-stakes decision. Once duplicates are harmless, getting a lease length slightly wrong costs some wasted compute rather than an incident review.

The trade-off is that you own a small amount of glue code: the lock table, the run-history table, and the external missed-run check. For a monolith with a modest number of independent jobs, that is usually less work than adopting a dedicated scheduler. The illustrative estimate at the end of the next section shows how to compare the two with your own inputs.

Choosing an approach: decision guide and impact on engineering cost

Pick the lightest option that fits your longest job and its failure mode.

The DB-backed options reuse a database you already back up, monitor and page on. Schedules for ShedLock and advisory locks live in application code, so they go through code review. pg_cron schedules live in a database table, so manage them through migrations if you want the same review.

When to avoid each:

• ShedLock: avoid it when runtimes are unpredictable. A job that outlives its lease can run twice.
• Advisory locks: avoid them behind a transaction-mode connection pooler, because session-level locks need a stable session.
• pg_cron: avoid it for anything that needs application logic or external API calls.
• Kubernetes CronJob: avoid it for sub-minute jobs, because each run pays pod startup.

When to move to a dedicated scheduler: cross-job dependencies, backfills, or many teams sharing schedules.

Illustrative estimate (assumed numbers, not a benchmark)

Annual cost = setup hours + (12 × monthly operating hours) × hourly rate.

Assume the following:

• Hourly rate: $100.
• ShedLock: 16 setup hours and 1 operating hour per month, which comes to (16 + 12) × $100 = $2,800.
• Dedicated scheduler: 80 setup hours and 10 operating hours per month, which comes to (80 + 120) × $100 = $20,000.

Under these assumptions, the dedicated scheduler costs about seven times as much in the first year.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cron jobs multiple instances, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.