Separate Worker Process in a Monolith: Splitting Web and Worker

What a separate worker process is and when a monolith needs one

A separate worker process means the monolith's single build artifact runs as two process types. The web process serves HTTP and enqueues work. The worker process pulls jobs from a durable queue and executes them. The code, dependencies and deploy pipeline stay the same. Only the entrypoint and the resources each process gets differ. You are not extracting a service. You are giving background work its own runtime so it stops competing with requests.

Teams usually make this change after the symptoms become hard to ignore:

• Latency spikes that line up with job bursts. A nightly import or a fan-out of email jobs takes CPU and database connections away from request handlers in the same process.
• Memory growth from long jobs. A report generator that loads large datasets inflates the shared process, and the web tier pays for it with restarts or OOM kills.
• Deploys that kill in-flight work. Restarting web nodes also aborts half-finished jobs, which then have to be retried or repaired by hand.
• Coupled scaling. Adding instances to absorb a queue backlog also adds idle web capacity, and the reverse is true too.

Who should use it

This pattern is for teams that run a single deployable application and do a meaningful amount of background work alongside request handling. Typical examples include order processing, notification fan-out, imports, exports, report generation and webhook delivery. It fits best when:

• The team owns one codebase and one deploy pipeline and wants to keep it that way. The split gives runtime isolation without the contracts, versioning and separate repository that a new service would require.
• Someone can own a second process type in production. That means deploy configuration, autoscaling rules, alerts and on-call runbooks for workers, as covered later in this article. If nobody can take that on, the split adds risk instead of removing it.
• The platform can run the same artifact with different commands. Procfile-based platforms, container orchestrators and Compose all support this. If your hosting only allows one process per application, you would have to work around that first.

It is a poor fit for teams whose background jobs need their own domain model, data ownership or release cadence. Those teams are better served by a separate job service, as listed in the alternatives table below.

When should it be used

Make the split when at least one of the symptoms above is recurring and you can trace it to background work rather than to request handling itself. Practical triggers include:

• Request latency or error rates degrade during known job windows, such as nightly batches or bulk sends.
• Deploys have to be timed around long-running jobs, or someone regularly repairs jobs that a deploy interrupted.
• You keep scaling the whole application to clear a queue backlog.
• Job memory use is forcing larger instances for the web tier.

Check the prerequisites before starting. Jobs must go through a durable queue, not threads or in-process adapters. Handlers must be idempotent or made so. The database must have connection headroom for a second pool. The following sections cover each of these. Do the split before considering service extraction. It is cheaper, it can be reversed one queue at a time, and it often removes the pressure that would otherwise push a team toward a new service.

Avoid the split when job volume is light and jobs are short, such as a few transactional emails per minute. In that case a co-located worker thread or in-process queue is cheaper. It has fewer moving parts and still meets your latency targets.

How it works: one artifact, two entrypoints

Once volume outgrows that, the usual first move is not a new service. You keep the same codebase and build artifact and run it in two roles. The CI pipeline produces one container image. The web process starts it with the HTTP server command. The worker process starts the same image with the job runner command. Both connect to the same database and the same queue broker, so a web request enqueues a job and a worker on another host dequeues it.

Why one artifact matters: the job's code and the code that enqueued it always come from the same commit. You avoid version skew where a worker deserializes a payload shape that the web tier no longer produces. One test suite also covers both roles.

Declaring process types

The role is chosen at deploy time, not in code. In a Procfile-style definition, each line maps a process type such as web or worker to a start command. On a container platform, both deployments reference the same image tag and the worker overrides the container command. Configuration still comes from environment variables, so each environment can run a different number of each process type without a branch or build flag. Staging might run one worker, and production might run twelve.

The limit

A worker still boots the whole monolith. That includes the web framework, every model, and every initializer. Each worker process therefore carries roughly the same memory footprint and startup time as a web process, even if it only sends emails. That cost shapes the scaling decisions covered later.

Copyx-app: &app
  # Both roles use the same CI-built tag, so the enqueuing code and the
  # job code always come from the same commit.
  image: registry.example.com/shop:3f9c2ab
  environment:
    # Shared state: both roles talk to the same database and queue broker.
    DATABASE_URL: postgres://shop:changeme@db:5432/shop
    REDIS_URL: redis://redis:6379/0
  depends_on: [db, redis]

services:
  web:
    <<: *app
    command: ["bundle", "exec", "puma", "-C", "config/puma.rb"]
    ports: ["3000:3000"]

  worker:
    <<: *app
    # Only the start command differs. No branch, no build flag.
    command: ["bundle", "exec", "sidekiq"]
    deploy:
      replicas: 2  # Scale per environment, e.g. 1 in staging, more in production.

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: shop
      POSTGRES_PASSWORD: changeme

  redis:
    image: redis:7

A Compose file runs one Rails image as both a web process and a Sidekiq worker. Only the container command differs between them, and both connect to the same Postgres database and Redis queue. The comments show the equivalent Procfile lines.

Making the job boundary explicit in code

Before splitting anything, find every place the web process runs work outside a request without going through the queue. Typical offenders are "fire and forget" thread pools, after_commit hooks that call a mailer directly, and an in-process adapter left on from development. Each of these breaks once the web and worker roles are separate. The work dies with the web pod on deploy, competes with request threads for CPU, and never shows up in queue metrics. Route all of it through the durable queue, even jobs that feel too small to bother with.

Next, audit job code for anything that assumes it shares a process with the request that enqueued it. Look for reads from an in-memory cache the web process warmed, request-scoped context such as locale or tenant, and "the current user" pulled from a thread-local. In a worker, these resolve to nothing or to stale data, and the job does not raise an error. Pass that context explicitly as job arguments.

Rolling deploys mean old web pods will enqueue jobs that new workers run, and the reverse can happen too. Three rules keep this safe:

• Enqueue IDs, not objects. A serialized object freezes its schema at enqueue time. An ID lets the handler load current state.
• Add arguments as optional with defaults. Old producers will omit them.
• Deploy handlers before producers. A worker must understand a job shape before anything emits it.

No queue library enforces these rules. Make them part of code review.

Copyfrom celery import Celery

app = Celery("shop", broker="redis://queue.internal:6379/0")

@app.task
def send_receipt(order_id, user_id, locale="en", include_invoice=False):
    # Load current state at run time instead of trusting a snapshot frozen at enqueue.
    order = load_order(order_id)
    user = load_user(user_id)
    # locale was request-scoped on the web side; it arrives as an argument here.
    # include_invoice is new: optional with a default, so jobs from old
    # web pods that omit it still run on new workers.
    render_and_send(order, user, locale=locale, include_invoice=include_invoice)


def on_order_committed(order, request):
    send_receipt.delay(
        order.id,
        request.user.id,
        locale=request.locale,
        include_invoice=True,
    )


def load_order(order_id): ...
def load_user(user_id): ...
def render_and_send(order, user, locale, include_invoice): ...

A Celery task that replaces a thread-pool call and a thread-local user lookup. It takes IDs and explicit context as arguments, and its new argument is optional so it stays compatible with older producers during a rolling deploy.

Graceful shutdown and idempotent jobs

When the orchestrator stops a worker, it sends SIGTERM, waits out a grace period, then sends SIGKILL. The worker has to use that window in a specific order. It should first stop fetching: close the consumer, stop polling, or release its claim on the queue so no new job starts. Next, it lets in-flight jobs run to completion. Only then does it close connections and exit with a zero status. If the worker ignores SIGTERM, every deploy turns into a hard kill. If it exits immediately, every deploy abandons work mid-flight. Set the worker's internal drain timeout a few seconds shorter than the platform's grace period, so the worker can log what it failed to finish before SIGKILL arrives.

Idempotency is a requirement, not a nicety

Most queues deliver at least once. A job whose acknowledgement was lost, whose visibility timeout expired, or whose worker was killed will run again. Every handler therefore needs to produce the same result when it runs twice. This usually requires schema changes. Examples include a dedupe key stored with a unique constraint, an INSERT that tolerates conflicts, or a state column the handler checks before acting. For external side effects such as emails or payments, pass an idempotency key to the provider when it supports one. Where it doesn't, record "sent" in the same transaction that guards the send.

Jobs longer than the shutdown window

A job that runs for twenty minutes will not finish inside a thirty-second grace period. You have two honest options. The first is to split the work into resumable chunks: process a batch, persist a cursor, and enqueue the next chunk. The second is to accept that a deploy will kill the job and a retry will rerun it from the start, along with any non-idempotent side effects it already triggered.

Copyimport os, signal, logging, threading, time
from concurrent.futures import ThreadPoolExecutor, wait
import psycopg
from myapp.queue import queue  # placeholder: your queue client (fetch/ack/close)

log = logging.getLogger("worker")
DSN = "postgresql://app:changeme@db:5432/app"
DRAIN_TIMEOUT = 25  # a few seconds under the platform's 30s grace period
stopping = threading.Event()
signal.signal(signal.SIGTERM, lambda signum, frame: stopping.set())

def handle(job):
    # processed_jobs.job_id has a UNIQUE constraint, so a redelivered job is a no-op
    with psycopg.connect(DSN) as conn, conn.transaction():
        first = conn.execute(
            "INSERT INTO processed_jobs (job_id) VALUES (%s) "
            "ON CONFLICT (job_id) DO NOTHING RETURNING job_id", (job.id,)).fetchone()
        if first is None:
            return  # an earlier delivery already committed this work
        conn.execute("UPDATE invoices SET status = 'paid' WHERE id = %s", (job.invoice_id,))
    queue.ack(job)  # ack only after commit; a crash before this means redelivery

pool = ThreadPoolExecutor(max_workers=4)
in_flight = set()
while not stopping.is_set():
    in_flight = {f for f in in_flight if not f.done()}
    if len(in_flight) >= 4:
        time.sleep(0.1)
        continue
    job = queue.fetch(timeout=1)  # short poll so SIGTERM is noticed quickly
    if job:
        in_flight.add(pool.submit(handle, job))

queue.close()  # step 1: stop fetching new jobs
done, not_done = wait(in_flight, timeout=DRAIN_TIMEOUT)  # step 2: drain
if not_done:
    log.warning("%d jobs unfinished; they will be redelivered", len(not_done))
logging.shutdown()
os._exit(0)  # step 3: exit without blocking on stuck threads

A Python worker that stops fetching on SIGTERM, drains in-flight jobs within a timeout shorter than the grace period, and uses a unique-constrained dedupe key so redelivered jobs do no duplicate work.

Connection pools, scaling signals and queue separation

Once the worker is its own process, it needs its own connection budget. Each process type opens its own pools, and scaling one tier can quietly consume connections the other tier needs.

Illustrative estimate (assumed inputs): The database allows 200 connections, with 10 reserved for admin and migrations, so 190 are usable. The web tier runs 6 replicas × 4 processes × a pool of 5, which is 120 connections. The worker tier runs 4 replicas at a concurrency of 10, which is 40 connections. The total is 160, which fits. During a backlog, someone doubles the workers to 8. Workers now hold 80 connections, and the total reaches 200, which exceeds 190. Web requests start waiting on the pool and time out. The backlog drain has caused a user-facing outage. To prevent this, cap the worker pool size and the maximum replica count so that the worst-case total stays under the limit. If you use a connection pooler in front of the database, give web and worker separate pools.

Scale workers on queue signals rather than CPU. Worker CPU stays low when jobs spend most of their time waiting on I/O, so CPU does not show that work is piling up.

Bursty load can cause the autoscaler to oscillate between scaling up and down. Use cooldown windows and scale down slowly to dampen this. Scaling to zero saves money, but the first job after an idle period pays for scheduling, image pull and boot time before it runs.

Separate queues by priority or workload, each with its own workers. This keeps a slow report export from blocking password-reset emails.

Copyimport sys

DB_MAX_CONNECTIONS = 200
RESERVED = 10  # kept free for admin sessions and migrations

TIERS = {
    "web":    {"max_replicas": 6, "procs_per_replica": 4, "pool_size": 5},
    # Worker pool sized to job concurrency: each running job may hold one connection.
    "worker": {"max_replicas": 4, "procs_per_replica": 1, "pool_size": 10},
}

def check(tiers):
    usable = DB_MAX_CONNECTIONS - RESERVED
    total = 0
    for name, t in tiers.items():
        n = t["max_replicas"] * t["procs_per_replica"] * t["pool_size"]
        total += n
        print(f"{name:>7}: {n} connections at max scale")
    print(f"  total: {total} of {usable} usable -> {'OK' if total <= usable else 'OVER LIMIT'}")
    return total <= usable

if __name__ == "__main__":
    ok = check(TIERS)  # 120 + 40 = 160, fits

    # What-if: raising worker max replicas to 8 to drain a backlog.
    # 120 + 80 = 200 > 190, so web requests would wait on the pool.
    proposed = {k: dict(v) for k, v in TIERS.items()}
    proposed["worker"]["max_replicas"] = 8
    print("--- proposed: worker max_replicas = 8 ---")
    check(proposed)

    # Fail the pipeline only if the configured limits exceed the budget.
    sys.exit(0 if ok else 1)

A small CI check that multiplies each tier's maximum replicas, processes and pool size, compares the worst-case total with the usable database connections, and shows that doubling workers to 8 pushes the total to 200 against a usable limit of 190.

Health checks and observability without a request path

A worker has no request path, so the usual readiness check doesn't fit. Pointing an HTTP probe at a web route only proves that the web process can answer HTTP. A worker can be completely stuck while that route returns 200: blocked on a lock, wedged in a fetch loop, or holding a dead broker connection. The probe has to measure the worker's own loop.

Liveness for workers

There are two common options:

• Heartbeat. The worker writes a timestamp to a file, a key, or a table each time it completes a poll cycle. The liveness check fails when that timestamp is older than a threshold. Set the threshold well above your longest legitimate poll interval so slow jobs don't trigger restarts.
• Small health endpoint. The worker runs a minimal HTTP listener on a separate port. It reports healthy only if the main loop has made progress recently. Don't report healthy just because the listener thread is running.

Per-queue metrics

Track these per queue, not in aggregate. A healthy default queue can hide a starved one.

• Throughput: jobs completed per interval.
• Failures: jobs that exhausted retries or were dead-lettered.
• Retries: a rising rate often comes before failures and points to a flaky dependency.
• Oldest-job age: how long the oldest waiting job has been queued.

Alert on oldest-job age exceeding the queue's latency budget. Depth alone misleads, because a deep queue that drains quickly is fine. Also alert on a sustained failure rate and on throughput dropping to zero while jobs are waiting.

All of this adds surface. You now maintain dashboards, alerts, and on-call runbooks for two process types, and each one needs an owner.

A staged migration plan, alternatives and cost

That surface is the price. The migration should be cheap to reverse at every step.

Rollout in four steps

1. Run dedicated workers alongside web. Deploy the worker entrypoint with no queue subscriptions, or with one low-risk queue that web also consumes. Idempotency makes the double consumption safe. Rollback: scale workers to zero.
2. Move one queue at a time. Subscribe workers to the queue, then remove it from web's subscription list using a config flag, not a code change. Rollback: flip the flag back.
3. Verify over a full traffic cycle. Compare queue age, failure rate and duplicate executions against the baseline from before the move. One quiet afternoon is not enough. Include a deploy and a peak period.
4. Remove job execution from web. Delete the consumer code only after every queue has moved. Rollback: redeploy the previous release. Keep the flag for one release cycle.

Alternatives

How it saves engineering time

The time savings come from removing recurring manual work and shortening investigations, not from making jobs run faster. The main sources are:

• Less repair work after deploys. Workers drain on SIGTERM and handlers are idempotent, so a deploy no longer leaves half-finished jobs that someone has to find, inspect and re-run by hand.
• No deploy scheduling around batch work. Web deploys don't touch workers. Engineers stop waiting for a nightly import to finish or coordinating release windows with whoever owns a long job.
• Faster triage. With separate process types, per-queue metrics and worker-specific liveness checks, a latency incident can be attributed to the web tier or the job tier from dashboards. Engineers don't have to correlate request slowdowns with job timestamps in a shared process's logs.
• Less capacity firefighting. Draining a backlog means raising worker replicas within a pre-checked connection budget. It doesn't require resizing the whole application or watching web latency while doing so.
• Avoided extraction work. Because the split reuses one artifact, one test suite and one pipeline, it removes much of the pressure to build a separate job service. That avoids the cost of new contracts, a second repository and cross-service versioning.

Against that, count the ongoing upkeep: worker deploy configuration, autoscaling rules, alerts, runbooks, and the code-review discipline around job arguments. The cost model below puts both sides in one formula.

Cost

Illustrative estimate (assumed numbers, not a benchmark).

Payback months = one-time cost ÷ (monthly hours saved − monthly upkeep)

Assumed inputs:

• One-time cost: 80 hours for deploy configs, autoscaling rules, alerts and idempotency fixes.
• Hours saved: 6 hours a month from fewer lost jobs and 4 hours a month from faster triage, for 10 hours total.
• Upkeep: 2 hours a month.

Result: 80 ÷ (10 − 2) = 10 months.

If you can't measure a meaningful number of saved hours today, stay co-located.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to separate worker process monolith, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.