What is the difference between an in-process queue and a separate worker process?

Teams use background jobs to move work out of the request path when it is slow, depends on an unreliable third party, or may need retrying. Sending email, resizing images, calling payment webhooks, and rebuilding search indexes all fit. The user gets a response quickly, and the work happens afterward. The two models differ in where that work runs.

An in-process queue runs jobs inside the web server's own process. Depending on the runtime, it uses a thread pool, goroutines, or callbacks scheduled on the event loop. The queue is often an in-memory data structure, so enqueueing costs almost nothing and there is no extra infrastructure. The jobs also share the web process's CPU, memory, and lifetime. When the process restarts or crashes, pending jobs in memory are lost.

A separate worker process boots the same application code, including models, configuration, and database connections, but it serves no HTTP traffic. It only pulls jobs from a shared backend, usually Redis or a database table. The web process writes a job record and returns. The worker reads that record and executes it. Because the job sits in durable storage between those two steps, it survives restarts, and you can scale workers separately from web capacity.

In-process queue                 Separate worker process

+---------------------+          +-------------+      +--------------+
| Web process         |          | Web process |      | Worker proc  |
|  request handlers   |          |  (enqueue)  |      | (same code,  |
|  + in-memory queue  |          +------+------+      |  no HTTP)    |
|  + job threads      |                 |             +------+-------+
+---------------------+                 v                    ^
                                 +--------------------------+|
                                 | Queue backend            |+
                                 | (Redis / DB table)       |
                                 +--------------------------+
 IN-PROCESS QUEUE                  SEPARATE WORKER PROCESS

       HTTP requests                 HTTP requests
             v                             v
+--------------------------+       +----------------+      +----------------+
| Web process              |       | Web process    |      | Worker process |
|                          |       | handlers only  |      | same app code  |
|  request handlers        |       | no job work    |      | no HTTP        |
|        |                 |       +-------+--------+      +-------+--------+
|        | enqueue         |               |                       ^
|        v                 |               | 1. write job          | 2. fetch job
|  +--------------------+  |               v                       |
|  | in-memory queue    |  |       +-------------------------------+--------+
|  +---------+----------+  |       | Queue backend (Redis or DB table)      |
|            |             |       | durable job records; survive restarts  |
|            v             |       +----------------------------------------+
|  +--------------------+  |
|  | job threads /      |  |       Jobs persist between steps 1 and 2;
|  | goroutines / loop  |  |       workers scale apart from web capacity.
|  +--------------------+  |
|                          |
| shares CPU, memory, and  |
| lifetime with requests;  |
| lost on restart/crash    |
+--------------------------+

An in-process queue keeps jobs in the web server's memory and runs them on its threads, while a separate worker process runs the same code, serves no HTTP, and pulls jobs from a shared Redis or database backend that the web process writes to.

How does it work? The lifecycle of a background job

Both models move a job through the same stages. What differs is where each stage's state lives and what survives a crash.

  1. Enqueue. The request handler builds a job: a name that identifies the handler, plus a small payload, usually IDs and primitives. In an in-process queue, this is a push onto an in-memory structure such as a channel, a thread-safe queue, or the event loop's task list. In a worker setup, the job is serialized, typically to JSON, and written to the backend: a Redis data structure or a row in a jobs table. The request then returns.

  2. Fetch and claim. A consumer takes the next job. In-process, a thread or goroutine reads from the in-memory structure. A worker process asks the backend for work, either by blocking on a Redis command until a job arrives or by polling a table. The claim must be atomic, so two workers never take the same job. A database queue usually does this with SELECT ... FOR UPDATE SKIP LOCKED, the same technique used by the outbox relay later in this article. Redis-based systems use atomic list operations or scripts. Many systems also attach a lock or lease with an expiry to the claimed job. If the worker stops renewing it, the job is treated as abandoned.

  3. Execute. The worker looks up the handler by job name, deserializes the payload, reloads any records it needs from the database, and runs the code. Because the payload carries IDs instead of objects, the handler sees current data, not a snapshot from enqueue time.

  4. Acknowledge. On success, the job is removed or marked complete. Until this happens, the backend still considers the job outstanding. The gap between "side effect done" and "acknowledged" is why delivery is at-least-once.

  5. Retry or give up. On an exception, the job is rescheduled with a delay, usually exponential backoff, and its attempt count increases. After the configured maximum, it moves to a dead or failed set where it waits for inspection or a manual retry. If the worker dies mid-job, the expired lock or a stalled-job check returns the job to the queue.

The in-process model can run stages 1 through 3, but its state for all five stages is process memory. Retries, attempt counts, and pending work vanish on restart. The worker model keeps that state in the backend, which is what makes retries, dead-letter handling, and independent scaling possible.

Why do in-process queues break down as a monolith grows?

The appeal is real. An in-process queue needs no new dependency, starts with the app, and shares its config and logging. You can set a breakpoint in a job and step through it in the same debugger session as the request that enqueued it. For a small app on one instance, that simplicity is often the right trade.

The problems come from the queue living in the same memory as the process. A deploy, a crash, or an OOM kill clears that memory. Jobs that were waiting are lost, and so are jobs that were halfway done, with no record that they ever existed. As traffic and job volume grow, more work is in that unrecorded state at any moment, so each restart loses more.

Jobs and requests also share the same resources. A report export that allocates heavily, a CPU-bound image resize, or a batch that holds database connections takes those from request handlers in the same process. In Node.js the cost is higher, because a synchronous stretch of job code blocks the event loop and every request on that process waits for it. You usually see this in p99 latency rather than in averages, and it is hard to trace because the slow request did nothing wrong.

Shutdown makes it worse. To restart cleanly, the process has to wait for running jobs to finish. A ten-minute job either holds up the deploy or runs past the platform's shutdown timeout and gets force-killed, which loses its work too.

How does the worker process model work with Sidekiq and BullMQ?

Moving jobs into a separate worker process fixes this by giving the work its own lifecycle. Sidekiq and BullMQ follow the same pattern: the web process serializes a small job payload into Redis and returns immediately, and a different process pulls jobs off Redis and runs them.

In Sidekiq, you define a class that includes Sidekiq::Job and implements perform, then enqueue it from a controller with perform_async, passing only JSON-friendly arguments such as IDs. The worker is the same codebase booted with bundle exec sidekiq. It runs jobs on a pool of threads sized by its concurrency setting. In BullMQ, the web process calls queue.add(name, data) on a Queue. A separate Node process creates a Worker with the queue name, a processor function, and a concurrency option. In both cases you can scale and restart workers independently of web traffic.

Neither system can promise exactly-once execution. A worker can finish the side effect and then crash before acknowledging the job, so the job runs again. Plan for at-least-once delivery.

The durability guarantee is only as strong as Redis. Without AOF or RDB persistence, a Redis restart can lose queued jobs. If Redis has maxmemory-policy set to anything other than noeviction, it can silently drop job keys under memory pressure. If you would rather not run Redis for this, database-backed queues store jobs as rows in your existing Postgres or MySQL. They trade some throughput for transactional enqueueing and one less system to operate.

// queue.ts - shared by the web and worker processes
import { Queue, Worker } from 'bullmq';

const connection = { host: 'redis.internal', port: 6379 };
export const emailQueue = new Queue('emails', { connection });

// web process: enqueue a small, JSON-friendly payload and return immediately
export async function onSignup(userId: number) {
  await emailQueue.add(
    'welcome',
    { userId },
    { attempts: 3, backoff: { type: 'exponential', delay: 1000 } }
  );
}

// worker process (separate entrypoint, e.g. `node dist/worker.js`)
export function startWorker() {
  const worker = new Worker(
    'emails',
    async (job) => {
      // Delivery is at-least-once: a crash after sending but before the job
      // is marked complete means this runs again, so the handler must be idempotent.
      const { userId } = job.data as { userId: number };
      if (await alreadySent(userId, 'welcome')) return;
      await sendWelcomeEmail(userId);
      await markSent(userId, 'welcome');
    },
    { connection, concurrency: 5 } // jobs processed in parallel by this process
  );

  worker.on('failed', (job, err) => {
    console.error(`job ${job?.id} failed: ${err.message}`);
  });
}

// app-specific helpers, assumed to exist elsewhere
declare function alreadySent(userId: number, kind: string): Promise<boolean>;
declare function sendWelcomeEmail(userId: number): Promise<void>;
declare function markSent(userId: number, kind: string): Promise<void>;

A minimal BullMQ setup. The web process enqueues a job with retries, and a separate worker process consumes it with its own concurrency and an idempotency check for at-least-once delivery.

How do you make background jobs safe to retry and consistent with your database?

Durable queues guarantee that a job runs at least once, not exactly once. A worker can finish sending an email and then crash before acknowledging the job. A lock can expire on a slow job, or a timeout can fire after the side effect already happened. In each case the queue assumes the job failed and delivers it again. Sidekiq retries failed jobs, and BullMQ reprocesses jobs it marks as stalled. Every handler must therefore produce the same outcome whether it runs once or three times.

Idempotency and state checks

Pass an idempotency key derived from the business event, such as order_id plus the action name. Pass it through to payment providers that accept one, and store it under a unique constraint so a second insert fails cleanly. For internal work, check state before acting: load the order, skip if receipt_sent_at is already set, and set that field in the same transaction as the write. Re-running then becomes a no-op.

Enqueueing and transactions

If you enqueue inside a transaction, the job can run before the commit, or after a rollback, and find no row to act on. Enqueue after commit instead. Most frameworks provide a commit callback for this. To avoid losing jobs when the process dies between commit and enqueue, use an outbox: write the job as a row in the same transaction, and have a relay publish it to the queue.

Arguments across deploys

During a rolling deploy, old workers pick up jobs enqueued by new code, and new workers pick up old jobs. Add arguments as optional with defaults. Remove an argument only after no queued jobs still carry it.

-- PostgreSQL outbox: the job row commits or rolls back with the business change.
CREATE TABLE job_outbox (
  id              bigserial PRIMARY KEY,
  idempotency_key text NOT NULL UNIQUE,   -- business event + action, e.g. 'order:1042:send_receipt'
  job_type        text NOT NULL,
  payload         jsonb NOT NULL,         -- add new fields as optional so old workers can ignore them
  published_at    timestamptz             -- NULL until the relay has pushed it to the queue
);

-- Request path: one transaction. A rollback discards both rows, so no job runs against missing data.
BEGIN;
UPDATE orders SET status = 'paid' WHERE id = 1042;
INSERT INTO job_outbox (idempotency_key, job_type, payload)
VALUES ('order:1042:send_receipt', 'send_receipt', '{"order_id": 1042}')
ON CONFLICT (idempotency_key) DO NOTHING;  -- a retried request does not create a second job
COMMIT;

-- Relay: claim a batch. SKIP LOCKED lets several relay processes run without double-claiming.
BEGIN;
SELECT id, job_type, payload
FROM job_outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;
-- ...enqueue each row to the queue, then mark the batch published.
-- A crash before this UPDATE republishes the batch, so handlers must still be idempotent.
UPDATE job_outbox SET published_at = now() WHERE id = ANY('{101,102,103}'::bigint[]);
COMMIT;

-- Handler: guard the side effect with a state check; zero rows returned means already done, so skip.
UPDATE orders SET receipt_sent_at = now()
WHERE id = 1042 AND receipt_sent_at IS NULL
RETURNING id;

A PostgreSQL outbox writes the job in the same transaction as the order update, a relay publishes it at least once, and a unique idempotency key plus a conditional UPDATE keep duplicate requests and redelivered jobs from repeating work.

What changes operationally when you add a worker process?

A worker becomes its own deployment target with its own replica count, resource limits, and health checks. The web process's HTTP readiness probe says nothing about whether workers are fetching jobs. Use a liveness signal such as a heartbeat the worker updates after each fetch loop.

Concern In-process queue Worker process
Scaling Tied to web replicas Independent of web
Health check Web endpoint Worker heartbeat
Failure visibility HTTP 500 Failed-job queue and alerts
Infrastructure None extra Redis or queue table

Graceful shutdown. On SIGTERM, a worker should stop fetching, give in-flight jobs a bounded time to finish, then return unfinished ones to the queue. Sidekiq and BullMQ both follow this pattern. Set the orchestrator's termination grace period longer than the worker's shutdown timeout. Otherwise the worker is killed mid-job and the job runs again, which is why retry safety matters.

Connection budget (illustrative estimate). Assume Postgres allows 200 connections and you reserve 20 for admin and migrations. Six web pods with pools of 10 use 60. Four worker pods at concurrency 10 need pools of at least 10 each, which uses 40. That totals 100, and a rolling deploy can briefly double it to 200, which exceeds the 180 available. Either cap surge or shrink the pools.

Monitoring. Track queue depth and the age of the oldest waiting job. Depth alone looks fine when a few slow jobs are stuck. Job age shows the delay users actually experience.

Asynchronous failures. Errors no longer surface as 500s. Route jobs that exhaust their retries to a dead-letter set and alert on its growth.

Redis or the queue table also needs provisioning, monitoring, and a backup plan.

When should you use an in-process queue, and when should you avoid it?

An in-process queue is acceptable when losing a job costs nothing. Cache warming, analytics pings and "someone viewed your profile" notifications fit: they finish in milliseconds, the next request can redo them, and nobody files a ticket when a deploy drops a few. If a job must survive a restart, runs longer than a request, needs retries with backoff, or has volume that rises and falls independently of web traffic, move it to a worker process.

In-process queue Redis-backed worker (Sidekiq, BullMQ) Database-backed worker
Durability Lost on restart or crash Survives restarts; depends on Redis persistence settings As durable as your primary database
Latency isolation None; jobs compete with requests for CPU Full; separate process Full; separate process
Operational cost Lowest Adds Redis to run, monitor and back up Adds polling load to the primary database
Transactional enqueue No No; needs an outbox pattern Yes; same transaction as the write
Scaling Tied to web instances High throughput; scale workers independently Limited by database write and lock capacity

A database-backed worker suits small teams with moderate job volume. Transactional enqueue removes a whole class of consistency bugs, and there is no new datastore to operate. A Redis-backed worker fits teams that run Redis already or process enough jobs to put polling pressure on the primary database.

These choices also shape long-term cost. In-process queues are cheap to build but expensive to debug, because their failures are silent. Worker processes cost more up front but make failures visible and retryable.

What are the alternatives to in-process queues and worker processes?

The two models above are not the only ways to get work out of the request path. Each alternative below solves part of the problem and fits a narrower set of jobs.

Do the work synchronously. If the operation takes a few hundred milliseconds and the user needs its result anyway, keep it in the request. You avoid every consistency and retry question in this article. This stops working once the operation is slow, calls an unreliable third party, or must be retried independently of the user's request.

Scheduled batch jobs. Instead of enqueueing per event, write state to the database (for example, receipt_sent_at IS NULL) and let a cron job or the platform's scheduler sweep for pending rows every few minutes. This is durable and simple, and the database row doubles as the job record. The costs are latency up to one schedule interval and a sweep query that must stay cheap as the table grows. It fits digest emails, reconciliation, and cleanup well.

Spawning a subprocess per job. The web process can fork or spawn a short-lived child process for a heavy task. This isolates CPU and memory from request handling, but the job still has no durable record, so it is lost if the host restarts. It also gives you no retries, concurrency limits, or visibility unless you build them yourself.

A general-purpose message broker. Dedicated brokers offer routing, fan-out to multiple consumers, and stronger delivery controls than a job library on Redis. They are worth considering when several services consume the same events. For a single monolith that only needs "run this function later," they add a system to operate without adding much.

A managed queue service from your cloud provider. The provider runs the storage and availability, and you still run your own worker process, or a provider-run function, to consume messages. This removes Redis operations from your plate. In exchange, you accept the service's message size limits, visibility-timeout semantics, and local-development friction, since you need an emulator or a real account to test.

Serverless functions triggered by a queue. Each job runs in a function invocation that scales with queue depth. This works for spiky, stateless jobs. It fits poorly with a monolith, because the function has to package and boot your application code and open database connections on each cold start, which strains the connection budget described above.

For most monoliths, the practical choice is still between a database-backed worker and a Redis-backed worker, with scheduled sweeps covering periodic work. The alternatives become relevant when a specific constraint, such as cross-service fan-out or no capacity to operate Redis, rules those out.

How can a worker process save engineering time?

The case for a worker process is usually made on reliability, but most of the payoff shows up as engineering hours that stop going to a few recurring tasks.

Less time investigating jobs that left no trace. When an in-process job is lost on deploy, the only evidence is a customer reporting that something never happened. An engineer then reconstructs events from logs, works out which other users were affected, and re-runs the work by hand. With a durable queue, a failed job sits in a dead or failed set with its arguments, error, and attempt count. Recovery is a retry, and the affected users are a query away.

Fewer hand-written retry loops. Without a queue, each integration tends to grow its own retry, backoff, and timeout code, written slightly differently each time. A job system provides these per job through configuration, as in the attempts and backoff options in the BullMQ example. Reviewers stop re-checking the same logic in every pull request.

Shorter latency investigations. When a background job and a request share a process, a p99 spike sends someone profiling request code that did nothing wrong. With jobs in their own process, web latency reflects web work, and slow jobs show up as queue age, which points directly at the cause.

Less deploy friction. Deploys no longer wait on, or kill, long jobs running inside web processes. Web and worker restarts are decoupled, so a deploy that only changes web code does not need to drain the job pool.

Capacity changes without code changes. A backlog is handled by adding worker replicas or raising concurrency, instead of resizing the whole web tier or rewriting a job to be faster under deadline.

Illustrative estimate. Assume a team sees two lost-job incidents a month after deploys, each taking an engineer an assumed four hours to detect, scope, and repair by hand. That is about eight hours a month. Assume the migration in the next section takes an assumed forty engineer-hours, mostly the Step 1 refactor. Under these assumed inputs, the migration pays for itself in roughly five months from incident handling alone, before counting retry code or latency debugging. Your inputs will differ. The useful exercise is to count your own incidents and the time each one took.

The time is not free in the other direction. Someone has to operate Redis or the queue table, set up dead-letter alerts, and make handlers idempotent. Those are one-time or low-ongoing costs, while lost-job incidents tend to grow with traffic.

How do you migrate from in-process jobs to a worker process without splitting the monolith?

The migration does not require a new service, a new repository, or a rewrite. It is a refactor followed by a configuration change, in three steps.

Step 1: Put an interface in front of every enqueue

Define one interface, such as JobQueue.enqueue(jobName, payload), and route every background call site through it. Payloads should be plain, serializable data like IDs and primitives. Live objects and open connections cannot cross a process boundary later. This step usually exposes hidden coupling, which is worth finding before you depend on a queue.

Step 2: Back it with in-process execution

The first adapter, InProcessQueue, runs the handler the way your code does today. Behavior does not change, so you can ship this step on its own and confirm that every call site now goes through the interface.

Step 3: Swap in a durable adapter

Add a second adapter, DurableQueue, that writes the job to a broker. A worker process then runs the same handler code from the same codebase. In Ruby this is typically Sidekiq. In Node.js it is typically BullMQ. Select the adapter per job type through configuration, so you can move one low-risk queue first, watch it, and then move the next.

Why this costs less than teams expect

  • No service extraction. The worker is the monolith started with a different entrypoint.
  • One codebase. Models, validation, and tests are shared, so nothing drifts between web and worker.
  • Queue-by-queue migration. Each queue moves on its own schedule instead of in one big cutover.
  • Simple rollback. Reverting a queue is a config change back to InProcessQueue, with no data migration.

Most of the work is the refactor in Step 1, which reduces coupling regardless of which queue you end up using.

import { Queue, Worker } from "bullmq";

type Payload = Record<string, string | number | boolean>; // IDs and primitives only
export interface JobQueue { enqueue(jobName: string, payload: Payload): Promise<void>; }

// Shared handlers: the same code runs in the web process or the worker.
export const handlers: Record<string, (p: Payload) => Promise<void>> = {
  sendWelcomeEmail: async ({ userId }) => { /* load user by ID, send email */ },
};

// Step 2: today's behavior, now behind the interface.
class InProcessQueue implements JobQueue {
  async enqueue(jobName: string, payload: Payload) {
    handlers[jobName](payload).catch((err) => console.error(jobName, err));
  }
}

// Step 3: persist the job in Redis. A worker process runs it later.
const connection = { host: "redis.internal", port: 6379 };
class DurableQueue implements JobQueue {
  private queue = new Queue("default", { connection });
  async enqueue(jobName: string, payload: Payload) {
    await this.queue.add(jobName, payload);
  }
}

// Choose the adapter per job type in config. To roll back, remove the name.
const durableJobs = new Set((process.env.DURABLE_JOBS ?? "").split(","));
const inProcess = new InProcessQueue();
const durable = new DurableQueue();
export const jobQueue: JobQueue = {
  enqueue: (name, payload) =>
    (durableJobs.has(name) ? durable : inProcess).enqueue(name, payload),
};

// worker.ts: the same monolith started with a different entrypoint.
export const startWorker = () =>
  new Worker("default", (job) => handlers[job.name](job.data), { connection });

A single JobQueue interface has an in-process adapter and a BullMQ-backed durable adapter. Each job type picks its adapter from config. The worker reuses the monolith's handlers, so call sites never change and rolling back a job is a one-line config edit.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to monolith background jobs, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.