BullMQ in NestJS: Retries, Concurrency and Stalled Job Recovery

How BullMQ fits into a NestJS app and when to use it

BullMQ is a job queue for Node.js that stores jobs, their state and their locks in Redis. In a NestJS app it usually sits behind an injectable queue that controllers or services use to enqueue work. Separate worker processes, which can run the same codebase, pull jobs from Redis and execute them. The HTTP request returns as soon as the job is persisted. The slow or unreliable part, such as sending an email, calling a third-party webhook or rendering a large report, happens elsewhere with its own retry policy and its own scaling.

This model suits backend and platform teams that already run Redis and want two things a plain async call cannot give them. The first is durable retries that survive a process crash. The second is the ability to add workers horizontally when the backlog grows.

It is a poor fit in several cases:

• Work that needs exactly-once execution. BullMQ delivers at least once, so a job can run twice.
• Teams that cannot operate Redis reliably. That includes persistence, memory limits and failover. When Redis loses data, the queue loses jobs.
• Trivial fire-and-forget tasks. If a lost task costs nothing, an in-process call is simpler.
• Cross-service event streaming. Consumers that need replayable, ordered logs are better served by a log-based broker.

The rest of this tutorial treats retries, concurrency and stalled-job detection as one system. A lock that is too short for your concurrency level triggers stalls. Each stall leads to a re-run, and every re-run is effectively a retry that your idempotency design has to absorb.

Copy+--------+                   +--------------------------------+
| Client |------------------>| NestJS API process             |
|        |<------------------| Controller -> Service          |
+--------+ 202, job persisted| -> injected Queue.add()        |
                             +----------------+---------------+
                                              | enqueue
                                              v
+-------------------------------------------------------------+
| Redis: job data, state and locks                            |
|   waiting --> active --> completed                          |
|      ^          |                                           |
|      |          +--> failed (attempts exhausted)            |
|      +----------+  retry (backoff) or stalled re-run        |
+------------+--------------------------------+---------------+
             | fetch + lock                   | fetch + lock
             v                                v
+--------------------------+     +--------------------------+
| Worker process A         |     | Worker process B         |
| concurrency = N          |     | concurrency = N          |
| renews lock while busy   |     | (add more to scale out)  |
+------------+-------------+     +------------+-------------+
             |                                |
             v                                v
+-------------------------------------------------------------+
| Slow or unreliable work: email, webhooks, report rendering|
| At-least-once delivery: handlers must be idempotent        |
+-------------------------------------------------------------+

The NestJS API enqueues a job into Redis and responds immediately. Horizontally scaled workers then lock and run jobs at a set concurrency, and both retries and stalled-lock re-runs send jobs back to waiting, so handlers must be idempotent.

Wiring queues, producers and processors with the NestJS BullMQ module

The @nestjs/bullmq package wraps BullMQ's Queue and Worker classes in Nest's dependency injection, so wiring comes down to three pieces: a connection, a queue registration and a processor.

Connection and queue registration

Call BullModule.forRoot() once in your root module with the Redis connection options, usually read from ConfigService through forRootAsync(). Every queue registered afterwards shares this configuration. In the feature module, BullModule.registerQueue({ name: 'email' }) declares the queue. This is where defaultJobOptions belongs: set attempts, backoff (for example exponential with a base delay), and removeOnComplete / removeOnFail retention limits. When these live on the queue registration, a producer that calls add() without options still gets retries and bounded retention. Producers can override them for individual jobs, but they cannot forget to set them.

Producers

Inject the queue with @InjectQueue('email') and type it as BullMQ's Queue. A producer service then calls this.emailQueue.add('welcome', payload). Keep payloads small and serializable, such as IDs rather than whole entities, because the job data sits in Redis and the processor should load current state when it runs.

Processors

Decorate a class with @Processor('email'), extend WorkerHost, and implement async process(job). If the method returns normally, the job completes. If it throws, the job fails, and BullMQ applies the retry policy. You can branch on job.name when one queue carries several job types.

Redis itself is part of the reliability design. Set maxmemory-policy to noeviction. Under any other policy, Redis can silently evict job hashes or lock keys under memory pressure. When that happens, jobs disappear or appear stalled, and no error reaches your application.

Copyimport { Injectable, Module } from '@nestjs/common';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { BullModule, InjectQueue, Processor, WorkerHost } from '@nestjs/bullmq';
import { Job, Queue } from 'bullmq';

@Injectable()
export class EmailProducer {
  constructor(@InjectQueue('email') private readonly emailQueue: Queue) {}
  // No per-job options needed: queue defaults supply attempts, backoff, retention.
  // Pass an ID, not the entity; the processor loads current state.
  async sendWelcome(userId: string) {
    await this.emailQueue.add('welcome', { userId });
  }
}

@Processor('email')
export class EmailProcessor extends WorkerHost {
  async process(job: Job<{ userId: string }>): Promise<void> {
    // Returning completes the job; throwing fails it and triggers the retry policy.
    if (job.name === 'welcome') { /* load user by job.data.userId, send email */ }
  }
}

@Module({
  imports: [
    ConfigModule.forRoot(),
    // Shared Redis connection (normally in the root module); that Redis must use maxmemory-policy noeviction.
    BullModule.forRootAsync({
      imports: [ConfigModule],
      inject: [ConfigService],
      useFactory: (config: ConfigService) => ({
        connection: { host: config.get('REDIS_HOST', 'localhost'), port: Number(config.get('REDIS_PORT', 6379)) },
      }),
    }),
    // Queue-level defaults apply to every add() call that does not override them.
    BullModule.registerQueue({
      name: 'email',
      defaultJobOptions: { attempts: 5, backoff: { type: 'exponential', delay: 1000 }, removeOnComplete: 1000, removeOnFail: 5000 },
    }),
  ],
  providers: [EmailProducer, EmailProcessor],
})
export class EmailModule {}

A NestJS module that sets up the shared Redis connection and registers an 'email' queue with default retry and retention options, then adds a producer that enqueues jobs by ID and a WorkerHost processor that handles them.

Retries done right: attempts, backoff and unrecoverable errors

By default a job gets one attempt, so a failure is final unless you configure retries. Set attempts and backoff in defaultJobOptions when you register the queue, and override them per job where needed. backoff takes a type (fixed or exponential) and a delay in milliseconds. A fixed backoff waits delay between every retry. An exponential backoff doubles the wait after each failed attempt, starting from delay.

Retries only help when the next attempt could succeed. A timeout or a 503 might pass on the next try. A failed schema check or a reference to a deleted record will fail on every attempt. For those, throw UnrecoverableError from bullmq. The job moves straight to failed, and the remaining attempts are skipped. You get a clear failure reason, and the job does not spend minutes cycling through retries that cannot work.

Exponential backoff has a real cost. Illustrative estimate: with attempts: 10 and a 60-second base delay, the nine waits add up to 511 minutes, roughly 8.5 hours. That spacing protects the downstream service, but a bug can also sit unnoticed for most of a working day. Keep attempt counts modest, and alert when a job first fails rather than only when it fails for the last time.

Copyimport { Module } from '@nestjs/common';
import { BullModule, Processor, WorkerHost } from '@nestjs/bullmq';
import { Job, UnrecoverableError } from 'bullmq';

// Queue-wide default: 5 attempts with exponential backoff from 2s (waits of 2s, 4s, 8s, 16s).
// To use fixed backoff for one job instead:
// queue.add('sync', data, { attempts: 3, backoff: { type: 'fixed', delay: 1000 } })
@Module({
  imports: [
    BullModule.registerQueue({
      name: 'webhooks',
      defaultJobOptions: { attempts: 5, backoff: { type: 'exponential', delay: 2000 } },
    }),
  ],
})
export class WebhooksModule {}

@Processor('webhooks')
export class WebhookProcessor extends WorkerHost {
  async process(job: Job<{ url: string; payload: unknown }>) {
    // Bad input fails on every attempt, so skip the remaining retries.
    if (!job.data.url?.startsWith('https://')) {
      throw new UnrecoverableError(`Invalid webhook URL: ${job.data.url}`);
    }

    // A timeout throws a normal error, so BullMQ retries it with backoff.
    const res = await fetch(job.data.url, {
      method: 'POST',
      body: JSON.stringify(job.data.payload),
      signal: AbortSignal.timeout(10_000),
    });

    // 429 and 5xx may pass on a later attempt. Other 4xx responses will not.
    if (res.status === 429 || res.status >= 500) throw new Error(`Retryable: HTTP ${res.status}`);
    if (!res.ok) throw new UnrecoverableError(`Rejected: HTTP ${res.status}`);
  }
}

A NestJS queue registered with exponential backoff by default, and a processor that throws ordinary errors for timeouts, 429s and 5xx responses so they are retried, and UnrecoverableError for bad input and other 4xx responses so the job fails at once.

Designing idempotent processors for at-least-once delivery

BullMQ delivers jobs at least once, not exactly once. A worker can charge a card, then crash or lose its lock before it marks the job complete. The stalled-job check then moves the job back to waiting, and another worker runs the whole processor again. Retries behave the same way: a job that sent an email and then threw on a later step sends the email again on the next attempt. Retries decide whether a job runs again. They do nothing to make running it again harmless.

Give the job a stable identity

Derive the job ID from the business event, such as order-1234-receipt, instead of letting BullMQ generate one. BullMQ ignores an add with an ID that already exists, so a producer that retries its own request does not create a second job. This protection only holds while the job record exists. If removeOnComplete deletes completed jobs quickly, a late duplicate can be enqueued again.

Check before side effects

Before an irreversible step, look for a durable record that it already happened, such as a sent_at column or a payment row. Store that record in your database. Job progress or job data is a poor place for it because it may not survive a stall.

Make writes safe to repeat

Use upserts and unique constraints so a second insert fails or merges instead of duplicating rows. For external payment and messaging APIs that accept an idempotency key, pass a key derived from the job ID. The provider can then recognise the repeat even when your own check runs too late.

Copyimport { Processor, WorkerHost } from '@nestjs/bullmq';
import { Job, Queue } from 'bullmq';
import { Db } from './db'; // app-defined Postgres wrapper
import { PaymentsClient } from './payments-client'; // app-defined provider wrapper

type ChargeData = { orderId: string; amountCents: number };

// Producer: the job ID comes from the business event, so a repeated add is ignored
// while this job record still exists.
export async function enqueueCharge(queue: Queue, data: ChargeData) {
  await queue.add('charge', data, {
    jobId: `order-${data.orderId}-charge`,
    attempts: 5,
    backoff: { type: 'exponential', delay: 1000 },
  });
}

@Processor('payments')
export class ChargeProcessor extends WorkerHost {
  constructor(private readonly db: Db, private readonly payments: PaymentsClient) {
    super();
  }

  async process(job: Job<ChargeData>): Promise<string> {
    const { orderId, amountCents } = job.data;
    const key = `order-${orderId}-charge`; // same on every attempt and after a stall

    // 1. Check a durable record in the database before the irreversible step.
    const done = await this.db.query('SELECT provider_charge_id FROM payments WHERE order_id = $1', [orderId]);
    if (done.rows.length > 0) return done.rows[0].provider_charge_id;

    // 2. The provider deduplicates on the key if we crashed after charging but before recording.
    const charge = await this.payments.charge({ orderId, amountCents, idempotencyKey: key });

    // 3. The unique constraint on order_id makes a repeated insert a no-op instead of a second row.
    await this.db.query(
      'INSERT INTO payments (order_id, provider_charge_id) VALUES ($1, $2) ON CONFLICT (order_id) DO NOTHING',
      [orderId, charge.id],
    );
    return charge.id;
  }
}

A NestJS BullMQ processor combines three safeguards so that a retried or stall-recovered charge job runs its side effect at most once: a business-derived job ID at enqueue time, a database check before charging, and a provider idempotency key plus a conflict-safe insert.

Concurrency, locks and stalled jobs: how the pieces interact

Worker concurrency lets one Node.js process hold several jobs in flight at once. This works because most jobs spend their time awaiting the network or a database. CPU-bound work gets no benefit, because a single event loop still runs one callback at a time. Every unit of concurrency is multiplied across processes and pods: 10 per worker on 6 pods means 60 simultaneous queries, connections and calls against rate-limited APIs.

Each active job holds a lock in Redis that the worker renews periodically. If the process crashes, renewal stops. If synchronous code blocks the event loop, renewal can't run either. In both cases the lock expires and the stalled-job checker, which runs every stalledInterval, moves the job back to waiting. After maxStalledCount stalls the job fails. A blocked loop is the dangerous case, because the original run may still finish while a second worker starts the same job.

For heavy computation, use sandboxed processors (a separate file, optionally run in worker threads). The cost is that the sandboxed code sits outside the NestJS container, so you must construct its dependencies by hand.

Illustrative estimate

All inputs are assumed. Jobs take 2 s on average and 20 s at worst. Concurrency is 8 per worker across 5 pods, giving 40 jobs in flight, roughly 20 jobs/s, and up to 40 database connections. Set lockDuration to 60 s, which is three times the worst case.

With 5 attempts and exponential backoff starting at 5 s, the four retry delays are 5 + 10 + 20 + 40 = 75 s. Add five 20 s runs and the worst case reaches about 175 s before the job is marked failed.

Graceful shutdown and observability in production

Shutting down without losing work

NestJS does not run shutdown lifecycle hooks unless you call enableShutdownHooks() on the application during bootstrap. Without it, SIGTERM kills the process mid-job. The job's lock then expires and another worker picks it up as stalled, so it runs twice and has to rely on the idempotency from section 4. With hooks enabled, the BullMQ module closes its workers on shutdown. A closing worker stops fetching new jobs and waits for the active ones to finish.

That wait only helps if the platform allows it. The termination grace period (in Kubernetes, terminationGracePeriodSeconds) must be longer than your longest single job run plus some headroom. Use the duration of one run, not the full retry chain, because a retry scheduled for later sits in Redis and is not held by the pod. If a job runs 20 s, a 30 s grace period is tight. Anything longer than a few minutes is a sign the job should be split.

Watching the queue

Listen to completed, failed and stalled events and turn them into metrics, not just log lines. Two alerts matter most:

• Steady growth in the failed set. This means retries are exhausted faster than anyone triages them.
• Repeated stalls on the same queue. This usually points to CPU-bound handlers blocking the event loop, or to pods killed before their grace period ends.

Bounding Redis memory

By default, finished jobs stay in Redis indefinitely. Set removeOnComplete and removeOnFail with a count or age limit. Keep failed jobs longer than completed ones.

Tight retention has a cost. Once a job is evicted, its payload, error and stack trace are gone. If you need debugging or audit history, export them from the event listeners to your log pipeline or to durable storage before Redis drops them.

Alternatives and impact on cost and engineering time

That retention trade-off applies to every option below. The real question is which delivery model and operational burden your team wants to own.

Every row except cron is at-least-once, so the idempotency work from section 4 carries over to any of them. That is the strongest argument for doing it early. Adding a deduplication key and tuning lock duration takes a few days during design. Tracing a duplicate charge or a silently stuck queue happens under incident pressure, and the cost includes refunds and lost trust.

The work also compounds across teams. Shared queue-level defaults for attempts, backoff and retention, a common unrecoverable-error class, and standard alerts on failed-job count and queue age mean each new queue starts correctly configured.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to bullmq nestjs, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.