FastAPI Background Tasks vs Celery vs ARQ: Choosing a Job Runner

What each option actually is: in-process tasks, an asyncio Redis queue, a distributed task queue

FastAPI's BackgroundTasks is not a queue. It is a list of callables attached to the response object. They run in the same process after the response has been sent. Sync functions are dispatched to the threadpool. Async functions are awaited on the same event loop that serves requests. Nothing is written anywhere. There is no retry, no job ID, and no way to ask whether a task finished. The promise is narrow: do this soon, in this process, if it is still alive.

ARQ is a job queue built on asyncio and Redis. Enqueuing writes a job to Redis. A separate worker process pulls jobs and runs them as coroutines, many at once on a single event loop, up to a configured concurrency limit. Because the job lives in Redis rather than in a web worker's memory, it survives a web process restart. Because everything is async, it suits I/O-heavy work such as HTTP calls, database writes and sending email, and it shares idioms with an async FastAPI codebase. The promise: this job is recorded outside the request process and will be picked up by an async worker.

Celery is a distributed task queue that can use several message brokers, commonly RabbitMQ or Redis. Its task functions are synchronous by design. Concurrency comes from worker pools: prefork processes by default, with thread and green-thread options. It also offers:

• routing tasks to named queues
• periodic scheduling via beat
• composition primitives such as chains and groups
• per-task rate limits

The promise: a general-purpose job system you can shape around many workloads and many machines.

These are three different contracts, not three sizes of the same tool.

Copy[1] FastAPI BackgroundTasks: same process, after the response
+--------------------------------------------------------------+
| FastAPI process (each web worker)                            |
|  request --> handler --> response sent --> run task list     |
|    sync def  --> threadpool                                  |
|    async def --> same event loop as requests                 |
+--------------------------------------------------------------+
  nothing persisted: no retry, no job ID, no status

[2] ARQ: job recorded in Redis, run by an asyncio worker
+-------------+  enqueue  +-------+  pull   +-----------------+
| FastAPI app | --------> | Redis | ------> | ARQ worker      |
+-------------+           +-------+         | one event loop  |
                                            | N coroutines    |
                                            +-----------------+
  job survives a web process restart; suits I/O-bound async work

[3] Celery: broker-agnostic distributed queue, sync task functions
+-------------+  send   +-----------------+  route  +---------------+
| FastAPI app | ------> | Broker          | ------> | worker A      |
+-------------+         | RabbitMQ/Redis  |         | prefork pool  |
+-------------+  beat   | queues:         |         +---------------+
| celery beat | ------> |  default        | ------> | worker B      |
+-------------+         |  reports        |         | thread pool   |
                        +-----------------+         +---------------+
  also: chains, groups, per-task rate limits, many machines

Three different contracts: BackgroundTasks run in the web process with nothing persisted, ARQ records jobs in Redis for an async worker, and Celery routes sync tasks through a broker to named queues and worker pools.

The decision test: what happens to a job when the process dies?

Ask one question of every job: if the process running it is killed mid-flight, what happens to the work?

With BackgroundTasks, the job lives only in the memory of the web worker that served the request. A rolling deploy sends SIGTERM, and work still running when the grace period expires is lost. An OOM kill loses it immediately. An unhandled exception ends it with nothing to retry it. Nothing records that the job existed. Long tasks also hold web capacity, because the same worker processes and event loop must also serve requests.

ARQ and Celery write the job to a broker before any worker touches it, so a restart, a retry or a worker on separate hardware can pick it up.

If losing the job is acceptable and it finishes in seconds, BackgroundTasks is enough. If an async codebase sends email and calls webhooks, ARQ adds durability without a second concurrency model. Celery earns its operational weight once you need queues per workload, periodic schedules or process-based parallelism for CPU-bound tasks. For a team that only sends a few async jobs, that weight is overhead.

How concurrency works and where CPU-bound work breaks it

BackgroundTasks and ARQ both run jobs as coroutines on a single asyncio event loop. That loop is cooperative: a job only yields control at an await. If a job calls a blocking function such as a synchronous HTTP client, time.sleep, or a heavy JSON parse, nothing else on that loop runs until the call returns. In ARQ, every concurrent job on that worker stalls. With BackgroundTasks, the API requests served by the same process stall too.

Starlette runs plain def background tasks in a threadpool, which keeps blocking I/O off the loop. CPU-bound work is different, because those threads still contend for the GIL. Threads help you wait. They do not help you compute.

In ARQ, the fix is to hand CPU work to a process pool explicitly. Create a ProcessPoolExecutor in the worker's startup hook, store it in ctx, and await loop.run_in_executor(...) from the job. The function and its arguments must be picklable, so pass plain data rather than clients or sessions.

Celery's default prefork pool takes the opposite approach. Each child is a separate OS process with its own interpreter and GIL, so CPU-bound tasks run in parallel. A segfault or memory leak takes down one child, not the whole worker. The cost is memory, since every child carries a full copy of your application. worker_max_tasks_per_child exists partly to recycle children that bloat over time.

Celery tasks are synchronous, which makes reusing async code awkward. Calling asyncio.run() inside a task creates and tears down a new event loop on every invocation. Async clients and connection pools are tied to the loop that created them, so you cannot build them once at worker start and share them across tasks.

Copyimport asyncio
from concurrent.futures import ProcessPoolExecutor

from arq.connections import RedisSettings


def compute_score(values: list[float]) -> float:
    # Pure-Python CPU work. In a coroutine this would hold the GIL
    # and stall every other job on the event loop.
    total = 0.0
    for v in values:
        for _ in range(10_000):
            total += (v * v) % 7.0
    return total


async def startup(ctx: dict) -> None:
    # Create the pool once per worker. Each child process has its own GIL.
    ctx["cpu_pool"] = ProcessPoolExecutor(max_workers=2)


async def shutdown(ctx: dict) -> None:
    ctx["cpu_pool"].shutdown(wait=True)


async def score_report(ctx: dict, report_id: int, values: list[float]) -> float:
    loop = asyncio.get_running_loop()
    # Pass plain, picklable data only: no Redis clients or DB sessions.
    # The loop stays free to run other jobs while the child computes.
    return await loop.run_in_executor(ctx["cpu_pool"], compute_score, values)


class WorkerSettings:
    functions = [score_report]
    on_startup = startup
    on_shutdown = shutdown
    redis_settings = RedisSettings(host="localhost", port=6379)
    max_jobs = 10  # concurrent coroutines on this worker's single event loop

An ARQ worker that creates a ProcessPoolExecutor in its startup hook and offloads a CPU-bound function to it with run_in_executor, so the event loop keeps serving other jobs.

Delivery guarantees, idempotency and Redis durability

Both ARQ and Celery give you at-least-once delivery when configured for safety. The cost of that guarantee is that a job can run more than once.

With Celery's task_acks_late enabled, the message is acknowledged only after the task finishes. On the Redis transport, an unacknowledged message is redelivered once the visibility_timeout in broker_transport_options expires. If a task runs longer than that timeout, a second worker picks it up while the first is still working. ARQ also leaves a job queued until it completes or fails, so a worker killed mid-job, or a job that exceeds its timeout after doing partial work, will be retried. Either way, if the job charges a card or sends an email, the customer can see it twice.

The fix is to make the side effect idempotent and key it on something stable, such as the job ID or a natural key like order_id plus the action:

• Record completion with a unique constraint, or with Redis SET NX, and check that record before acting.
• Pass the same key to payment providers that accept idempotency keys.

Redis is only as durable as its configuration, and this applies equally to ARQ and Celery:

• Persistence: RDB snapshots and AOF fsync settings determine how much data a crash loses.
• Eviction: a maxmemory-policy other than noeviction can silently delete queue keys under memory pressure.
• Failover: replication is asynchronous, so a promoted replica may be missing recently enqueued jobs.

Finally, skip result storage unless a caller actually reads results. Use ignore_result in Celery, or a short or zero keep_result in ARQ. Stored results that no one reads accumulate in the same Redis instance that holds your queue.

Copyimport redis
from celery import Celery

app = Celery("billing", broker="redis://localhost:6379/0")
app.conf.task_acks_late = True  # ack only after the task finishes
# Must exceed the longest expected runtime, or Redis redelivers mid-run
app.conf.broker_transport_options = {"visibility_timeout": 3600}
app.conf.task_ignore_result = True  # nobody reads results; keep Redis lean

# Use a separate DB for idempotency records, not the queue DB
store = redis.Redis.from_url("redis://localhost:6379/1")


def provider_charge(amount_cents, idempotency_key):
    """Placeholder for a payment API call that accepts an idempotency key."""


@app.task(bind=True, max_retries=5)
def charge_order(self, order_id, amount_cents):
    # Natural key: same order + same action = same key across redeliveries
    key = f"idem:charge:{order_id}"
    if store.exists(f"{key}:done"):
        return  # a previous delivery already completed the charge

    # Short-lived lock stops a concurrent redelivery from running in parallel
    if not store.set(f"{key}:lock", self.request.id, nx=True, ex=600):
        raise self.retry(countdown=60)

    try:
        # Provider dedupes on the key if we crash after charging
        # but before recording completion below
        provider_charge(amount_cents, idempotency_key=key)
        store.set(f"{key}:done", "1", ex=7 * 24 * 3600)
    finally:
        store.delete(f"{key}:lock")

A Celery task with late acknowledgement that uses a Redis SET NX lock, a completion marker keyed on order_id, and a provider idempotency key so a redelivered job does not charge the customer twice.

Implementing jobs safely: pass IDs, not objects or request-scoped resources

Take one job, sending a receipt after an order is placed, and implement it all three ways. In each case the argument should be order_id, not the Order instance. The job should load the row itself, so it acts on current state rather than a snapshot taken at request time.

FastAPI BackgroundTasks. The task runs after the response is sent, in the same process. That makes it tempting to pass the request's db session or HTTP client straight in. Don't. Those resources come from dependencies whose cleanup is tied to the request lifecycle, and you should not assume they are still open when the task runs. Open a new session inside the task function and close it in a finally block.

ARQ. The function receives ctx as its first argument, followed by your arguments. Arguments are pickled into Redis by default. Pickle will accept an ORM object, but you get a detached copy of stale data that is tied to your model's import path. Create long-lived clients, such as an HTTP client or an engine, once in the worker's on_startup hook and store them on ctx. Open a fresh session per job from that engine.

Celery. The default JSON serializer rejects ORM objects outright, which is a useful guardrail. Keep it. Open a session at the start of the task. Do not share connections that were created before the worker forked.

Large payloads such as file contents or rendered reports also belong outside the broker. Store them in object storage or a table, and enqueue a reference to them. Broker messages should stay small enough to retry cheaply.

Copy# Assumes Order, get_db and create_order are defined elsewhere.
import os, httpx
from celery import Celery
from fastapi import BackgroundTasks, Depends, FastAPI
from sqlalchemy import create_engine
from sqlalchemy.orm import Session, sessionmaker
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine

# Engines connect lazily, so no connection exists before a Celery worker forks.
SessionLocal = sessionmaker(create_engine("postgresql+psycopg://app:change-me@db/shop"))
app = FastAPI()
celery_app = Celery("shop", broker="redis://redis:6379/0")  # JSON serializer by default

def send_receipt(order_id: int) -> None:
    db = SessionLocal()  # the task owns this session; the request's session may be closed
    try:
        order = db.get(Order, order_id)  # load current state, not a request-time snapshot
        ...  # render and send the receipt
    finally:
        db.close()

@app.post("/orders")
def place_order(bg: BackgroundTasks, db: Session = Depends(get_db)):
    order = create_order(db)
    bg.add_task(send_receipt, order.id)  # pass the ID, never `order` or `db`
    return {"id": order.id}

async def startup(ctx):  # ARQ: build long-lived clients once per worker
    engine = create_async_engine("postgresql+asyncpg://app:change-me@db/shop")
    ctx["Session"] = async_sessionmaker(engine)
    ctx["http"] = httpx.AsyncClient(timeout=10)

async def send_receipt_arq(ctx, order_id: int):
    async with ctx["Session"]() as session:  # fresh session per job
        order = await session.get(Order, order_id)
        await ctx["http"].post(os.environ["MAILER_URL"], json={"order_id": order.id})

class WorkerSettings:
    functions = [send_receipt_arq]
    on_startup = startup

@celery_app.task
def send_receipt_celery(order_id: int):  # an int serializes as JSON; an ORM object would not
    send_receipt(order_id)  # session opened inside the task, after the fork

The same receipt job written as a FastAPI BackgroundTask, an ARQ function and a Celery task: each one takes only order_id and opens its own database session instead of reusing the request-scoped one.

Operational surface area and impact on development cost

Each option adds a different set of things you have to run, watch and debug.

Celery has the heaviest configuration. Several defaults affect correctness, and it is not obvious which ones until something goes wrong. Acknowledgement timing, prefetch behaviour and the Redis visibility timeout together decide whether a crashed or slow job is lost, run twice or held by a busy worker. When something breaks, you have to work out whether the cause is the broker, the worker pool or the serializer. ARQ is small enough to read in an afternoon. The cost is that you build the missing pieces yourself: a dashboard, alerts on stale jobs and a dead-letter process.

Any separate worker fleet has a version-skew risk. During a rolling deploy, new web pods can enqueue jobs with new arguments while old workers are still consuming. Old pods can also enqueue jobs that new workers no longer accept. To reduce this risk:

• Add new parameters only as optional, with defaults.
• Deploy workers before producers.
• Give a task a new name when you change its signature in a breaking way.
• Keep the old handler until the queue has drained.

A migration path that saves engineering time: start behind a thin enqueue interface

Most teams don't need a separate queue on day one. They do need to avoid scattering background_tasks.add_task(...) across dozens of endpoints. The cheap fix is a single module that exposes one function, such as enqueue(job_name, **kwargs). Route handlers call that function and never import FastAPI's BackgroundTasks, ARQ or Celery directly. At first, the implementation looks up a registered handler and passes it to BackgroundTasks. Later, the same function can call ARQ's enqueue_job or Celery's apply_async, and no endpoint changes.

The abstraction only works if the jobs already behave like queued jobs. Enforce these constraints from the start:

• JSON-serializable arguments only. Pass IDs, strings and numbers. Reject ORM instances, open sessions and request objects at the interface, even though in-process execution would accept them.
• Idempotent handlers. Assume every job may run twice, because after the switch some will.
• Stable job names. Register handlers by string name, so the name becomes the contract a worker resolves.

With these in place, migrating is a backend change plus deploying workers. You don't have to audit every call site.

Watch for these signals that it's time to switch:

• Jobs lost during deploys or restarts are causing real incidents.
• You need retries, scheduling or delayed execution.
• CPU-heavy jobs are adding latency to API responses.
• Job load needs to scale independently of web traffic.

This approach avoids two expensive mistakes. You don't run a broker, workers and monitoring for a workload that doesn't need them yet. And when the workload does outgrow in-process tasks, you change one module instead of rewriting every endpoint.

Alternatives to consider

The three options above are not the only choices. Several other approaches fit specific constraints better.

RQ and Dramatiq

RQ is a Redis-backed queue built around synchronous functions and forked worker processes. It has fewer features than Celery and is easier to reason about. It suits teams whose job code is mostly blocking I/O or CPU work and who don't need asyncio. Dramatiq is in a similar position. It defaults to acknowledging a message only after the task completes, and it supports RabbitMQ as well as Redis. Both are reasonable middle ground when ARQ's async-only model doesn't match your code but Celery's configuration surface feels excessive.

A Postgres table as the queue

If you already run Postgres, a jobs table that workers poll with SELECT ... FOR UPDATE SKIP LOCKED avoids adding Redis or a broker to your infrastructure. The main advantage is transactional enqueue. The job row commits in the same transaction as the business data, so you never enqueue work for a record that rolled back, and you never commit a record whose job was lost. The cost is extra write and polling load on your primary database. That load becomes a concern at high job volume.

Workflow engines

When a job is really a multi-step process with retries per step, timers lasting days, and compensation logic, a durable workflow engine models it directly. Building that state machine on top of a plain task queue usually turns into hand-rolled persistence and is hard to debug.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to fastapi background tasks vs celery, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.