FastAPI Background Tasks vs Task Queues: Where Each Breaks Down

How FastAPI BackgroundTasks actually executes

BackgroundTasks is a deferred function call attached to a response. You declare a parameter of type BackgroundTasks on your endpoint, call background_tasks.add_task(send_receipt, order_id, email) with the function and its plain arguments, and return. FastAPI hands the collected tasks to the response object (the mechanism comes from Starlette). After the response body has been sent to the client, the tasks run one after another, in the order they were added.

They run in the same worker process that served the request. Execution depends on how the function is defined. A plain def function is dispatched to the threadpool so it doesn't block the event loop. An async def function is awaited directly on the loop, which means any blocking call inside it, such as a synchronous HTTP client or a CPU-heavy loop, stalls every other request that worker is handling.

This is not a job system. Nothing is persisted, nothing is retried, and nothing outside the process knows the task exists. Throughout this article, the line between BackgroundTasks and a task queue is drawn by one question: what happens to the work if the process dies before it finishes?

Teams reach for it first for good reasons:

• No extra infrastructure. There is no broker, no worker deployment, and no serialization format to agree on.
• Near-zero start latency. The task begins as soon as the response is sent, with no polling interval or queue hop in between.
• Easy testing. With the test client, background tasks finish before the call returns, so assertions can check their side effects directly.

Copyfrom fastapi import BackgroundTasks, FastAPI
from fastapi.testclient import TestClient

app = FastAPI()
sent: list[tuple[int, str]] = []  # stands in for an email provider


def send_receipt(order_id: int, email: str) -> None:
    # Plain def: dispatched to the threadpool after the response is sent.
    # If the worker process dies before this runs, the work is simply lost.
    sent.append((order_id, email))


async def audit_log(order_id: int) -> None:
    # async def: awaited on the event loop. Any blocking call in here
    # stalls every other request this worker is handling.
    print(f"order {order_id} checked out")


@app.post("/orders/{order_id}/checkout")
async def checkout(order_id: int, email: str, background_tasks: BackgroundTasks):
    # Tasks run sequentially, in the order added, after the response is sent.
    background_tasks.add_task(send_receipt, order_id, email)
    background_tasks.add_task(audit_log, order_id)
    return {"status": "accepted", "order_id": order_id}


def test_checkout_sends_receipt() -> None:
    client = TestClient(app)
    resp = client.post("/orders/42/checkout", params={"email": "buyer@example.com"})
    assert resp.status_code == 200
    # With the test client, background tasks have finished by the time
    # post() returns, so their side effects can be asserted directly.
    assert sent == [(42, "buyer@example.com")]

A checkout endpoint that queues a sync and an async background task with add_task, plus a TestClient test showing the tasks have completed by the time the request returns.

Why deferred work is used at all

Both BackgroundTasks and task queues solve the same basic problem. Some of the work an endpoint triggers does not need to finish before the client gets an answer. Sending a receipt email, writing an audit record, invalidating a cache, or calling a slow third-party API are side effects of the request, not part of its result. If the handler does them inline, the client waits for every one of them. A slow or failing downstream service then also becomes a slow or failing endpoint.

Moving that work out of the response path does three things:

• It shortens response time. The client gets its answer once the essential work is done, not after every side effect has finished.
• It decouples the endpoint from downstream latency. A slow email provider delays the email, not the HTTP response.
• It keeps the response honest about what matters. The status code reflects the core operation, such as "the order was recorded", instead of mixing in the outcome of optional follow-ups.

BackgroundTasks is the cheapest way to get these benefits in a FastAPI app. A task queue gets the same benefits and adds guarantees about whether the deferred work actually happens. The rest of this article is about when those guarantees are worth their cost.

Where background tasks sit in the request lifecycle

When a path operation returns, FastAPI builds the response, and Starlette attaches the queued tasks to it. Starlette sends the status line, headers, and body, and only then runs the tasks in order. From the client's side the request is finished. From the server's side the ASGI call is still open. The worker process still holds that work. Async tasks run on its event loop, and sync tasks occupy a slot in its threadpool. A load balancer that counts open requests or a graceful shutdown that waits for in-flight work may see the request differently than the client does.

Dependencies that use yield make this riskier. Their cleanup code closes database sessions, returns connections to the pool, and releases locks. Depending on your FastAPI version, that cleanup may have already run by the time the task starts. A task that receives the request's session or an ORM object loaded through it can fail in several ways. It might hit a closed session. It might trigger a lazy load on a detached instance. It might write through a pooled connection that another request has already taken.

Before and after

The fragile pattern passes the request's session or a loaded model instance into add_task. The robust pattern passes primary keys and plain values such as strings, numbers, and small dicts. The task then opens its own session from the session factory, re-reads what it needs, and closes the session when done. The task owns its resources, so it no longer depends on when the request's cleanup happens.

Failures need separate handling. By the time a task raises, the client has already received its response, often a 200. The exception reaches only your server logs. No retry happens, and nothing alerts anyone unless you have set that up yourself.

Copyfrom fastapi import BackgroundTasks, Depends, FastAPI, HTTPException
from sqlalchemy import create_engine
from sqlalchemy.orm import Session, sessionmaker

from myapp.models import Order  # your ORM model

engine = create_engine("postgresql+psycopg://app:change-me@db.internal:5432/shop")
SessionLocal = sessionmaker(bind=engine)
app = FastAPI()

def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()  # depending on version, this can run before the task starts

def send_receipt_fragile(db: Session, order: Order) -> None:
    order.receipt_sent = True  # may hit a closed session or a detached instance
    db.commit()

def send_receipt(order_id: int, email: str) -> None:
    with SessionLocal() as db:  # fresh session, closed when the block exits
        order = db.get(Order, order_id)  # re-read current state
        if order is None:
            return
        # ... send the receipt to `email` ...
        order.receipt_sent = True
        db.commit()
    # Any exception here only reaches server logs; the client already got its 200.

@app.post("/orders/{order_id}/checkout")
def checkout(order_id: int, tasks: BackgroundTasks, db: Session = Depends(get_db)):
    order = db.get(Order, order_id)
    if order is None:
        raise HTTPException(status_code=404)
    # tasks.add_task(send_receipt_fragile, db, order)  # fragile
    tasks.add_task(send_receipt, order.id, order.customer_email)
    return {"status": "accepted"}

Compares a background task that borrows the request's session and an ORM object with one that takes an ID and an email string and opens its own session from the session factory.

Where BackgroundTasks breaks down

The deeper problem is durability. Tasks live only in the worker process's memory. A crash, an OOM kill, or a SIGTERM during a deploy or scale-down loses every in-flight and queued task, and nothing records that they existed. You cannot replay what was never written down.

Graceful shutdown helps less than it appears to. The server waits for running work only up to its shutdown timeout. The container orchestrator then sends SIGKILL once the termination grace period expires; Kubernetes, for example, defaults to 30 seconds. Together these set a hard ceiling on safe task duration. In an autoscaled service that deploys several times a day, workers terminate often, so every task longer than that window is at risk on each rollout.

What is missing

• Retries and backoff for transient failures
• Scheduling for delayed or periodic work
• Result tracking, so callers can check status
• Dead-lettering for tasks that keep failing

You can build each of these yourself. Once you add persistence so retries survive restarts, you have rebuilt a task queue, and usually a weaker one.

Resource contention

Tasks run in the same process as your endpoints. A blocking call inside an async def task stalls the event loop for every request. Heavy def tasks run in the threadpool that sync endpoints also use, so a burst of them can starve request handling.

asyncio.create_task and bare threads

These are not a cheaper middle ground. They have the same durability gap and add new problems. The event loop holds only weak references to tasks, so a task you do not keep a reference to can be garbage-collected before it finishes. Exceptions raised in an unawaited task or a bare thread are easy to lose without anyone noticing.

What a task queue adds, and how it works

A task queue moves the work out of the process that accepted the request. The handler serializes a job description, typically a task name plus arguments, and sends it to a broker that persists it. Separate worker processes pull jobs from the broker and run them. The response no longer waits on the work, and the work no longer depends on the web process staying alive.

CopyBackgroundTasks:
  client -> [web worker: handler -> response -> task]
                         process dies here = task lost

Task queue:
  client -> [web process] --enqueue--> [broker: persisted job]
                                              |
                                              v
                                  [worker pool: run -> ack]
            web dies: job already persisted, survives
            worker dies before ack: job redelivered

What the broker-and-worker split gives you:

• Acknowledgement semantics. If a worker acknowledges a job only after finishing it, a crash mid-job leaves the job unacknowledged, and the broker hands it out again. This gives at-least-once delivery, so tasks must be idempotent.
• Retries with backoff. A failed job can be re-enqueued with increasing delays. A flaky downstream service then gets time to recover instead of being hit by a burst of immediate retries.
• Scheduling. Jobs can carry a delay or a target run time, and a scheduler process can enqueue recurring jobs.
• Per-queue concurrency limits. You can route work to named queues and cap how many workers consume each one. This protects rate-limited APIs and keeps slow jobs from starving fast ones.
• Result and status tracking. A result store records each job's state, so clients can poll by job ID.
• Independent scaling. You add workers to match queue depth without touching web capacity, and the reverse.

Copy(A) BackgroundTasks: the task runs inside the web worker process
+--------+  request   +------------------------------------------+
| client |----------->| web worker process                       |
|        |<-----------|  handler --> response --> task (memory)  |
+--------+  response  +------------------------------------------+
                         X process dies after response: task LOST

(B) Task queue: web process -> broker -> worker pool
+--------+  request   +-------------+  enqueue  +-------------------+
| client |----------->| web process |---------->| broker            |
|        |<-----------| (handler)   |           |  persisted jobs   |
+--------+   job ID   +-------------+  delayed/ |  queue: default   |
    |                 +-------------+  recurring|  queue: slow      |
    |                 | scheduler   |---------->|                   |
    |                 +-------------+           +-------------------+
    |             X web dies after enqueue:        | deliver  ^ ack, or
    |               job SURVIVES in broker         v          | retry+backoff
    |                             +---------------------------+
    |                             | worker pool (cap/queue)   |
    |                             |  [w1] [w2] [w3] ... [wN]  |
    |                             +---------------------------+
    |                              X worker dies before ack:
    |                                job REDELIVERED
    |                                           |
    |                                           v
    |   poll status by job ID     +---------------------------+
    +---------------------------->| result store: job state   |
                                  +---------------------------+

With BackgroundTasks the task lives only in the web worker's memory and is lost if that process dies after responding. With a task queue the job is persisted in the broker before the response is sent, is redelivered if a worker dies before acknowledging it, and can be tracked by job ID.

What a task queue costs

Every guarantee above comes from new moving parts, and each one carries its own cost.

The broker is a new failure point. When enqueue fails inside a request, you have three options:

• Fail the request. This is honest, but the broker's uptime now limits your API's uptime.
• Fall back to in-process execution. This keeps the request alive but quietly loses durability.
• Use a transactional outbox. Write the job to your database in the same transaction as the business data, and let a relay publish it later. This is the most correct option, and it is one more component to run.

Delivery is at-least-once. A worker can crash after doing the work but before acknowledging it, so the job runs again. Every task must therefore be idempotent. Use an idempotency key checked against a dedup table, or a unique constraint that makes a second write a no-op.

Payloads must be serializable. You pass IDs and plain data, not ORM objects or open clients. That is better design. The cost is that porting existing BackgroundTasks code means rewriting each call site.

Deploys get harder. During a rolling deploy, old workers receive payloads from new web pods, and the reverse also happens. Payload schemas must stay backward-compatible: add fields as optional, and remove them only after both sides have shipped.

Daily overhead grows. Each job pays a network round-trip to enqueue, plus pickup delay before a worker starts it. You must alert on queue depth and worker health. Local development and CI also need a broker, either a real one or a faithful fake.

Decision rubric: which tool for which job

Run each job through five questions, in order:

1. Is losing this task on a deploy acceptable? BackgroundTasks run inside the worker process. A rolling restart, an OOM kill, or a crashed pod discards anything still pending. If the answer is no, you need a queue.
2. Does it take longer than a few seconds? Long tasks hold a worker slot and widen the window in which a restart can kill them. They also delay graceful shutdown.
3. Must it retry on failure? BackgroundTasks has no retry. You would have to build backoff, attempt limits and dead-lettering yourself, and that work is most of what a queue already provides.
4. Does anyone need to know its status? A user polling for "done" or an operator asking "did it run?" needs a task ID and a result store. Neither exists once the response has been sent.
5. Is it CPU-heavy? CPU-bound work in the same process competes with request handling. In an async app it can stall the event loop. Separate workers isolate that load.

The first question is the gate: a "no" sends the job to a queue. After that, any "yes" also points to a queue. A job that passes all five is the cheap case that BackgroundTasks handles well.

When to use each

Use BackgroundTasks when all of the following hold:

• The work is short, I/O-bound, and finishes well within your shutdown grace period.
• Losing it occasionally, for example during a deploy or a crash, causes no lasting harm, or the system recovers on its own, as with a cache that refills on the next miss.
• Nobody needs to check whether it ran, and a failure only needs to show up in logs.
• It does not need to run later than "right after this response" or on a schedule.

Use a task queue when any of these hold:

• The work has business consequences if it is lost: money moves, users wait for a message, or data must end up consistent.
• It must retry on transient failures, run at a specific time, or run periodically.
• It is long-running or CPU-heavy enough to compete with request handling.
• Clients or operators need to see its status.
• You need to cap concurrency against a rate-limited downstream service, or scale task capacity separately from web capacity.

A useful timing signal: the right moment to introduce a queue is when the first workload that fails the rubric appears, not before. Adding one earlier means paying for a broker, workers, and monitoring while nothing depends on their guarantees.

Who should use each

BackgroundTasks fits small teams and early-stage services that run a single FastAPI deployment and want to keep operations minimal. It also fits internal tools, prototypes, and any service whose deferred work is purely best-effort. If your team does not already operate a broker and has no workload that fails the rubric, BackgroundTasks is the sensible default.

A task queue fits teams whose deferred work carries real consequences: payments, transactional email, data pipelines, report generation. It also fits services that already run a broker for other purposes, since much of the operational cost is already paid. The team needs to be prepared to own the extra pieces. That means monitoring queue depth and worker health, writing idempotent tasks, and keeping payload schemas backward-compatible across deploys.

Many services end up using both. Best-effort side effects stay on BackgroundTasks, and the workloads that need durability go to the queue.

Alternatives beyond BackgroundTasks and a broker-backed queue

These two are not the only options. Depending on the workload, one of these may fit better:

• Doing the work inline. If the operation is fast and the client needs to know whether it succeeded, keep it in the request. Deferring work that the client depends on only hides failures.
• asyncio.create_task or bare threads. As covered above, these share the durability gap of BackgroundTasks and add the risk of lost references and swallowed exceptions. They rarely improve on BackgroundTasks for request-triggered work.
• A database-backed job table. Write jobs to a table in your existing database and have a worker process poll it, claiming rows with row-level locking. This gives durability and retries without a separate broker. It also composes naturally with the transactional outbox described below. The trade-off is extra polling load on the database and more queueing logic you maintain yourself.
• Scheduled batch jobs. For work that does not need to happen per request, such as nightly reports or periodic cleanup, a scheduled job run by cron or your orchestrator's scheduled-job feature can be simpler than a queue. The request records what needs doing, and the batch job processes it later.
• A managed queue service from your cloud provider. This removes the work of running a broker yourself. You still write workers, handle at-least-once delivery, and keep payloads backward-compatible.
• A separate service fed by events. When the deferred work belongs to a different domain, such as notifications or billing, publishing an event and letting another service own the processing can be cleaner than running that work in your API's own workers.

The middle ground and a migration path that saves engineering time

Most teams don't need to choose on day one. They need to avoid rewriting every endpoint the day the rubric tips toward a queue.

Put an interface between handlers and execution

Define a small enqueue protocol: one method that takes a task name and a serializable payload. The first implementation wraps BackgroundTasks and is provided through a FastAPI dependency. Handlers call enqueue("send_receipt", {"order_id": ...}) and never import a queue library. Later, a queue-backed implementation replaces the dependency, and the handlers stay as they are. Passing IDs instead of ORM objects or open sessions matters here. It is what makes the payload safe to cross a process boundary.

Close the commit-then-enqueue gap with an outbox

Even with a real queue, a handler can commit the order and then crash, or fail to reach the broker, before it enqueues the receipt. The transactional outbox pattern removes that gap. The handler inserts a row describing the task into an outbox table in the same transaction as the business write. Either both persist or neither does. A separate dispatcher polls the table, runs or forwards each pending row, and marks it done.

Migrate in steps

1. Keep BackgroundTasks, but route all calls through the interface.
2. Make each task idempotent, keyed on something like the order ID. The outbox and queues both deliver at least once, so retries will happen.
3. Add the outbox or a broker only for the task types the rubric flags.

Each step is small and can ship on its own. Infrastructure arrives only when a specific workload requires it.

Copyfrom typing import Any, Callable, Protocol

from fastapi import BackgroundTasks, Depends, FastAPI

app = FastAPI()

TASKS: dict[str, Callable[[dict[str, Any]], None]] = {}

def task(name: str):
    def register(fn):
        TASKS[name] = fn
        return fn
    return register

@task("send_receipt")
def send_receipt(payload: dict[str, Any]) -> None:
    ...  # Load the order by payload["order_id"]. Skip it if the receipt was already sent (idempotent).

class Enqueuer(Protocol):
    def enqueue(self, name: str, payload: dict[str, Any]) -> None: ...

class InProcessEnqueuer:
    """Step 1: runs tasks in this process after the response is sent."""
    def __init__(self, background: BackgroundTasks) -> None:
        self.background = background

    def enqueue(self, name: str, payload: dict[str, Any]) -> None:
        self.background.add_task(TASKS[name], payload)

def get_enqueuer(background: BackgroundTasks) -> Enqueuer:
    # Later, return an outbox- or broker-backed Enqueuer here. Handlers stay unchanged.
    return InProcessEnqueuer(background)

@app.post("/orders/{order_id}/pay")
def pay(order_id: int, enqueuer: Enqueuer = Depends(get_enqueuer)):
    # ... record the payment and commit ...
    enqueuer.enqueue("send_receipt", {"order_id": order_id})
    return {"status": "paid"}

A minimal enqueue protocol whose first implementation wraps FastAPI's BackgroundTasks and is injected through a dependency, so an outbox- or queue-backed implementation can replace get_enqueuer later without changing the handler.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to fastapi background tasks, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.