What structured concurrency means for a request handler

Structured concurrency is one rule applied to async code: concurrent work has a lexical scope. Child tasks started inside a block cannot outlive that block. When execution leaves the block, every child has finished, failed, or been cancelled. If a child fails, its exception reaches the parent instead of disappearing into a task object nobody awaits.

A request handler is a natural scope. A typical endpoint might fetch a user profile, query a pricing service, and check a feature-flag backend at the same time, then combine the results. Without structure, a few problems recur:

  • Leaked background work. One call fails and the handler returns a 500, but the other two coroutines keep running. They hold connections, consume upstream quota, and may write to logs or databases after the client has already received a response.
  • Hidden exceptions. A task created with asyncio.create_task and never awaited can fail without anyone noticing. You might see only a "Task exception was never retrieved" warning, long after the failure and far from its cause.
  • No single deadline. Each call gets its own timeout, so the request's total latency is the sum of guesses rather than one enforced limit.

An asyncio TaskGroup combined with an asyncio timeout gives the handler one boundary that owns its children, their errors, and their time budget. This matters most to backend and platform engineers running async Python services, often FastAPI, where every request fans out to several backends and slow or failing dependencies are normal.

  client request
        |
        v
+--------------------------------------------------------------------+
| request handler                                                    |
|  +--------------------------------------------------------------+  |
|  | async with asyncio.timeout(budget):  one request deadline    |  |
|  |  +--------------------------------------------------------+  |  |
|  |  | async with asyncio.TaskGroup() as tg:                  |  |  |
|  |  |                                                        |  |  |
|  |  |  +--------------+   +--------------+   +--------------+|  |  |
|  |  |  | fetch_profile|   | get_pricing  |   | check_flags  ||  |  |
|  |  |  +--------------+   +--------------+   +--------------+|  |  |
|  |  |         |                  |                  |        |  |  |
|  |  |         v                  v                  v        |  |  |
|  |  |    profile API        pricing API         flags API    |  |  |
|  |  |                                                        |  |  |
|  |  | child fails -> siblings cancelled -> ExceptionGroup    |  |  |
|  |  +--------------------------------------------------------+  |  |
|  | deadline hit -> TaskGroup cancelled -> TimeoutError          |  |
|  +--------------------------------------------------------------+  |
| exit: every child done, failed or cancelled; nothing leaks         |
+--------------------------------------------------------------------+

A request handler's timeout scope wraps a TaskGroup that owns its three backend calls, so a failure or missed deadline cancels the remaining children and surfaces as an exception in the parent before the block exits.

How asyncio TaskGroup works, and how it compares to gather and create_task

asyncio.TaskGroup is an async context manager that owns every task created through tg.create_task(). The block cannot exit until all of those tasks have finished. Task lifetime is therefore bounded by lexical scope: when the async with block ends, no child is still running.

Failure handling follows the same rule. If any child raises an exception other than CancelledError, the group cancels the remaining children and waits for them to unwind. It then raises every collected failure together as an ExceptionGroup. The same cleanup happens if the body of the async with block raises or if the enclosing task is cancelled. A failed downstream call cannot leave its siblings consuming connections after the handler has given up.

The older tools each lack one of these guarantees:

Approach Task lifetime When one task fails Error visibility
TaskGroup Bounded by the async with block Siblings cancelled and awaited All failures raised as ExceptionGroup
gather() Unbounded after the first error First exception propagates, siblings keep running First error only, later ones lost
gather(return_exceptions=True) Until every task finishes Nothing is cancelled Exceptions returned as values, so you must check each result
Bare create_task() Unowned No effect on anything else Logged as "never retrieved" only when the task is collected

Bare tasks have an additional hazard. The event loop keeps only a weak reference to each task, so a task you don't store elsewhere can be garbage-collected before it finishes.

TaskGroup requires Python 3.11 or later. Services on older runtimes still use gather with wait_for, plus explicit cancellation in finally blocks, or a backport.

import asyncio

# Requires Python 3.11+ (TaskGroup and except*).

async def fetch(name: str, delay: float, fail: bool = False) -> str:
    try:
        await asyncio.sleep(delay)  # stand-in for a downstream HTTP call
        if fail:
            raise ConnectionError(f"{name}: upstream returned 503")
        return f"{name}: ok"
    except asyncio.CancelledError:
        print(f"{name}: cancelled by the group")
        raise  # always re-raise so the group can finish unwinding


async def main() -> None:
    try:
        async with asyncio.TaskGroup() as tg:
            profile = tg.create_task(fetch("profile", 0.1))
            orders = tg.create_task(fetch("orders", 0.2, fail=True))
            history = tg.create_task(fetch("history", 5.0))  # slow sibling
        # Reached only if every child succeeded; all tasks are done here.
        print(profile.result(), orders.result(), history.result())
    except* ConnectionError as eg:
        # By now 'history' has been cancelled and awaited, so nothing
        # is still holding a connection after the handler gives up.
        for exc in eg.exceptions:
            print("failed:", exc)


asyncio.run(main())

When the 'orders' call fails, the TaskGroup cancels the slow 'history' sibling, waits for it to unwind, and raises the failure as an ExceptionGroup that the code handles with except.*

Handling ExceptionGroup with except* without breaking cancellation

When a child task fails, TaskGroup cancels its siblings, waits for them, and raises an ExceptionGroup that holds every non-cancellation failure. A handler that catches only TimeoutError will not see it, because the group wraps it. except* matches the leaves inside the group, so an except* TimeoutError clause can map to 504 and an except* ValidationError clause (raised when a backend's response fails to parse) can map to 502.

Map failures to a single status

Several except* clauses can run for one group. If more than one of them raises HTTPException, Python combines those exceptions into a new group. Also, return is not allowed inside except*. A cleaner pattern is to record a status in each clause and raise once after the try. Put the precedence rule in that one place, for example 504 wins over 502 because a timed-out upstream says more about the outage.

Don't swallow cancellation

CancelledError derives from BaseException, so except Exception and except* Exception leave it alone. A bare except:, except BaseException, or except* BaseException will catch it. If you catch it and don't re-raise, you break the machinery: asyncio.timeout can no longer turn the cancellation into TimeoutError, and a cancelled request keeps running. Catch the narrowest type you can handle. If you must catch something broad, re-raise anything you did not intend to handle.

Existing error handlers need an unwrapping step

FastAPI exception handlers and most middleware dispatch on the exception's type, and an ExceptionGroup matches none of them. Add an unwrapping step at the boundary. When the group has exactly one leaf, re-raise that leaf using from the group so handlers that expect one exception type still match. When it has several, log the whole group before you choose one status, or the sibling failures disappear from your telemetry.

import asyncio
import httpx
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, ValidationError

app = FastAPI()
client = httpx.AsyncClient(base_url="")

class Profile(BaseModel):
    id: int
    name: str

class Orders(BaseModel):
    count: int

async def fetch(path: str, model: type[BaseModel]) -> BaseModel:
    async with asyncio.timeout(2.0):  # per-upstream deadline, raises TimeoutError
        resp = await client.get(path)
    return model.model_validate_json(resp.content)  # bad payload raises ValidationError

@app.get("/users/{user_id}/summary")
async def summary(user_id: int):
    failures: set[int] = set()
    try:
        async with asyncio.TaskGroup() as tg:
            p = tg.create_task(fetch(f"/profiles/{user_id}", Profile))
            o = tg.create_task(fetch(f"/orders/{user_id}", Orders))
    except* TimeoutError:
        failures.add(504)  # only record here, because several except* clauses can run
    except* ValidationError:
        failures.add(502)
    # CancelledError is a BaseException, so neither clause matches it and it propagates.
    # Any other unmatched leaf is re-raised in an ExceptionGroup for the boundary unwrapper.
    if failures:
        status = 504 if 504 in failures else 502  # single precedence rule
        raise HTTPException(status_code=status, detail="upstream failure")
    return {"profile": p.result(), "orders": o.result().count}

A FastAPI handler that runs two upstream calls in a TaskGroup, records a status in each except clause, and then raises one HTTPException after the try block, with 504 taking precedence over 502 and cancellation left to propagate.*

One deadline per request with asyncio.timeout and deadline budgeting

Wrap the whole fan-out, not each call, in asyncio.timeout(). When the delay expires, the context manager cancels the current task. That cancellation propagates into any TaskGroup inside the block, which cancels every child. When control leaves the block, the internal CancelledError is converted into TimeoutError. Catch TimeoutError outside the async with, because that is the only place it appears. Code inside the block only sees cancellation.

Per-call asyncio.wait_for timeouts look safe, but they add up. Each call gets its full allowance no matter how much time the request has already used.

Per-call wait_for One outer asyncio.timeout
Worst-case latency Sum of all timeouts and retries The deadline
Knows the time already spent No Yes, from deadline - loop.time()
What gets cancelled One await The whole fan-out
Behaviour when the gateway gives up The server keeps working Work stops at or before the deadline

Illustrative estimate (assumed numbers, not a benchmark). Assume a gateway timeout of 30 s, 3 sequential downstream calls, a 8 s wait_for on each, and 1 retry per call.

worst case = calls × attempts × per-call timeout = 3 × 2 × 8 = 48 s

In that worst case the client receives a 504 at 30 s. The server then keeps spending connections and CPU on a response nobody will read for another 18 s.

To budget, record deadline = loop.time() + budget once at request entry, where loop comes from asyncio.get_running_loop(). Use asyncio.timeout_at(deadline) for the outer block. Before each downstream call, compute remaining = deadline - loop.time(), subtract a small margin for serialization and network, and pass the result as that client's timeout. Also forward it to the next service in a request header so that service can enforce the same budget.

import asyncio

BUDGET_S = 25.0  # stay under an assumed 30 s gateway timeout
MARGIN_S = 0.2   # headroom for serialization and network

async def call_service(name: str, timeout_s: float) -> str:
    # Stand-in for a real client call that accepts a timeout.
    # Forward the budget so the next hop can enforce it too.
    headers = {"X-Timeout-Ms": str(int(timeout_s * 1000))}  # placeholder header name
    await asyncio.sleep(0.5)
    return f"{name}: ok, allowed {headers['X-Timeout-Ms']} ms"

def remaining(loop: asyncio.AbstractEventLoop, deadline: float) -> float:
    left = deadline - loop.time() - MARGIN_S
    if left <= 0:
        raise TimeoutError("request budget exhausted")
    return left

async def handle_request() -> list[str]:
    loop = asyncio.get_running_loop()
    deadline = loop.time() + BUDGET_S  # recorded once, at request entry
    try:
        # One outer deadline for the whole fan-out.
        async with asyncio.timeout_at(deadline):
            profile = await call_service("profile", remaining(loop, deadline))
            # Each child gets only what is left, not a fresh full allowance.
            async with asyncio.TaskGroup() as tg:
                orders = tg.create_task(call_service("orders", remaining(loop, deadline)))
                recs = tg.create_task(call_service("recs", remaining(loop, deadline)))
            return [profile, orders.result(), recs.result()]
    except TimeoutError:
        # Only seen here, outside the block. Inside, children were cancelled.
        return ["504: deadline exceeded"]

if __name__ == "__main__":
    print(asyncio.run(handle_request()))

A single request-level deadline enforced with asyncio.timeout_at around a TaskGroup fan-out. Each downstream call receives the remaining budget minus a safety margin, and that budget is forwarded to the next hop.

Writing cancellation-safe code and knowing what cancellation cannot stop

Cancellation arrives as a CancelledError raised at whatever await the task is currently suspended on. Cleanup therefore belongs in try/finally blocks or async with context managers, so that connections return to the pool and locks are released no matter where the error lands. If you catch CancelledError to log or roll back, re-raise it. A swallowed CancelledError makes the task look like it finished normally. Then asyncio.timeout cannot convert the cancellation into TimeoutError, and a TaskGroup waits on a sibling that will never stop. Catching a bare BaseException causes the same problem.

Cancellation is also cooperative. It only takes effect at an await. A CPU-bound loop, a blocking driver call, or time.sleep keeps the event loop busy until it returns. Work handed to run_in_executor or asyncio.to_thread runs on a thread that cannot be interrupted. The awaiting coroutine is cancelled, but the thread keeps running and keeps its worker slot. The deadline protects the response, not the resources. Under sustained timeouts the executor can fill with abandoned work.

The same applies across the network. Cancelling an outbound HTTP call closes your side of the connection. The downstream service may still commit the write. A retry after a timeout can apply the operation twice unless the endpoint is idempotent, for example through an idempotency key the server deduplicates on.

Finally, a TaskGroup gives no limit on how many tasks run at once. Creating one task per item in an unbounded list can exhaust a connection pool or overload a dependency. Wrap each task's I/O in an asyncio.Semaphore sized to the pool or to the downstream service's capacity.

Implementing it in FastAPI handlers and dependencies

The request handler

Declare the endpoint async def. Inside it, open one asyncio.timeout() block for the whole request and put an asyncio.TaskGroup inside that block. Each backend call becomes a child task. When the deadline expires, the timeout cancels the handler's task. The TaskGroup then cancels every child that is still running and waits for all of them to finish. Outside the block you get a single TimeoutError, with no orphaned coroutines still writing to sockets.

Create the semaphore once, in the lifespan, and store it on app.state. A semaphore created per request only limits concurrency inside that request. A hundred concurrent requests could still open a hundred times as many connections.

Map failures to responses at the boundary. Catch TimeoutError outside the timeout block and raise HTTPException with status 504. Use except* to catch backend errors and translate them to 502. If only some calls are optional, catch their errors inside those child tasks so they cannot cancel their siblings.

Work that must outlive the request

Fire-and-forget calls to create_task from a handler bring back the problem the TaskGroup solved. For durable work such as emails or billing events, enqueue a job in a task queue with retries. For in-process loops such as cache refreshers, open a TaskGroup in the lifespan context manager. On shutdown, signal those loops to stop or cancel them, because the group waits for every task before it exits.

The trade-off is that long-lived tasks lose the request's deadline, its cancellation and its error path. Each one needs its own exception handling, logging, metrics and a shutdown timeout. A crash in one child cancels every other task in the lifespan group.

When to avoid it, alternatives, and the impact on cost and engineering time

That fail-fast default is also the main reason not to use a TaskGroup everywhere. If a product page can render without its recommendations panel, cancelling the price and inventory lookups because recommendations raised throws away work the user needed. When partial results are acceptable, catch and log expected errors inside each child coroutine and return a sentinel, so the group only sees failures that really should abort the request. Alternatively, use gather with return_exceptions=True and inspect each result. Both keep the shared deadline from asyncio.timeout while deciding failure per task instead of per group.

Adoption costs

  • Runtime version. You need a Python runtime recent enough to include TaskGroup, ExceptionGroup and asyncio.timeout. That can block services pinned to older interpreters or base images.
  • ExceptionGroup handling. Error handlers, middleware and error-reporting hooks that match on a single exception type will miss wrapped failures until they unwrap groups with except*.
  • Broad catches. Audit every except BaseException and every bare except:. Code that swallows CancelledError silently stops cancellation from propagating.

Where it saves time

The savings show up in incidents you no longer debug: fewer orphaned tasks writing after the response was sent, and fewer requests hanging past their deadline. The behaviour also becomes testable. In the test, one child raises and another sleeps forever, and the handler runs under a short timeout. The test then asserts three things:

  • the sleeping sibling received CancelledError
  • its cleanup released the connection
  • the handler returned before the deadline

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to asyncio taskgroup, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.