What is worker starvation and why do slow endpoints cause it?

Worker starvation happens when every worker in a fixed pool is occupied and new requests can only wait. It applies to teams running synchronous app servers with a fixed number of workers (processes or threads) behind a reverse proxy, where each in-flight request holds one worker until it finishes. A worker blocked on a slow query does no other work, so capacity is measured in worker-seconds, not requests.

Illustrative estimate

These numbers are assumed for illustration, not measured.

  • Pool: 32 workers, so 32 worker-seconds are available per second.
  • Traffic: 200 requests/second in total.
  • Fast endpoints: 97% of traffic (194 req/s) at about 50 ms each, which demands 194 × 0.05 ≈ 9.7 worker-seconds/s, or about 30% of the pool.
  • Slow endpoint: 3% of traffic (6 req/s) holding a worker for 20 s each, which demands 6 × 20 = 120 worker-seconds/s.

Total demand is about 130 worker-seconds/s against 32 available. On its own, the slow endpoint asks for almost four times the whole pool. At 6 arrivals per second, slow requests occupy all 32 workers in roughly 5 seconds (32 ÷ 6).

After that, a 50 ms request no longer waits 50 ms. It waits in the proxy or listen queue until a slow request releases a worker, and then competes with more slow arrivals for that worker. The queue keeps growing, so latency for healthy endpoints rises toward the proxy's own timeout.

The fast endpoints did nothing wrong. Without a cap on how long one request may hold a worker, 3% of traffic effectively takes capacity from the other 97%.

Who should use a layered timeout budget?

This approach is for teams that run a monolith on a synchronous app server with a fixed pool of workers, where one request occupies one worker from start to finish. It matters most when the same pool serves a mix of routes: fast reads alongside exports, reports, search, admin bulk operations or calls to slow third-party APIs. In that setup a few slow routes can starve everything else, as the estimate above shows.

It is most useful to:

  • Application engineers who own routes. They decide each route's budget and whether slow work belongs in a background job.
  • Platform or infrastructure engineers. They own the load balancer, proxy and app server settings and are responsible for keeping the layers in the right order.
  • On-call engineers. Per-layer timeout labels tell them where a request was stopped and why.

It matters less if your server handles requests on an event loop with no fixed per-request worker, or if every route has similar, predictable latency. Even then, statement and pool checkout timeouts still protect the database, and deadline propagation still bounds outbound calls.

Which layers can hold a request, and how should their timeouts be ordered?

Every layer between the user and the database can hold a request open, and each has its own timer. The goal is to set them so the innermost layer always gives up first. Inner layers can actually stop work. Outer layers can only stop waiting.

Layer What it bounds On expiry Stops backend work?
Client / CDN Total wait for a response Client error or retry No
Load balancer (idle/response) Idle connection or time to response Connection closed, 504 No
Reverse proxy upstream read Wait for upstream bytes 504 to caller No
App server worker timeout Wall time per worker Worker killed and restarted Yes, abruptly
In-app request deadline Handler's total budget Handler aborts at next check Yes, cooperatively
Outbound HTTP connect/read One remote call Exception in handler Local wait only
Pool checkout Wait for a DB connection Fast error, no query sent Yes
DB statement timeout One query's execution Query cancelled, transaction aborted Yes

The ordering rule is a strictly decreasing budget:

statement timeout < app deadline < worker timeout < proxy timeout < load balancer timeout

Pool checkout and outbound HTTP timeouts must fit inside the app deadline as well.

Consider what happens when this order is inverted. Say the proxy times out at 30 s while the worker's timeout is 60 s. The proxy returns a 504 and closes the connection. Nothing tells the worker. It keeps running its query and holding a database connection for up to another 30 s, and nobody will read the result.

Your edge metrics count one failed request. The real cost is a worker and a connection held for twice the time the user waited. If clients also retry the 504, each retry adds another orphaned worker, and this happens precisely when capacity is already short. Edge error rates then understate resource consumption, often by a wide margin.

  Client / CDN  (can only stop waiting; may retry the 504)
        |
        v
+--------------------------------------------------------------------------+
| Load balancer idle/response timeout                        stops waiting |
| +----------------------------------------------------------------------+ |
| | Reverse proxy upstream read timeout                    stops waiting | |
| | +------------------------------------------------------------------+ | |
| | | App server worker timeout                           kills worker | | |
| | | +--------------------------------------------------------------+ | | |
| | | | In-app request deadline                 aborts cooperatively | | | |
| | | | +-----------------------+  +-------------------------------+ | | | |
| | | | | Pool checkout timeout |  | Outbound HTTP connect/read    | | | | |
| | | | +-----------+-----------+  +-------------------------------+ | | | |
| | | |             |                                                | | | |
| | | |             v                                                | | | |
| | | | +---------------------------------+                          | | | |
| | | | | DB statement timeout (shortest) |                          | | | |
| | | | +---------------------------------+                          | | | |
| | | +--------------------------------------------------------------+ | | |
| | +------------------------------------------------------------------+ | |
| +----------------------------------------------------------------------+ |
+--------------------------------------------------------------------------+
 statement < app deadline < worker < proxy < load balancer  (inner first)

 Inverted (illustrative): proxy 30 s, worker 60 s
 proxy  +------ 30 s ------> 504 to client, connection closed
 worker +--------------- 60 s ---------------> still holds a DB connection
                           ^ orphaned work: nobody reads the result

Each timeout must sit inside the one that wraps it, so the inner layers that can actually stop work expire before the outer layers that can only stop waiting; inverting proxy and worker timeouts leaves an orphaned worker holding a database connection after the 504.

How does a layered timeout budget work?

The mechanism has three parts, and the following sections cover each one in detail.

  1. Nested limits. Every layer gets a timeout shorter than the layer outside it. When a request runs long, the innermost layer that can actually stop work fires first: the database cancels the query, the pool refuses a checkout, or the handler notices its deadline has passed. The app then returns a deliberate error and frees its worker. Outer layers such as the proxy and load balancer only act as a backstop when the inner ones fail, for example when a handler is stuck in code that never checks its deadline.
  2. One shrinking budget per request. On entry, the handler records an absolute deadline. Each downstream operation (pool checkout, query, outbound HTTP call) receives the smaller of its own cap and the time remaining. A sequence of calls therefore cannot add up to more than the request's budget, and work that cannot finish in time is never started.
  3. Containment for legitimately slow work. Routes that need more time than the shared budget allows are not given a bigger global timeout. They move to a separate worker pool or a background job, so they consume capacity that was sized for them and not capacity meant for fast routes.

Around these three parts sits observability. Every timeout is counted by layer and route, so you can see which limit fired and whether it protected the system or broke a valid request.

How do you set a database statement timeout safely?

Start from the database, because it is the only layer that can actually stop work it is doing. In PostgreSQL, set a conservative default on the application's role with ALTER ROLE app_web SET statement_timeout = '2s', rather than one generous value in the server config. The role setting applies only to that role's new sessions, so pooled connections keep the old value until they reconnect. Give batch workers, reporting jobs and migrations their own roles, or raise the limit for a single transaction with SET LOCAL statement_timeout. SET LOCAL reverts at commit or rollback, so a raised limit cannot leak into the next request on a reused connection. A plain SET can. Other databases have equivalent controls under different names; check your engine's documentation for scope and semantics.

statement_timeout limits each statement, not the request. Ten 1.9-second queries all pass a 2-second limit, so the request deadline in the next section still matters.

Know what cancellation does and does not do:

  • The application keeps running. The driver raises an error, and the request continues in whatever code catches it.
  • Writes roll back. In PostgreSQL the whole transaction is aborted. Every later statement fails until you issue a rollback, so never swallow the error and continue on that connection.
  • Retries need care. Retrying a timed-out query immediately repeats the load that caused the timeout. Return a clear error, or retry once with backoff only if the operation is idempotent.

Per-route overrides drift. Each raised limit should have an owner and a reason, and it should be reviewed when its query plan or data volume changes.

-- Conservative default for the web app's role. Applies to NEW sessions only;
-- pooled connections keep the old value until they reconnect.
ALTER ROLE app_web SET statement_timeout = '2s';

-- Long-running work gets its own role instead of raising the web default.
-- Owner: reporting team. Reason: nightly aggregates. Review on plan/volume change.
ALTER ROLE app_reports SET statement_timeout = '5min';

-- Known slow operation on the web role: raise the limit for ONE transaction.
BEGIN;
SET LOCAL statement_timeout = '30s';  -- reverts at COMMIT or ROLLBACK
SELECT customer_id, sum(amount)
FROM invoices
WHERE issued_at >= now() - interval '90 days'
GROUP BY customer_id;
COMMIT;

-- Avoid: a plain SET persists for the session and leaks into the next
-- request that reuses this pooled connection.
-- SET statement_timeout = '30s';

-- If a statement is cancelled inside a transaction, PostgreSQL aborts it.
-- Later statements fail with "current transaction is aborted" until you
-- roll back, so the application must issue:
-- ROLLBACK;

-- Verify the effective value for the current session.
SHOW statement_timeout;

Sets a 2-second statement_timeout default on the web role, gives reporting its own role, and raises the limit for a single transaction with SET LOCAL so a pooled connection does not carry it into the next request.

How do pool checkout timeouts and request deadlines protect workers?

Pool checkout is the timeout teams most often forget. When every database connection is busy, a worker asks the pool for one and blocks. Query logs and the statement timeout don't see this wait because no query has started yet. The worker sits idle and holds its slot. Set the checkout timeout well below the request budget. A saturated pool then produces a fast, countable error, such as a 503 with a metric, instead of a growing queue of stalled workers.

Propagate a deadline, not fixed per-call timeouts

Fixed per-call timeouts add up. Three outbound calls at 5 s each can hold a worker for 15 s against a 10 s request budget. Instead, compute one absolute deadline when the request arrives and pass it down:

  1. On entry, set deadline = now + request_budget.
  2. Before each call, compute remaining = deadline - now - margin. If remaining <= 0, fail immediately.
  3. Use min(per_call_cap, remaining) as that call's HTTP timeout, pool checkout timeout, or statement timeout.

Later calls automatically get less time, so the total can never exceed the budget.

Retries launch second copies

A client-side timeout stops the client from waiting. It does not stop the server's work. If a proxy or client retries on timeout, a second copy runs on another worker while the first is still executing. This doubles load at the moment the system is already slow. Allow retries at one layer only, cap them at one attempt, and disable proxy retries for non-safe methods like POST. Where retries are necessary, require idempotency keys so the server can detect and discard duplicates.

import queue, time, requests

REQUEST_BUDGET = 10.0  # seconds allowed for the whole request
MARGIN = 0.2           # reserve time to build and send an error response

class DeadlineExceeded(Exception):
    pass

class Deadline:
    def __init__(self, budget):
        self.expires_at = time.monotonic() + budget  # absolute, computed once on entry

    def timeout(self, per_call_cap):
        remaining = self.expires_at - time.monotonic() - MARGIN
        if remaining <= 0:
            raise DeadlineExceeded()  # fail fast; don't start work that can't finish
        return min(per_call_cap, remaining)

pool = queue.Queue()  # stand-in for a DB connection pool holding open connections

def handle_checkout(order_id, idempotency_key):
    dl = Deadline(REQUEST_BUDGET)
    try:
        conn = pool.get(timeout=dl.timeout(0.5))  # short checkout cap: saturation fails fast
    except queue.Empty:
        return 503, "db pool saturated"  # visible, countable; emit a metric here
    except DeadlineExceeded:
        return 504, "deadline exceeded"
    try:
        ms = int(dl.timeout(3.0) * 1000)
        conn.execute("SET LOCAL statement_timeout = %d" % ms)  # Postgres, inside a txn
        conn.execute("SELECT total FROM orders WHERE id = %s", (order_id,))
        # Later calls get whatever budget is left, never a fixed 5 s each.
        requests.post("",
                      json={"order_id": order_id},
                      headers={"Idempotency-Key": idempotency_key},  # lets server drop duplicates
                      timeout=dl.timeout(5.0))
        return 200, "ok"
    except (DeadlineExceeded, requests.Timeout):
        return 504, "deadline exceeded"  # no retry here: retries live at one layer only
    finally:
        pool.put(conn)

A per-request deadline computed once on entry supplies the pool checkout timeout, the Postgres statement timeout and the outbound HTTP timeout from the remaining budget, with an idempotency key on the non-safe POST.

When should you avoid hard timeouts, and what are the alternatives?

A tight global timeout is the wrong tool when the slow requests are legitimate. If p99 for a search endpoint sits just above your cutoff, that tail latency becomes a steady stream of 504s. Rare operations such as a year-end export or a bulk admin update fail every time, so the timeout encodes a policy nobody chose.

Hard worker kills cost even more. A process killed mid-request skips finally blocks and connection cleanup. It can leave a multi-step write half applied if that write was not wrapped in one transaction. The supervisor then starts a fresh process, which pays startup cost and arrives with cold in-process caches, so latency rises right when the system is already struggling.

Two alternatives contain slow work instead of cutting it off:

  • Separate worker pool. Route exports, reports and /admin paths at the proxy to their own pool, with its own size and longer budgets. Slow work can then exhaust only its own workers.
  • Background jobs. The endpoint enqueues a job and returns 202 with a job ID. The client then polls a status endpoint or receives a callback. The queue needs its own concurrency limit and per-job timeout, or you have only moved the starvation problem.
Approach Blast radius Capacity fragmentation API contract change Operational complexity
Global timeouts only Whole app None None Low
Per-route budgets Shared pool None None Low–medium
Separate worker pool Isolated pool Yes, idle capacity per pool None Medium
Background jobs Queue workers Yes, queue workers sized separately Yes, async flow High

How do you make timeouts observable?

Tag every timeout with two labels: the layer that fired (proxy, app deadline, pool checkout, statement timeout) and the route template, not the raw URL. Count them as a metric and log them with the elapsed time at each layer. Alongside those counters, track busy workers divided by total workers, request queue depth at the proxy or server, and pool checkout wait time. Alert on checkout wait first. It usually rises before error rates do, because requests queue for a connection before anything times out.

Protective timeouts versus broken requests

A timeout protected the system if it fired during saturation (high utilization, growing queue) on an endpoint whose normal latency sits well under its budget. A timeout broke a legitimate request if it fired at low load on an endpoint whose typical latency is already close to the budget. That pattern means the budget is wrong, or the work belongs in a background job. Comparing each endpoint's timeout rate against utilization at the time it fired separates the two cases.

Rank by worker-seconds, not p99

Worker-seconds = requests per hour × mean duration in seconds.

Illustrative estimate (assumed numbers, not a benchmark):

Endpoint Requests/hour Mean duration p99 Worker-seconds/hour
Export 100 3 s 8 s 300
Search 50,000 0.2 s 0.4 s 10,000

The export endpoint has the worse p99, but search holds workers about 33 times longer per hour. Search is where capacity goes.

The per-layer labels take little effort to add. During an incident they let you tell, from a single dashboard, whether workers are starved (checkout waits and queue depth climbing) or the work itself is slow (statement timeouts climbing).

How can layered timeouts save engineering time?

Most of the time saved comes from shorter incidents and fewer investigations that start from scratch.

  • Faster diagnosis. Without per-layer labels, a spike of 504s at the edge looks the same whether the pool is saturated, one query has regressed, or a third-party API is slow. Engineers then go through proxy logs, app logs and database logs by hand to work out which it is. With every timeout labelled by layer and route, the dashboard answers that question directly.
  • Smaller blast radius, fewer incidents. When a slow route can only exhaust its own budget, its own pool or its own queue, a regression in one endpoint stays a bug in that endpoint and does not become an outage for the whole app. That means fewer pages, fewer people pulled into an incident, and less time spent restoring unrelated features.
  • No orphaned-work investigations. Correct ordering removes a class of confusing failures, such as workers busy after the client has already received a 504, connections held by requests nobody is waiting on, and retry storms that multiply load. These are slow to diagnose because edge metrics understate them.
  • Cheaper decisions about slow routes. Ranking by worker-seconds tells the team which endpoint to optimize, isolate or move to a background job first. Otherwise effort tends to go to whichever route has the most alarming p99.
  • Clear ownership. When the global ordering belongs to the platform team and each route budget has a named owner, a timeout change becomes a reviewed decision and not an emergency edit during an incident.

Illustrative estimate (assumed inputs, not a measured result): assume a team has 4 starvation-related incidents a quarter. Each one pulls in 3 engineers for 3 hours, and 2 of those hours go to locating which layer and which route caused it. That is 4 × 3 × 2 = 24 engineer-hours a quarter spent on localization alone. If per-layer timeout labels cut localization to about 15 minutes per engineer, that work drops to 4 × 3 × 0.25 = 3 engineer-hours. If isolating the worst routes also prevents some incidents entirely, the saving is larger. Substitute your own incident counts and durations. The structure of the calculation matters more than these assumed values.

Checklist: layering request timeouts in your monolith

Use this checklist for any synchronous monolith with a fixed worker pool that serves mixed workloads. In that setup, one slow route can take capacity that fast routes need.

  1. Inventory every layer that can hold a request: load balancer, reverse proxy, app server, request deadline, pool checkout, database statement, and each outbound client.
  2. Record current values, and write "unset" where a layer has none. An unset timeout is effectively unbounded.
  3. Verify strictly decreasing order from the edge inward. The inner layer should fail first, so the app returns a clear error and the proxy does not cut the connection mid-work.
  4. Set a conservative global statement timeout. Grant longer limits only through scoped overrides tied to named jobs or roles.
  5. Set a short pool checkout timeout. Saturation then shows up as fast, countable errors, and requests do not sit queued.
  6. Propagate the remaining deadline to outbound calls. Do not give each call its own fixed timeout.
  7. Cap retries and retry only idempotent operations. Every retry must fit inside the remaining budget.
  8. Rank endpoints by total worker-seconds, not by latency percentiles.
  9. Isolate or offload the top offenders to a separate pool or a background job.
  10. Add per-layer timeout counters and alert on changes in their rate.
Layer Example value Must be shorter than
Load balancer 60 s —
Reverse proxy 55 s Load balancer
Request deadline 30 s Proxy
Statement timeout 20 s Remaining deadline
Pool checkout 2 s Deadline minus statement

Illustrative estimate (assumed numbers, not a benchmark): worker-seconds = requests × mean duration.

  • Endpoint A: 50,000 requests × 0.2 s = 10,000 worker-seconds.
  • Endpoint B: 500 requests × 30 s = 15,000 worker-seconds.

B ties up more worker time than A even though it gets 1% of the traffic.

Route owners own the budgets for their routes. The platform team owns the global defaults and the ordering check. Review both whenever SLOs or traffic mix change.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to request timeouts web app, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.