What is timeout layering in a monolith request path?
Every request in a monolith crosses several independent timers, each owned by a different layer. Timeout layering means setting them deliberately so that the outer layers give up after the inner ones. Each timer only stops its own layer. A timeout firing at the edge does not mean work has stopped further in.
| Timeout | What it stops | What it leaves running |
|---|---|---|
| Client timeout | The client's wait | Everything server-side |
| Proxy idle/read timeout | Proxy wait; client gets a 504 | App worker and its query |
| App server worker/request timeout | The worker (often by killing it) | Postgres backend, until it notices the dropped connection |
| Application deadline | Code paths that check it | Calls that ignore it |
| Pool checkout timeout | Waiting for a free connection | Queries on connections already checked out |
statement_timeout |
The running statement; the transaction becomes aborted | The session and its connection |
lock_timeout |
A statement waiting on a lock | The lock holder |
idle_in_transaction_session_timeout |
The whole session, releasing its locks | Nothing in Postgres; the app holds a dead connection |
Teams layer timeouts to protect capacity. Workers and pooled connections are both fixed resources. A slow dependency holds both for as long as nothing cancels the work. You can estimate the effect with Little's law: busy workers ≈ arrival rate × time each request is held.
Illustrative estimate (assumed numbers, not a benchmark): assume 50 requests/s and a pool of 40 connections. If queries normally take 0.1 s, about 5 connections are busy. If an incident stretches queries to 30 s, demand rises to about 1,500, so the pool is exhausted within a second. A 2 s statement_timeout caps demand near 100. That still exceeds the pool, but the queue drains once the incident ends.
This guide is for backend and platform engineers who run a monolith on Postgres behind a reverse proxy.
+--------+ client timeout
| Client | stops: client wait leaves: all server work
+--------+
|
v
+--------------------+ idle / read timeout -> 504 to client
| Reverse proxy / LB | leaves: app worker and its query running
+--------------------+
|
v
+----------------------------------------------+
| App server worker worker/request timeout |
| +----------------------------------------+ |
| | Application code application deadline | |
| +----------------------------------------+ |
| | |
| v |
| +----------------------------------------+ |
| | Connection pool checkout timeout | |
| +----------------------------------------+ |
+----------------------------------------------+
|
v
+----------------------------------------------+
| Postgres backend (one per connection) |
| statement_timeout -> cancels statement |
| lock_timeout -> cancels lock wait |
| idle_in_transaction_session_timeout |
| -> ends session |
+----------------------------------------------+
A monolith request crosses independent timers at the client, proxy, app worker, application, pool and Postgres layers. Each timer stops only its own layer, so outer timeouts should be set longer than inner ones.
Who should use timeout layering?
Timeout layering is worth the effort for teams whose request path matches the diagram above: one application process pool, one shared database connection pool, and a reverse proxy or load balancer in front. Specifically:
- Backend engineers who own the monolith's data-access code. They write the transaction helper, the deadline plumbing and the error classification, so most of the work lands on them.
- Platform and SRE engineers who own the proxy, app server and database configuration. They set the outer timers and the per-role Postgres defaults, and they are the ones paged when the pool runs dry.
- Teams whose monolith serves mixed traffic. Interactive HTTP requests, background jobs, migrations and reports often share one database. Layered budgets and per-role limits stop the slowest class from starving the fastest.
- Teams that have already seen 504s with no clear cause, duplicate writes after user retries, or connection pool exhaustion during a slow-query incident. These are the symptoms this approach targets.
It matters less for a small internal tool with low, predictable traffic and no shared pool pressure. In that case one sensible statement_timeout and a proxy timeout set above it may be enough.
When should you use timeout layering?
Apply it when one or more of these is true:
- Your timeouts were set independently by different people. If nobody can say whether
statement_timeoutis shorter than the proxy timeout, the order is probably wrong somewhere. - A single slow query can exhaust the pool. If the Little's law calculation above, with your own arrival rate and pool size, shows the pool saturating during a plausible slowdown, inner timeouts are a capacity control, not just a latency control.
- You are adding a reverse proxy, changing its timeout, or moving to a transaction-mode pooler. Each of these changes which timer fires first, and the pooler changes how session settings behave.
- You run migrations against a live database.
lock_timeouton the migration role and on request traffic keeps a blockedALTER TABLEfrom queueing every request behind it. - You are about to raise a timeout to fix an incident. That is the moment to decide whether the raise should be scoped to one role or one transaction instead of applied globally.
The best time to introduce deadline propagation is when you already have a central transaction helper or data-access layer. Retrofitting it into a codebase where each call site opens its own connection is much more expensive, because every call site must change.
Why does a 504 not stop the query that caused it?
A timeout at the proxy is a decision to stop waiting. It is not a decision to stop working. When the proxy's upstream timer expires, it sends a 504 to the client and closes its connection to the app. The app worker usually doesn't notice until it tries to write the response. Meanwhile the query it issued is still executing in a Postgres backend, holding a pooled connection, its locks, and possibly an open transaction. Postgres has no way to know the HTTP client has left unless something explicitly cancels the query.
How orphaned work compounds
Every abandoned request leaves behind a worker and a connection that still look busy. If the client retries, it adds a second copy of the same expensive query while the first is still running. A retrying proxy or SDK can add a third. Under a slow query, three layers that each retry twice can turn one user action into many concurrent executions. All of them compete for the same pool, so the pool runs out and healthy requests start queueing behind work nobody is waiting for.
The fix has two parts:
- Retry in one layer only. Give that layer a budget, such as a cap on retries as a fraction of total traffic, so retries stop when the error rate climbs.
- Make retried writes idempotent. Have the client send an idempotency key, and have the server record each key in the same transaction as the write it protects.
The opposite failure
If the proxy timeout is shorter than the backend's real latency, the request can commit successfully just after the 504 goes out. The user sees an error and resubmits, which creates a duplicate order or payment. Idempotency keys catch this case too. The other remedy is ordering the budgets so the edge timeout is the longest, which is the subject of the next section.
How should you order timeout budgets from the edge inward?
The rule is that each inner budget must be strictly shorter than the layer outside it, with headroom left for serialization, response writing and network time. If an inner layer can outlive its caller, nobody is left to receive its error, and the work keeps consuming resources after the client has already received a 504.
Illustrative estimate (assumed numbers, not a benchmark)
- Proxy timeout: 30s
- App request deadline: 25s
- Pool checkout timeout: 2s
- Postgres
statement_timeout: 20s
Check each layer against these two inequalities:
app_deadline ≤ proxy_timeout − edge_headroom, so 25 ≤ 30 − 5checkout + statement_timeout + app_work ≤ app_deadline, so 2 + 20 + 3 ≤ 25
The 5s edge headroom covers rendering and sending the error. The 3s inside the app covers its own logic and serialization. Plug in your own values. If either side goes negative, the order is broken.
| Inverted setup | What breaks |
|---|---|
statement_timeout (40s) > proxy (30s) |
Client gets a 504 at 30s. The query runs to 40s and holds a connection and locks. |
| App deadline (35s) > proxy (30s) | The app's structured timeout error never reaches the client, so the logs show success or a late failure. |
| Checkout (10s) + statement (20s) > app deadline (25s) | The app gives up while the query is still valid and running. |
Pool checkout is the hardest value to choose. An aggressive value such as 200ms sheds load quickly, but it turns a brief burst of contention into a spike of errors. A lenient value hides saturation, because requests queue silently until the proxy fires and every caller sees a 504.
Streaming responses, websockets and large uploads are exceptions. A single total timeout either kills legitimate long transfers or never fires. These paths need separately designed limits: an idle limit for time between bytes, and a total limit for the whole transfer.
How do you configure Postgres statement_timeout, lock_timeout and idle-in-transaction limits?
Postgres timeouts can be set at four scopes. A more specific scope overrides a broader one: ALTER DATABASE ... SET is overridden by ALTER ROLE ... SET, which is overridden by a session SET, which is overridden by SET LOCAL. SET LOCAL lasts only until the current transaction commits or rolls back.
Set a tight default and raise it only where you mean to. Give statement_timeout a low value for the database or role. Then use SET LOCAL statement_timeout inside the specific transactions you know are slow. Avoid one large global value. A large value lets every forgotten slow query run long, and it hides regressions until they pile up under load. With a tight default plus explicit overrides, each slow path is visible in the code and can be reviewed.
Run two other limits alongside it:
lock_timeoutcaps how long a statement waits for a lock. Without it, a query blocked behind a migration uses up its whole statement budget doing nothing.idle_in_transaction_session_timeoutterminates sessions that hold a transaction open while idle. This usually means application code that crashed or stalled mid-transaction while still holding locks.
Use separate login roles for each traffic class:
| Role | statement_timeout |
lock_timeout |
Why |
|---|---|---|---|
app_http |
Below the HTTP budget | Short | Fail before the proxy does |
app_jobs |
Longer | Moderate | Retries are cheap |
app_migrate |
Long or off | Very short | Never queue behind traffic |
app_report |
Minutes | Moderate | Isolated, read-only |
Pitfalls:
- A session
SETstays on a pooled connection after the request ends. The next request inherits it. - Behind a transaction-mode pooler, a session
SETmay run on a different backend than your next query. It can then leak to other clients. SET LOCALis the reliable option, but it must be issued in every transaction. Put it in your transaction helper rather than at individual call sites.- Role defaults only take effect at login. Pooled connections keep the old values until they reconnect.
-- Tight database-wide default: anything not overridden fails fast.
ALTER DATABASE appdb SET statement_timeout = '5s';
ALTER DATABASE appdb SET lock_timeout = '2s';
ALTER DATABASE appdb SET idle_in_transaction_session_timeout = '30s';
-- One login role per traffic class. Role settings override the database
-- defaults and are applied when a new session starts.
ALTER ROLE app_http SET statement_timeout = '2s'; -- below the HTTP budget
ALTER ROLE app_http SET lock_timeout = '500ms';
ALTER ROLE app_jobs SET statement_timeout = '60s'; -- retries are cheap
ALTER ROLE app_jobs SET lock_timeout = '5s';
ALTER ROLE app_migrate SET statement_timeout = 0; -- 0 disables the limit
ALTER ROLE app_migrate SET lock_timeout = '1s'; -- never queue behind traffic
ALTER ROLE app_report SET statement_timeout = '10min';
ALTER ROLE app_report SET lock_timeout = '10s';
ALTER ROLE app_report SET default_transaction_read_only = on;
-- Known slow path: raise the limit for this transaction only.
-- SET LOCAL reverts at COMMIT or ROLLBACK, so it cannot leak to the next
-- request on a pooled connection. It must be repeated in every transaction.
BEGIN;
SET LOCAL statement_timeout = '15s';
SELECT customer_id, sum(amount) FROM invoices
WHERE issued_at >= now() - interval '1 year'
GROUP BY customer_id;
COMMIT;
-- Avoid: a session-level SET outlives the request on a pooled connection,
-- and behind a transaction-mode pooler it may not apply to the next query.
-- SET statement_timeout = '15s';
-- Check the effective value in the current session.
SHOW statement_timeout;
Database-wide timeout defaults, per-role limits for each traffic class, and a per-transaction SET LOCAL override for one known slow query.
How do you propagate a request deadline down to every query?
Compute the deadline once, at the point where the request enters your code: arrival time plus the budget for that route. Every layer below receives that absolute deadline, never a duration. Durations restart their count at each hop, so a chain of call sites that each assume "5 seconds" can run far past the edge's limit. The carrier differs by language. Go has context.Context with a deadline. .NET has CancellationToken. Node has AbortSignal. Python can use a context variable or an explicit argument. The idea is the same in all of them: each function asks "how much time is left?" instead of reading a constant.
In the transaction helper, derive the query timeout like this:
- Compute
remaining = deadline - now. - Subtract headroom for serializing the response and for the network return. This is commonly tens of milliseconds, but tune it to your own measurements.
- If what is left is below a floor, fail immediately rather than starting work that cannot finish.
- Otherwise issue
SET LOCAL statement_timeoutwith that value in milliseconds as the first statement of the transaction. It resets automatically at commit or rollback.
The cost is plumbing. The deadline must thread through every layer: handlers, services, repositories, background helpers. Any library that accepts no context, or ignores the one it is given, silently breaks the chain. Partial adoption is the worst case, because dashboards look covered while the untouched paths keep running unbounded.
The payoff is fewer investigations into orphaned queries that outlive their requests, and a single place to reason about budgets.
import time
import contextvars
request_deadline = contextvars.ContextVar("request_deadline")
HEADROOM_S = 0.050 # response serialization + network return; tune from your own measurements
FLOOR_S = 0.010 # below this, starting a query is pointless
class DeadlineExceeded(Exception):
pass
def start_request(route_budget_s):
# Store an absolute deadline, never a duration, so lower layers cannot restart the clock.
request_deadline.set(time.monotonic() + route_budget_s)
def query_budget_ms():
remaining = request_deadline.get() - time.monotonic()
usable = remaining - HEADROOM_S
if usable < FLOOR_S:
# Fail fast instead of starting work that cannot finish in time.
raise DeadlineExceeded(f"{remaining * 1000:.0f} ms left")
return int(usable * 1000)
def run_in_transaction(conn, fn):
# conn is a psycopg 3 connection. The budget is computed per transaction,
# from the time remaining when it starts, not from a fixed constant.
timeout_ms = query_budget_ms()
with conn.transaction():
# First statement in the transaction. SET LOCAL resets at commit or rollback.
# timeout_ms is an int computed above, so formatting it into the SQL is safe.
conn.execute(f"SET LOCAL statement_timeout = {timeout_ms}")
return fn(conn)
A Python sketch that stores an absolute request deadline in a context variable, computes each transaction's remaining budget minus headroom, fails fast below a floor, and applies it with SET LOCAL statement_timeout as the transaction's first statement.
How does request cancellation reach Postgres and what happens to the transaction?
When a request deadline expires, the driver can ask Postgres to stop the running statement. It does this by opening a separate connection and sending a cancel message that carries the backend's process ID and secret key. This is best effort. If the network is congested or the statement finishes first, the cancel arrives late or does nothing. If the application process is stuck in garbage collection or has crashed, no cancel is sent at all. statement_timeout runs inside the server, so it fires whether or not the client is still alive. That makes it the backstop you can rely on.
| Client cancel request | Server statement_timeout |
|
|---|---|---|
| Sent from | Application, over a new connection | Postgres backend itself |
| Survives client hang or crash | No | Yes |
| Tracks the remaining request deadline | Yes | Only if set per statement |
| Error raised | Query canceled | Query canceled (timeout) |
Either way, the surrounding transaction is now aborted. Postgres rejects every later command until the transaction is rolled back. A pool must roll back, or discard, any connection that returns cancelled or errored. If it hands that connection to the next request, that request inherits an aborted transaction or leftover session state.
A rollback only covers the database. Emails, payment calls or queue messages sent earlier in the request have already happened. Make them idempotent or move them after commit.
Some work is legitimately slow, such as reports, exports and reads against a cold cache. Give it an explicit, reviewed path instead of quietly raising the global limit. Options include a dedicated role with its own timeout, a job queue, or a read replica.
How do you tell which timeout fired and fix the slow query behind it?
Name every layer in logs and metrics
Each layer should raise its own error type and increment its own counter, labelled with the route: proxy_timeout, app_deadline_exceeded, pool_checkout_timeout, statement_timeout, lock_timeout. A single generic "timeout" metric hides the information you need most, which is where the time went. A spike in pool checkout timeouts means connections are scarce. A spike in statement timeouts means queries are slow. These two problems need different fixes.
In application code, classify by SQLSTATE rather than by message text:
- 57014
query_canceledcovers bothstatement_timeoutand cancellation your app sent. Tell them apart by checking whether your own deadline had already expired when the error came back. - 55P03
lock_not_availablemeanslock_timeoutfired. Look for a competing transaction, not for a slow plan. - 25P03 means the session was terminated by
idle_in_transaction_session_timeout. Treat it as a bug in the code that holds transactions open.
Fix the query, not the limit
Postgres logs cancelled statements with their SQL text when log_min_error_statement is at error or lower, which is the default. Set log_min_duration_statement somewhat below your statement timeout. That way you also catch the queries that are close to the limit, before they start failing. Enable log_lock_waits to see who blocked whom. Run the logged statements through EXPLAIN (ANALYZE, BUFFERS) to find missing indexes, sequential scans and bad row estimates. In most cases the right fix is an index or a rewrite, not a higher timeout.
Review checklist for timeout changes
- Each layer's budget is shorter than the layer outside it, from proxy to app deadline to pool checkout to statement.
- Raised limits are scoped to a role or route, not set globally.
- Any new timeout comes with its own error type and metric.
- The PR states which slow query motivated the change and links its plan.
import time
import psycopg
from psycopg_pool import ConnectionPool, PoolTimeout
from prometheus_client import Counter
TIMEOUTS = Counter("db_timeouts_total", "Timeouts by layer", ["route", "layer"])
pool = ConnectionPool("postgresql://app:placeholder@db.internal/appdb", open=True)
def run_query(route: str, sql: str, params, deadline: float):
remaining = deadline - time.monotonic()
if remaining <= 0:
TIMEOUTS.labels(route, "app_deadline_exceeded").inc()
raise TimeoutError("app deadline exceeded before query")
try:
# Pool checkout gets its own budget, capped by what is left of the deadline.
with pool.connection(timeout=min(remaining, 1.0)) as conn:
return conn.execute(sql, params).fetchall()
except PoolTimeout:
TIMEOUTS.labels(route, "pool_checkout_timeout").inc() # connections are scarce
raise
except psycopg.Error as e:
# Classify by SQLSTATE, never by message text.
if e.sqlstate == "57014":
# Same code for statement_timeout and our own cancel: check our deadline.
layer = ("app_deadline_exceeded" if time.monotonic() >= deadline
else "statement_timeout")
elif e.sqlstate == "55P03":
layer = "lock_timeout" # look for a competing transaction
elif e.sqlstate == "25P03":
layer = "idle_in_transaction_timeout" # bug: transaction held open
else:
raise
TIMEOUTS.labels(route, layer).inc()
raise
Python data-access helper using psycopg 3 that maps pool checkout failures and the Postgres SQLSTATE codes 57014, 55P03 and 25P03 to distinct per-layer timeout metrics, labelled by route.
What are the alternatives to layered timeout budgets?
Layered budgets are not the only way to protect a monolith from slow work. Most alternatives work best alongside timeouts rather than replacing them, because none of them stop a query that is already running.
| Alternative | What it does | Where it falls short |
|---|---|---|
| Single global timeout | One proxy timeout, or one large statement_timeout, and nothing else |
Inner work outlives the outer limit; slow regressions stay hidden |
| Concurrency limits / load shedding | Cap in-flight requests per route or per worker pool and reject the excess early | Bounds how much work starts, not how long it runs |
| Bulkheads (separate pools per traffic class) | Give HTTP, jobs and reports their own connection pools or roles | One class can still exhaust its own pool without inner timeouts |
| Circuit breakers | Stop calling a dependency after repeated failures | Built for remote dependencies; a monolith's own database is usually not one you can stop calling |
| Moving slow work to async jobs | Return immediately and run the work in a queue worker | Requires a redesign of the endpoint and a way to report results |
| Read replicas for heavy reads | Send reports and exports to a separate server | Adds replication lag; does not protect the primary from slow writes |
| Query optimization alone | Fix the slow queries and skip the timeouts | The next regression or lock pile-up arrives with no limit in place |
In practice, the per-role limits shown earlier are already a simple bulkhead, and a concurrency limit at the app server pairs well with a tight pool checkout timeout. Ordered timeouts remain the layer that guarantees work actually stops.
How can timeout layering save engineering time?
The savings come from making incidents shorter and less ambiguous, and from removing classes of cleanup work entirely:
- Faster diagnosis. With a distinct metric per layer, a timeout incident starts with a known location: pool exhaustion, slow statement, lock wait or a transaction held open. Without it, engineers correlate proxy logs, app logs and database logs by timestamp to work out which timer fired.
- No manual hunting for orphaned queries. When
statement_timeoutsits below the request budget, abandoned queries end on their own. Nobody needs to search the database's activity view during an incident and cancel backends by hand. - Fewer duplicate-write cleanups. Ordered budgets plus idempotency keys remove the case where a request commits after the client has seen a 504. That avoids support tickets and data repair scripts for duplicate orders or payments.
- Safer migrations. A short
lock_timeouton the migration role makes a blocked migration fail quickly and retry, instead of queueing production traffic behind it and turning a deploy into an outage. - Cheaper reviews. A tight default with explicit
SET LOCALoverrides puts every slow path in the code, where a reviewer can see it. The review checklist above turns timeout changes into a short, repeatable check instead of a debate. - Slow queries surface before they fail. Logging statements that approach the limit lets the team fix a plan during normal work rather than during an incident.
Illustrative estimate (assumed numbers, not a measurement): assume a team handles four timeout-related incidents a quarter, and that without per-layer metrics each one takes two engineers about three hours to localize. If per-layer metrics cut localization to about one hour, the team gets back roughly 16 engineer-hours a quarter, before counting any duplicate-write cleanup that no longer happens. Substitute your own incident count and durations; the point is that the one-time cost of plumbing deadlines and metrics is paid once, while the diagnosis cost recurs with every incident.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to request timeout configuration backend, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.