Why do long-running HTTP requests time out even after you raise the limits?
A slow request crosses several hops, and each has its own timer. The request succeeds only if it finishes before the shortest one fires.
| Layer | What its timeout governs | What happens when it fires |
|---|---|---|
| Client (browser, SDK, cron script) | Total wait for a response | Gives up, often retries |
| CDN or edge | Time to first byte from origin | Returns a gateway error |
| Load balancer | Idle time on the connection | Drops the connection |
| Reverse proxy | Read time from upstream | Returns a gateway error to the caller |
| App server worker | Time a worker may hold a request | Kills or recycles the worker |
| Database statement timeout | Duration of one query | Cancels the query and raises an error |
Raising one value only moves the failure to the next-smallest timer. The upstream timers also have no effect on the work underneath. When the client disconnects, the worker and its database query usually keep running. They hold a slot for a response nobody will read. If the client retries, a second copy of the same work starts, and any side effects without idempotency protection (charges, emails, inserts) can happen twice.
Illustrative estimate (assumed numbers, not a benchmark)
Busy workers ≈ arrival rate × request duration × (1 + retry rate)
The assumed inputs are:
- 1.5 requests/s reach a report endpoint.
- Each request takes 40 s.
- Clients time out at 30 s.
- 50% of clients retry once.
That gives 1.5 × 40 × 1.5 = 90 busy workers. A pool of 64 workers is exhausted, and fast endpoints start queuing behind the slow ones.
What is the async request reply pattern and how does it work?
The async request reply pattern fixes this by separating the HTTP request from the work it triggers. The web tier only accepts and records the request, which takes milliseconds. The 40-second job runs elsewhere, so it no longer holds a request worker or races a client timeout.
The flow has five steps:
- Submit. The client sends the request as usual, for example
POST /reports. - Validate synchronously. The server checks authentication, the payload schema and business rules before anything else. Invalid input gets a 4xx immediately. This step stays synchronous so callers see their mistakes right away, and so the system never stores or processes jobs that were bound to fail.
- Accept. The server persists a job record, enqueues the work and returns
202 Accepted. TheLocationheader points to the job status endpoint, such as/jobs/8f3a. The body holds the job ID and the initial state, usuallyqueued. The202status tells the client that the request was accepted but is not finished. - Process. A worker takes the job from the queue, updates the record to
running, and then sets it to a terminal state:succeededorfailed. - Poll and retrieve. The client polls the status endpoint. A
Retry-Afterheader can tell it how long to wait between polls. When the job succeeds, the client either reads the result from the status response or follows a303 See Otherredirect to the result resource.
The 303 keeps the job's status separate from its output. The result can be cached, given its own permissions, or served from storage without the job system being involved.
+----------+ +--------------------------+
| | 1. POST /reports | Web tier |
| |--------------------------------->| validate auth, schema, |
| | 2. 4xx on invalid input | rules; no job on 4xx |
| |<---------------------------------| |
| | 3. 202 Accepted | POST /reports |
| | Location: /jobs/8f3a | GET /jobs/{id} |
| Client | {id: 8f3a, state: queued} | |
| |<---------------------------------| |
| | 5. GET /jobs/8f3a (poll) | |
| |--------------------------------->| |
| | 200 running, Retry-After: 5 | |
| |<---------------------------------| |
| | 303 See Other once succeeded | |
| | Location: /results/8f3a | |
| |<---------------------------------| |
+----------+ +--------------------------+
| then GET /results/8f3a | |
| | write/read | enqueue
v v v
+----------+ +-------------+ +-----------+
| Result | | Job store | | Queue |
+----------+ +-------------+ +-----------+
^ ^ |
| write result set running, then | | dequeue
| succeeded or failed | v
+------------------------------------------------------------------------+
| 4. Worker: runs the 40-second job off the request path |
+------------------------------------------------------------------------+
The web tier validates the request and returns either a 4xx or a 202 with a job location, a worker processes the queued job while the client polls its status, and a 303 sends the client to a separate result resource.
Which endpoints should you convert, and when should you avoid async jobs?
Start with latency percentiles, not averages. An endpoint averaging two seconds can still breach a 30-second proxy timeout if its p99 is 40 seconds. Variance matters as much as the median: an export that takes 1 second for small tenants and 10 minutes for large ones fails unpredictably, and those failures land on your biggest customers. Side effects decide the urgency. A timed-out read just wastes work. A timed-out write that keeps running on the server leaves the client unsure whether to retry, and a retry can charge a card twice.
Before converting, check whether the work is slow by nature or slow by accident. Report generation over a year of data, video transcoding and bulk imports are long by nature. A missing index, an N+1 loop over related records, or an endpoint that returns every row with no limit is slow by accident. Adding the index, batching the queries or paginating the response costs less than building a job pipeline. It also fixes the real problem rather than moving it into a worker.
| Situation | Better fix |
|---|---|
| p99 well above the timeout, work is inherently long | Async job |
| Slow due to N+1 queries or a missing index | Optimize the query |
| Unbounded result set | Paginate |
| Internal-only caller, rare use | Raise the timeout as a stopgap |
| Third-party or legacy client you can't update | Keep synchronous, optimize |
A longer timeout is acceptable for internal tools with a known caller. Avoid converting endpoints whose callers cannot learn to poll. A partner integration that expects a synchronous response will treat 202 as failure.
How do you implement 202 Accepted and a job status endpoint?
The submit endpoint does one thing: validate the input, persist a job, and return 202 Accepted before any slow work starts. Clients send an Idempotency-Key header with a value they generate, such as a UUID. Store it with a unique constraint scoped to the caller. A retried request or a double click that reuses the key gets the original job back instead of creating a second one.
A typical exchange: POST /reports with Idempotency-Key: 7f3c… returns 202, Location: /jobs/481, Retry-After: 5, and a body of {"id": "481", "status": "queued"}. Retry-After tells clients when to poll, so they don't hammer the status endpoint.
State machine
Jobs move through these states:
queuedtorunningorcancelledrunningtosucceeded,failedorcancelled
Terminal states never change. Report progress only when the worker can count real units, such as rows processed out of a known total. An invented percentage is worse than none.
When a job reaches succeeded, GET /jobs/481 returns 303 See Other with Location: /reports/481. The result then has its own URL, cache rules and permissions. Failures return 200 with status: "failed" and an error object containing code (stable and machine-readable, e.g. export_too_large), message (for humans) and retryable (boolean). Clients branch on code, never on message.
Minimal job table
idowner_ididempotency_key, unique withowner_idtypestatuspayload(JSON)progress_done,progress_total(nullable)result_referror_code,error_messageattemptscreated_at,updated_at
-- PostgreSQL: minimal job table backing POST /reports and GET /jobs/{id}
CREATE TABLE jobs (
id bigserial PRIMARY KEY,
owner_id bigint NOT NULL,
idempotency_key text NOT NULL,
type text NOT NULL, -- e.g. 'report_export'
status text NOT NULL DEFAULT 'queued'
CHECK (status IN ('queued','running','succeeded','failed','cancelled')),
payload jsonb NOT NULL,
progress_done integer, -- NULL unless the worker counts real units
progress_total integer,
result_ref text, -- e.g. '/reports/481', set on success
error_code text, -- stable, e.g. 'export_too_large'
error_message text, -- human-readable only
attempts integer NOT NULL DEFAULT 0,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
-- Scope keys to the caller so two clients can reuse the same UUID safely
UNIQUE (owner_id, idempotency_key)
);
-- Submit: a retry or double click with the same key returns the original job.
-- The no-op DO UPDATE makes RETURNING yield the existing row on conflict.
INSERT INTO jobs (owner_id, idempotency_key, type, payload)
VALUES (42, '7f3c9a2e-5b1d-4c8e-9f0a-1d2b3c4e5f60', 'report_export',
'{"from": "2026-01-01", "to": "2026-03-31"}')
ON CONFLICT (owner_id, idempotency_key)
DO UPDATE SET idempotency_key = EXCLUDED.idempotency_key
RETURNING id, status;
-- Worker claims a job: only queued -> running is allowed here.
-- Zero rows updated means another worker won or the job was cancelled.
UPDATE jobs
SET status = 'running', attempts = attempts + 1, updated_at = now()
WHERE id = 481 AND status = 'queued';
-- Terminal transitions guard on 'running' so terminal states never change.
UPDATE jobs
SET status = 'succeeded', result_ref = '/reports/481', updated_at = now()
WHERE id = 481 AND status = 'running';
A PostgreSQL job table with a per-owner unique idempotency key, an idempotent submit insert that returns the existing job on retry, and guarded UPDATEs that enforce the queued → running → terminal state machine.
How do you keep the job record and the queue consistent inside a monolith?
With the job row defined, the risky moment is the enqueue. If the handler commits the row and then publishes to a broker, those are two separate writes with no shared transaction. A crash between them leaves a queued row that no worker will ever see. If you publish first and the commit then fails, a worker picks up a job that has no record. Retrying either step blindly produces duplicates. This is the dual-write problem, and raising timeouts or adding retries does not fix it.
Fix 1: transactional outbox
Insert the job row and an outbox row in the same database transaction. A separate relay process reads unsent outbox rows, publishes them to the broker, and marks them sent. The commit is atomic, so a job exists if and only if its message will eventually be published. Delivery is still at-least-once, because the relay can crash after publishing but before marking the row. Workers must therefore deduplicate on job ID.
Fix 2: the database as the queue
Skip the broker. Workers claim work with SELECT ... FOR UPDATE SKIP LOCKED on status = 'queued', update the row to running, and commit. SKIP LOCKED lets concurrent workers pass over rows that another worker has already locked instead of blocking on them. PostgreSQL and recent MySQL versions support it.
| Concern | Outbox + broker | Database as queue |
|---|---|---|
| Moving parts | Relay process plus broker | None beyond the database |
| Load on primary DB | Outbox writes and relay polling | Every claim, heartbeat and status update |
| Contention | Low on the jobs table | Grows with worker count and poll frequency |
| Table bloat | Outbox rows need pruning | Frequent updates leave dead rows that need vacuum or purge |
| Throughput ceiling | Broker scales independently | Bounded by primary DB capacity |
| Consistency | At-least-once delivery, so workers must deduplicate | Claim and status change share one transaction |
For moderate job volume inside a monolith, the database queue is usually enough. Move to an outbox and broker when job traffic begins to compete with user-facing queries for the primary database.
-- PostgreSQL. Fix 1: transactional outbox.
-- The job row and its outbox message commit together or not at all.
BEGIN;
INSERT INTO jobs (id, status, payload, created_at)
VALUES ('7f3c2a10-5b1e-4c2d-9a8f-0e6d4b3c2a11', 'queued',
'{"report_id": 42}', now());
INSERT INTO outbox (job_id, topic, sent_at)
VALUES ('7f3c2a10-5b1e-4c2d-9a8f-0e6d4b3c2a11', 'report-jobs', NULL);
COMMIT;
-- Relay: fetch a batch of unsent messages without blocking other relays.
-- After publishing to the broker, it sets sent_at = now() for these rows and commits.
-- A crash between publish and that commit means the batch is published again.
SELECT id, job_id, topic
FROM outbox
WHERE sent_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;
-- Fix 2: the database as the queue.
-- Each worker claims one queued job. Rows already locked by another
-- worker are skipped instead of waited on.
WITH claimed AS (
SELECT id
FROM jobs
WHERE status = 'queued'
ORDER BY created_at
LIMIT 1
FOR UPDATE SKIP LOCKED
)
UPDATE jobs
SET status = 'running', started_at = now()
FROM claimed
WHERE jobs.id = claimed.id
RETURNING jobs.id, jobs.payload;
-- Every one of these UPDATEs leaves a dead row version for vacuum to reclaim.
PostgreSQL statements for both fixes: an atomic job-plus-outbox insert with a relay that fetches unsent rows, and a worker that claims a queued job with FOR UPDATE SKIP LOCKED and marks it running.
How should workers and clients handle retries, failures and polling?
Worker side
Run workers as a separate process type built from the same deploy artifact, so they share models and validation but scale and restart independently of web processes. Give every job a hard timeout and a lease: the worker claims a row with a locked_until or heartbeat column, or the broker's visibility timeout, and a crashed worker's job becomes claimable again once the lease expires. That recovery path means delivery is at least once, so job logic must be idempotent. Upserts and "set status if still pending" updates handle most cases. Emails, payments and external API calls need more: record the side effect against the job ID before or atomically with performing it, and pass an idempotency key when the provider accepts one.
Cap retries, typically at a small fixed number, with exponential backoff and jitter. Exhausted jobs move to a dead status with the last error stored, not silently dropped. On deploy, stop claiming new work on SIGTERM, then either finish or release in-flight jobs before the platform kills the process. Cancellation is cooperative. The worker rechecks the job's status at checkpoints between batches or steps and exits cleanly when it reads cancelled.
Client side
Polling is the right baseline. It works through proxies and firewalls and needs no client endpoint. Clients should back off, honor Retry-After, and enforce an overall deadline. Offer webhooks for server-to-server consumers and server-sent events for interactive UIs once polling load becomes noticeable. A hybrid also helps. Wait a few seconds for completion and return 200 directly, and fall back to 202 only for slow cases.
-- Claim one job: pending and due, or 'running' with an expired lease (crashed worker).
-- SKIP LOCKED lets many workers poll the same table without blocking each other.
UPDATE jobs
SET status = 'running', locked_by = 'worker-7f3a',
locked_until = now() + interval '60 seconds', attempts = attempts + 1
WHERE id = (
SELECT id FROM jobs
WHERE (status = 'pending' AND run_after <= now())
OR (status = 'running' AND locked_until < now())
ORDER BY created_at
LIMIT 1
FOR UPDATE SKIP LOCKED
)
RETURNING id, payload, attempts;
-- Heartbeat at each checkpoint. Zero rows returned means the job was cancelled
-- or the lease was lost, so the worker must stop without writing results.
UPDATE jobs SET locked_until = now() + interval '60 seconds'
WHERE id = $1 AND locked_by = $2 AND status = 'running'
RETURNING id;
-- Guard a non-idempotent side effect: send the email only if this insert returns a row.
INSERT INTO job_side_effects (job_id, effect) VALUES ($1, 'welcome_email')
ON CONFLICT (job_id, effect) DO NOTHING
RETURNING job_id;
-- On failure: exponential backoff with jitter, or 'dead' after 5 attempts with the error kept.
UPDATE jobs
SET status = CASE WHEN attempts >= 5 THEN 'dead' ELSE 'pending' END,
run_after = now() + interval '1 second' * power(2, attempts) * (0.5 + random()),
last_error = $3, locked_by = NULL, locked_until = NULL
WHERE id = $1 AND locked_by = $2 AND status = 'running';
-- On success: the status guard makes a late or duplicate completion a no-op.
UPDATE jobs SET status = 'succeeded', result = $3, locked_by = NULL, locked_until = NULL
WHERE id = $1 AND locked_by = $2 AND status = 'running';
PostgreSQL statements for a lease-based job worker: claiming with SKIP LOCKED, reclaiming expired leases, heartbeats that double as cancellation checkpoints, a side-effect guard keyed on job ID, bounded retries with jittered backoff, and a dead-letter status.
What does migrating and operating async jobs cost, and where does it save engineering time?
Migration path
Keep the synchronous endpoint working behind a feature flag or content negotiation (a Prefer: respond-async header, for example) while you ship the async variant. Move consumers over one at a time, watch traffic on the old path drop to zero, then retire it. Version the API contract explicitly, because changing a 200 with a body into a 202 with a Location header breaks clients.
Operating it
Track queue depth, job age, time-to-start, failure rate and stuck jobs. Time-to-start shows when workers fall behind, often before users notice. Check authorization on both the status and result endpoints, and use random job IDs so callers cannot enumerate other tenants' jobs. Set retention rules for job records and result artifacts, or storage grows without limit.
Cost versus savings
You pay for the queue, the workers, cleanup jobs, more complex clients, and rate limits to contain polling load. Debugging also gets harder when a failure appears hours after the request that caused it. In return, you get fewer timeout incidents and retries you can observe, and you can convert one endpoint at a time without splitting the monolith.
| Option | Fixes root cause | Client changes | Main risk |
|---|---|---|---|
| Optimize the query or code | Yes, if work can shrink | None | Hits a floor for inherently slow work |
| Paginate | Partly | Moderate | Doesn't fit single-result operations |
| Raise timeouts | No | None | Proxies and clients still cut connections |
| Streaming | Partly | Moderate | Long-lived connections tie up workers |
| Async request reply | Yes | Significant | Operational overhead |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to async request reply pattern, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.