Async Job Endpoint Pattern: Moving Slow Monolith Requests to Jobs

What is the async job endpoint pattern, and which slow endpoints actually need it?

In the async job endpoint pattern, the client submits work and the server does not do that work inside the request. The server records a job, responds with 202 Accepted and a reference to a job resource, and the client reads the outcome later. The hard part is the contract and the state management, not the queue. You have to decide what the client is promised, what states a job can be in, and how the server stops work from being duplicated or lost. Choosing a broker is the easy part.

The pattern fits endpoints whose duration is long or unpredictable for reasons you can't optimize away. Examples are report generation over large date ranges, bulk imports, media processing, and calls to slow third-party APIs whose latency you don't control. It does not fit an endpoint that is slow because of a missing index or an N+1 query loop. Moving that endpoint to a job hides the defect behind a status page and adds moving parts. Profile first and fix the query.

Teams adopt the pattern mainly for reliability. Request threads and database connections are freed quickly. A gateway or load balancer timeout can no longer kill work that was progressing. Retries also become explicit instead of accidental. The work itself does not get any faster.

How does the 202 Accepted request contract work?

Validation stays on the synchronous path. Before anything is queued, the web process checks authentication, parses the payload, and runs cheap business checks such as "does this account exist" or "is the date range under 90 days". Failures return 400, 403, or 422 immediately. Errors the client can fix should never become failed jobs it has to discover later by polling.

Once the request passes, the handler writes a job row (or an outbox row) in the same transaction as any related state. It then returns:

• POST /reports → 202 Accepted
• Location: /jobs/7f3c9a
• Body: {"job_id": "7f3c9a", "status": "queued"}

A 202 means only that the server accepted the request for processing. It does not promise that the job will succeed, start soon, or ever finish. Clients that treat 202 as "done" will show users results that may never exist. The Location header lets generic clients follow the status URL without parsing the body. The job ID in the body is for logs, support tickets, and UIs.

The flow end to end

1. Client → POST /reports → web process (validates, returns 4xx or continues)
2. Web process → INSERT → job table / outbox → returns 202 and Location to the client
3. Worker → claims the row → runs the work → writes status and result
4. Client → GET /jobs/7f3c9a → status endpoint → reads the job table → returns 200 with "status": "running" while the job is in progress, then "succeeded" with a result link, or "failed" with an error

The status endpoint reads the same table the worker writes to. A job's state therefore has a single source of truth.

Copy# 1. Invalid input is rejected synchronously. No job row is written.
POST /reports HTTP/1.1
Content-Type: application/json

{"account_id": "acct_123", "from": "2025-01-01", "to": "2025-12-31"}

HTTP/1.1 422 Unprocessable Entity
Content-Type: application/json

{"error": "date_range_too_large", "detail": "Range must be under 90 days"}

# 2. Valid input: job row committed, 202 returned. Nothing has run yet.
POST /reports HTTP/1.1
Content-Type: application/json

{"account_id": "acct_123", "from": "2025-10-01", "to": "2025-10-31"}

HTTP/1.1 202 Accepted
Location: /jobs/7f3c9a
Content-Type: application/json

{"job_id": "7f3c9a", "status": "queued"}

# 3. Poll the URL from Location. The job resource itself was found, so 200.
GET /jobs/7f3c9a HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json

{"job_id": "7f3c9a", "status": "running"}

# 4. Terminal state: either a result link or an error, never both.
GET /jobs/7f3c9a HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json

{"job_id": "7f3c9a", "status": "succeeded", "result_url": "/reports/7f3c9a"}

These HTTP exchanges show a synchronous 422 rejection, a 202 Accepted response with a Location header and job ID, and status polling that moves from running to succeeded.

How should you design the job resource and its lifecycle?

Treat the job as a row in your own database, not as a message in the broker. Queue messages get acknowledged, expire, or disappear when a broker is rebuilt. A jobs table survives all of that, and support engineers can query it.

A workable jobs table carries id, type, state, payload, attempts, max_attempts, a nullable progress (0–100), created_at, started_at, finished_at, locked_by, and lease_expires_at. It also needs result_ref and error. result_ref points to object storage or a results row; keep large blobs out of the hot table. error is structured, with a code, a message, and a retryable flag, so clients can branch on it without parsing strings.

The allowed transitions are:

• queued → running
• queued → cancelled
• running → succeeded
• running → failed
• running → cancelled
• running → queued, when a lease expires and attempts remain

Terminal states never change. Enforce this inside the UPDATE itself, for example ... WHERE id = $1 AND state = 'running' AND locked_by = $2. If zero rows change, the worker has lost its lease and must discard its output. A read-then-write check in application code races; the guarded write does not.

Workers renew lease_expires_at on a heartbeat. The lease tells everyone else whether "running" is still true.

A failed job still returns 200 because the status read itself succeeded. The outcome belongs in the body.

Copy-- PostgreSQL: the job is a durable row, not a broker message.
CREATE TABLE jobs (
  id               uuid PRIMARY KEY,
  type             text NOT NULL,
  state            text NOT NULL DEFAULT 'queued'
                   CHECK (state IN ('queued','running','succeeded','failed','cancelled')),
  payload          jsonb NOT NULL,
  attempts         int NOT NULL DEFAULT 0,  -- incremented when a worker claims the job
  max_attempts     int NOT NULL DEFAULT 3,
  progress         smallint CHECK (progress BETWEEN 0 AND 100),  -- NULL = not reported
  created_at       timestamptz NOT NULL DEFAULT now(),
  started_at       timestamptz,
  finished_at      timestamptz,
  locked_by        text,         -- worker id holding the lease
  lease_expires_at timestamptz,  -- pushed forward by heartbeats
  result_ref       text,         -- object storage key or results row id; no blobs here
  error            jsonb         -- {"code": "...", "message": "...", "retryable": false}
);

-- running -> succeeded, only if this worker still holds the lease.
-- If 0 rows are updated, the lease was lost: discard the output.
UPDATE jobs
SET state = 'succeeded', progress = 100, finished_at = now(),
    result_ref = $3, locked_by = NULL, lease_expires_at = NULL
WHERE id = $1 AND state = 'running' AND locked_by = $2;

-- Reaper: running -> queued when the lease expired and attempts remain.
-- Clearing locked_by makes the old worker's guarded UPDATE match 0 rows.
UPDATE jobs
SET state = 'queued', locked_by = NULL, lease_expires_at = NULL
WHERE state = 'running'
  AND lease_expires_at < now()
  AND attempts < max_attempts;

A PostgreSQL jobs table with a state CHECK constraint, plus guarded UPDATEs. One completes a job only while the worker still holds its lease, and one requeues jobs whose lease has expired.

How do you prevent duplicate jobs and lost jobs?

Duplicates start at submit time. Clients retry on timeouts, and a retried POST without protection creates a second job. Accept an Idempotency-Key header, or derive a key by hashing the user ID and the normalized request body. Store it in a column with a unique constraint. When an insert hits that constraint, look up the existing job and return its 202 response. Let the database decide which request was first, because an application-level "check then insert" races under concurrency.

Workers also need to be safe to run twice. Most queues deliver at least once, so a worker that crashes after doing the work but before acknowledging the message will get it again. For pure computation, writing the same result twice is harmless. Side effects such as charging a card or sending an email are different. The queue cannot make them idempotent, so pass a stable key to the payment provider and record "sent" markers inside the same transaction as the state change.

Lost jobs come from the dual write. If you commit the job row and then push to the queue, a crash between the two steps leaves a queued job that no worker will ever see. If you push first and then commit, a worker can pick up a job that has no row yet. Two approaches avoid this:

Copy-- PostgreSQL. The UNIQUE constraint dedupes retried submits.
CREATE TABLE jobs (
  id              bigserial PRIMARY KEY,
  idempotency_key text NOT NULL UNIQUE,
  status          text NOT NULL DEFAULT 'queued',
  payload         jsonb NOT NULL,
  created_at      timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE outbox (
  id     bigserial PRIMARY KEY,
  job_id bigint NOT NULL REFERENCES jobs(id)
);

-- Submit (outbox approach): job row and outbox row commit together.
-- On a duplicate key, nothing is inserted into either table.
BEGIN;
WITH new_job AS (
  INSERT INTO jobs (idempotency_key, payload)
  VALUES ('user-42:9f2c1e7a', '{"report": "q3-sales"}')
  ON CONFLICT (idempotency_key) DO NOTHING
  RETURNING id
)
INSERT INTO outbox (job_id) SELECT id FROM new_job;
COMMIT;
-- Return 202 for whichever job owns the key, new or existing.
SELECT id, status FROM jobs WHERE idempotency_key = 'user-42:9f2c1e7a';

-- Worker claim (database-as-queue approach, no outbox or broker):
-- SKIP LOCKED makes concurrent workers take different rows instead of waiting.
UPDATE jobs SET status = 'running'
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'queued'
  ORDER BY created_at
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
RETURNING id, payload;

PostgreSQL sketch showing an idempotent submit that writes the job and its outbox row in one transaction, plus the alternative SKIP LOCKED claim query for using the jobs table directly as the queue.

How should clients poll for job status and results?

A GET on the job resource returns its current state, timestamps, and progress if you track it. Treat this as the contract clients code against. Fields that change shape between states break integrations.

Poll interval. Include a Retry-After header on every non-terminal response. Clients then follow the server's pacing instead of hardcoding one. The server knows queue depth and typical job duration, so it can return a short value for a job that is about to finish and a longer one when workers are backlogged. Clients should treat the header as a minimum. They should add exponential backoff with jitter so many clients don't poll in sync, and they need an overall deadline after which they stop and report a timeout.

Completion. When the job succeeds, either embed the result in the status body or return 303 See Other with a Location pointing at the result resource. Embedding suits small payloads. Redirecting suits large or separately cacheable results. On failure, return a terminal state with a machine-readable error, not an HTTP 5xx, because the status request itself succeeded.

Unknown, expired, and unauthorized jobs.

• Return 404 for IDs that never existed.
• Return 410 Gone for jobs past retention, if you keep tombstones for them.
• Scope every lookup to the submitting user or tenant. Return 404 for jobs owned by someone else, so the response doesn't reveal that the job exists.

Cost. Some clients will ignore Retry-After. Serve status from a cache or a cheap indexed read, and rate-limit with 429 responses.

Webhooks or server push can reduce polling. Add them only alongside the pollable endpoint, never in place of it, because deliveries get lost and clients still need a way to recover.

Copyimport random
import time
from urllib.parse import urljoin

import requests


def poll_job(session, status_url, deadline_s=300, max_backoff_s=30.0):
    """Poll a job's status until it reaches a terminal state or the deadline passes."""
    deadline = time.monotonic() + deadline_s
    attempt = 0
    while time.monotonic() < deadline:
        # Handle the redirect ourselves so a 303 can be told apart from an embedded result.
        resp = session.get(status_url, allow_redirects=False, timeout=10)
        if resp.status_code == 303:
            result = session.get(urljoin(status_url, resp.headers["Location"]), timeout=30)
            result.raise_for_status()
            return result.json()
        if resp.status_code in (404, 410):
            raise LookupError(f"job unknown or expired: HTTP {resp.status_code}")
        if resp.status_code != 429:  # a 429 means back off and try again
            resp.raise_for_status()
            body = resp.json()
            if body["state"] == "succeeded":
                return body["result"]  # small result embedded in the status body
            if body["state"] == "failed":
                raise RuntimeError(body["error"])  # terminal failure, carried in the body
        # Retry-After is the minimum wait. Exponential backoff with jitter goes on top of it.
        try:
            floor = float(resp.headers.get("Retry-After", 1))
        except ValueError:
            floor = 1.0  # Retry-After was sent as an HTTP-date. Fall back to a 1 s floor.
        jitter = random.uniform(0, min(max_backoff_s, 2 ** attempt))
        remaining = deadline - time.monotonic()
        time.sleep(max(0.0, min(floor + jitter, remaining)))
        attempt += 1
    raise TimeoutError(f"job did not finish within {deadline_s}s")


# Usage: poll_job(requests.Session(), "")

A Python client polling loop that waits at least as long as Retry-After, adds jittered exponential backoff, stops at an overall deadline, and handles embedded results, 303 redirects, terminal failures, 404/410 and 429 responses.

Running workers and migrating an existing endpoint inside a monolith

Workers

Run workers as a separate process from the same codebase. They load the same models and validation rules, but they get their own entry point, deploy unit, and scaling. Open a fresh database session per job. A long-lived session can serve stale reads and miss updates made by the web tier.

Give every job a hard timeout. While a job runs, the worker holds a lease, such as a lease_expires_at column that it extends on each heartbeat. A sweeper requeues running jobs whose lease has lapsed, which recovers work from crashed or killed workers. During deploys, workers should stop claiming jobs on SIGTERM, finish or release in-flight work within the grace period, and let leases cover the rest.

Cap retry attempts. After the final attempt, mark the job failed and record the error in a dead-letter state. Otherwise a poison payload loops forever and starves healthy jobs.

Migration

Extract the business logic into one function that both paths call, so behavior can't drift between them. Add the async endpoint beside the old one, or switch behavior with a header or flag. Migrate clients one at a time while measuring real job durations. Retire the sync path once its traffic reaches zero.

Operations

Track these metrics:

• Queue depth
• Time in queue
• Run duration
• Failure and retry rates
• Count of jobs stuck in non-terminal states

Log the job ID alongside the originating request ID so you can trace a job back to the request that created it. Purge job records and results on a fixed retention schedule, because results often contain user data.

Choose the job runner only after these requirements are settled, and use them as your evaluation criteria.

When should you avoid async jobs, and what are the alternatives?

Async jobs carry real costs. Every client now has to handle a 202, store a job ID, poll, and read a second error channel. Shell scripts, plain HTML form posts, and third-party integrations built around one synchronous response often cannot adapt without rework. End-to-end latency can also get worse. A three-second operation that waits in a queue and is checked every five seconds may reach the user later than the slow synchronous version did. Failures that happen after the 202 live only in the job record, and many clients never read it, so errors disappear silently. You also now operate a second runtime with its own failure modes: stuck workers, queue backlog, and payloads that crash a worker on every retry.

Check the cheaper fixes first:

If you do adopt jobs, the build order affects development cost. Define the contract before you choose the runner, so that swapping the runner later does not force a client rewrite. Put the business logic in one function that both the synchronous path and the worker call. Idempotency checks then live in one place, fixes apply to both paths at once, and rolling back to synchronous execution is a routing change. Duplicating the logic for each path tends to cause duplicate-execution incidents.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to async job endpoint pattern, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.