Presigned URL Uploads: Moving Large Files Out of Your Monolith

What presigned URL uploads are and why proxying through the monolith hurts

A presigned URL upload lets the client write bytes directly to object storage. Your application never touches the file. Instead, it signs a storage request with its own credentials, scoped to one object key, one HTTP method and often a content type or size limit, and valid for a few minutes. The client sends the file to that URL. Storage checks the signature and accepts the write. Your app only handles small JSON requests before and after.

Proxying the upload through the monolith costs you in four ways:

• Held workers. A synchronous worker stays occupied for the full upload. The speed of the client's connection decides how long, not your code.
• Buffering. Frameworks often spool request bodies to memory or temp disk before your handler runs. Large files can exhaust RAM or fill /tmp.
• Limits in the path. Load balancers and reverse proxies enforce body-size caps and idle or request timeouts. You end up raising them for everyone.
• Double transfer. Every byte crosses the network twice, client to app and then app to storage, so you provision bandwidth and instance capacity for both legs.

Illustrative estimate

Assume 500 MB files, a 5 Mbps client uplink and a 32-worker pool.

• 500 MB is 4,000 megabits, so one upload takes about 800 seconds, roughly 13 minutes.
• Thirty-two such uploads running at once take every worker. Login and checkout requests then queue behind them.
• To keep all 32 workers busy, you only need one new upload starting every 25 seconds (800 s divided by 32).

The teams that benefit most run monoliths with fixed synchronous worker pools and accept user-generated media, documents or datasets.

How the upload lifecycle works

The flow has three steps, and only one of them touches the monolith. Even that step is a database write and a signature, not a byte copy.

1. The client requests an upload. It sends metadata only: original filename, declared content type, size, and what the file is for (an avatar, an invoice attachment). No file bytes cross this request.
2. The app authorizes and records. It checks that this user may upload this kind of file at this size. It generates the object key itself, such as a random ID under a tenant prefix, and never uses the client's filename as the key. It inserts an upload record in the pending state and returns a short-lived s3 presigned upload URL scoped to that single key.
3. The client writes directly to storage, then confirms. The bytes go straight to the bucket. Afterward, the client calls the app to report that the upload has finished.

This splits the system into a control plane and a data plane. The control plane covers auth, quotas, key generation and metadata. It stays in your app, where your existing middleware and audit logging already live. The data plane is the slow, bandwidth-heavy transfer, and it runs between the client and storage. Your worker pool never sees it.

Generating the URL is the easy part. The real work is the record's state machine: pending → uploaded → validated → available, with failed reachable from any step. Each transition needs an owner and a timeout:

• A record stuck in pending for an hour is an abandoned upload.
• A record stuck in uploaded means your validator is falling behind.

Nothing downstream should read an object until its record says available.

Copy CONTROL PLANE: auth, quotas, key generation, metadata (in the app)
 .........................................................................
 +--------+  1. request: name, type,  +-------------+     +--------------+
 |        |     size, purpose ------->|             |---->| upload table |
 |        |                           |  App        |     | state:       |
 | Client |<------ 2. presigned URL   | (monolith)  |     |   pending    |
 |        |        one key, short TTL |             |     +--------------+
 |        |                           |             |
 |        |  3b. confirm ------------>|             |
 +--------+                           +-------------+
     |
 .........................................................................
 DATA PLANE: bulk bytes flow client <-> storage only, never through app
     |
     | 3a. PUT file bytes to presigned URL
     v
 +--------------------------------+
 | Object storage bucket          |
 | key = <tenant>/<random-id>     |
 | (generated by app, not client) |
 +--------------------------------+

 RECORD STATE MACHINE (owned by the app; every step has an owner + timeout)
 pending ---> uploaded ---> validated ---> available
    |            |              |          (only state readers trust)
    +------------+--------------+--------> failed

The app handles authorization, key generation and the upload record's state machine in the control plane, while the client sends file bytes directly to storage over the data plane using a short-lived presigned URL.

Implementing presigned PUT and POST uploads safely

Start by inserting the pending record, then sign against it. The server generates the object key, for example uploads/{tenant_id}/{upload_id} with a random UUID. The key never comes from the client. Client-chosen keys let one user overwrite another user's object, smuggle ../ or odd Unicode into downstream paths, and guess neighbouring keys. A random server-side key ties each URL to exactly one database row.

With boto3, generate_presigned_url("put_object", ...) signs a single PUT for a fixed bucket, key and ContentType. generate_presigned_post returns a URL plus form fields governed by a policy document, and that policy can include conditions such as ["content-length-range", min, max].

A presigned PUT cannot enforce a size range. If size matters and you use PUT, you have to check size after the upload completes.

Sign with a dedicated identity whose only permission is to write objects under the upload prefix. It should not be able to read, delete or list. Keep expiry to minutes, not hours. The URL is a bearer token: anyone holding it can upload until it expires. Strip query strings from access logs, error trackers and frontend telemetry.

Clock skew is a common failure. If the signing host's clock drifts, or a slow client starts its request just after expiry, the storage service rejects the signature with a terse error the browser can't explain. Keep server clocks synced and log the expiry time server-side when you issue each URL.

Copyimport uuid
from datetime import datetime, timedelta, timezone

import boto3

# Client built from the dedicated signing identity: s3:PutObject on uploads/* only.
s3 = boto3.client("s3")
BUCKET = "example-app-uploads"
EXPIRES = 300  # seconds; keep it to minutes
MAX_BYTES = 50 * 1024 * 1024

def create_upload(db, log, tenant_id: str, content_type: str):
    upload_id = str(uuid.uuid4())
    key = f"uploads/{tenant_id}/{upload_id}"  # server-generated, never client-supplied
    expires_at = datetime.now(timezone.utc) + timedelta(seconds=EXPIRES)

    # 1. Pending record first, so every signed URL maps to exactly one row.
    db.insert_upload(id=upload_id, tenant_id=tenant_id, key=key, status="pending")

    # 2a. Presigned PUT: fixed key, expiry and Content-Type. No size range possible.
    put_url = s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
        ExpiresIn=EXPIRES,
    )

    # 2b. Presigned POST: the policy can also bound the object size.
    post = s3.generate_presigned_post(
        Bucket=BUCKET,
        Key=key,
        Fields={"Content-Type": content_type},
        Conditions=[{"Content-Type": content_type}, ["content-length-range", 1, MAX_BYTES]],
        ExpiresIn=EXPIRES,
    )

    # Log the id and expiry for debugging signature errors, never the URL itself.
    log.info("issued upload", extra={"upload_id": upload_id, "expires_at": expires_at.isoformat()})
    return {"upload_id": upload_id, "put_url": put_url, "post": post}

A boto3 sketch that inserts the pending record, builds a server-side key, and issues both a presigned PUT and a presigned POST with a content-length-range condition, logging only the upload ID and expiry time and never the URL.

Handling large file uploads with multipart upload

A single presigned PUT works until it doesn't: one dropped connection at 90% of a multi-gigabyte file means starting over. Multipart upload splits the object into independently uploaded parts. A failure then costs one part, not the whole file.

The flow

The app calls the storage service's create-multipart-upload operation and gets back an upload ID. It stores that ID alongside the file record and the expected size. The app then presigns an upload-part URL for each part number. The client uploads several parts in parallel and records the ETag returned for each one. When every part has succeeded, the client sends the part-number and ETag pairs to the app. The app calls complete with the list sorted by part number. Do this server-side so the app decides when an object becomes real.

On S3, every part except the last must be at least 5 MiB, and an upload can have at most 10,000 parts. Choose the part size from the expected file size rather than hardcoding it.

Resumability and expiry

If the client crashes or the network drops, the app lists the parts already uploaded for that upload ID. The client then sends only the missing parts. Retry a failed part on its own, with backoff, using the same part number.

Don't presign every part up front with a long expiry. Hand out short-lived URLs in small batches as the client needs them. A slow or resumed upload then just asks for fresh URLs. The upload ID stays valid until the upload is completed or aborted.

When a single PUT is enough

For files that typically finish in one short request, the extra round trips add complexity without much benefit. A common rule of thumb is to switch to multipart somewhere around 100 MB, or sooner if your users are on unreliable mobile connections.

Copyimport math
import boto3

s3 = boto3.client("s3")
BUCKET = "example-uploads-bucket"
MIB = 1024 * 1024

def choose_part_size(file_size):
    # S3: parts >= 5 MiB (except last), at most 10,000 parts per upload.
    needed = math.ceil(file_size / 10_000)
    return max(8 * MIB, math.ceil(needed / MIB) * MIB)

def start_upload(key, content_type):
    resp = s3.create_multipart_upload(Bucket=BUCKET, Key=key, ContentType=content_type)
    return resp["UploadId"]  # persist with the file record and expected size

def presign_parts(key, upload_id, part_numbers, expires=900):
    # Short-lived URLs in small batches; slow or resumed clients just ask again.
    return {
        n: s3.generate_presigned_url(
            "upload_part",
            Params={"Bucket": BUCKET, "Key": key, "UploadId": upload_id, "PartNumber": n},
            ExpiresIn=expires,
        )
        for n in part_numbers
    }

def uploaded_parts(key, upload_id):
    # For resume: tell the client which part numbers already succeeded.
    done, marker = {}, 0
    while True:
        resp = s3.list_parts(Bucket=BUCKET, Key=key, UploadId=upload_id, PartNumberMarker=marker)
        done.update({p["PartNumber"]: p["ETag"] for p in resp.get("Parts", [])})
        if not resp.get("IsTruncated"):
            return done
        marker = resp["NextPartNumberMarker"]

def finish_upload(key, upload_id, client_parts):
    # client_parts: [{"PartNumber": 1, "ETag": "..."}, ...]; S3 requires ascending order.
    parts = sorted(client_parts, key=lambda p: p["PartNumber"])
    s3.complete_multipart_upload(
        Bucket=BUCKET, Key=key, UploadId=upload_id, MultipartUpload={"Parts": parts}
    )

Server-side boto3 helpers that start a multipart upload, sizing the parts to stay within S3's limits, hand out short-lived per-part URLs in batches, list completed parts so an upload can resume, and complete the upload with the parts in ascending order.

Confirming completion and treating uploads as untrusted

When the client reports "upload finished," treat it as a hint, not a fact. A buggy or malicious client can send that call without uploading anything, after uploading a different file, or several times. The server must check storage itself before it changes any state.

On confirmation, issue a HEAD request for the object key stored in the pending record. Verify that the object exists, that its size matches the declared size (and falls within your limit), and that its content type matches what you signed for. The content type in the HEAD response is still whatever the client sent, so this is a consistency check, not a security check.

Storage event notifications are another way to trigger the same verification, and they still fire when the client disappears before confirming. These events are typically delivered at least once, so expect duplicates and out-of-order arrival.

Both triggers should run one idempotent transition. Use a conditional update such as "set state to available where id matches and state is pending." If zero rows change, another trigger already handled the upload, and you can return success without side effects.

Even a verified object is still untrusted content. Land uploads in a quarantine prefix or bucket. Run validation asynchronously: sniff magic bytes rather than trusting the extension, parse images or documents with hardened libraries, and run a virus scan. Promote the object only after it passes all checks.

When serving the file, set Content-Type explicitly, use Content-Disposition: attachment for anything you don't intend to render, and add X-Content-Type-Options: nosniff. Serve uploads from a separate domain. That way, a user-supplied HTML or SVG file cannot run script with your application's cookies.

Copyimport boto3
from botocore.exceptions import ClientError

s3 = boto3.client("s3")
QUARANTINE_BUCKET = "example-uploads-quarantine"  # uploads land here, never served directly
MAX_BYTES = 500 * 1024 * 1024

def confirm_upload(conn, upload_id):
    # The client's "done" call only triggers this. Storage events can call it too.
    with conn.cursor() as cur:
        cur.execute("SELECT object_key, declared_size, content_type FROM uploads"
                    " WHERE id = %s AND state = 'pending'", (upload_id,))
        row = cur.fetchone()
    if row is None:
        return "ok"  # unknown id, or another trigger already handled it
    key, declared_size, content_type = row

    try:
        head = s3.head_object(Bucket=QUARANTINE_BUCKET, Key=key)
    except ClientError as e:
        if e.response["Error"]["Code"] in ("404", "NoSuchKey", "NotFound"):
            return "not-uploaded"  # client said done but nothing is there
        raise
    size = head["ContentLength"]
    if size != declared_size or size > MAX_BYTES:
        return "size-mismatch"
    # Client-supplied value: a consistency check only. Real type comes from sniffing later.
    if head.get("ContentType") != content_type:
        return "type-mismatch"

    # Conditional on state, so duplicate confirmations and events change nothing.
    with conn.cursor() as cur:
        cur.execute("UPDATE uploads SET state = 'scanning'"
                    " WHERE id = %s AND state = 'pending'", (upload_id,))
        won = cur.rowcount == 1
    conn.commit()
    if won:
        # Only the winning trigger enqueues the async magic-byte check, parse and virus scan.
        # The worker moves state to 'available' only after the object passes every check.
        enqueue_scan(upload_id, key)  # defined elsewhere: your job queue
    return "ok"

A confirmation handler that checks the object with a HEAD request against the pending record, then uses a state-conditional UPDATE so that only one of the duplicate triggers moves the upload into asynchronous quarantine scanning.

Browser CORS, progress reporting, cleanup and observability

CORS

Browser uploads go straight to the bucket, so the bucket's CORS rules decide whether they succeed. Use a single rule. Set AllowedOrigins to your exact application origins rather than *. Set AllowedMethods to PUT and POST, and set AllowedHeaders to the headers your client signs, usually Content-Type. Add ETag to ExposeHeaders. Without it, browser JavaScript cannot read each part's ETag, and you need those values to call CompleteMultipartUpload. A modest MaxAgeSeconds caches preflight responses so each part doesn't trigger an extra OPTIONS round trip.

Progress reporting

fetch has no portable upload-progress event, so use XMLHttpRequest and listen to xhr.upload.onprogress. It reports loaded and total bytes. For multipart uploads, add up bytes across parts and count only completed parts toward the final total, because a failed part will be retried from zero.

Cleanup

Abandoned uploads still cost money. Add a lifecycle rule that aborts incomplete multipart uploads a day or two after they start, since their parts are billed but not visible in normal listings. Add a second rule that expires objects in the quarantine prefix that were never confirmed. Lifecycle rules don't touch your database, so also run a periodic reconciliation job. It finds pending records older than the presigned URL's expiry, checks storage for each one, and marks it failed or promotes it.

Observability

Your app servers no longer see upload failures. Expired signatures, CORS rejections and dropped connections all happen between the client and storage. To see them, report client-side errors with the upload ID attached, and enable storage access logs so you can correlate them.

S3-compatible providers differ in POST policy support, CORS behavior and event notifications. Check your provider's documentation before assuming any of the above works the same way.

When to avoid it, alternatives, and impact on cost and engineering time

Presigned uploads trade a simple request path for distributed state. You lose inline validation, so bad files reach storage before anything inspects them. Keeping the database and the bucket consistent becomes your job, which means writing a reconciliation process. You also add CORS rules, IAM policies, lifecycle rules and event pipelines that someone has to configure and monitor. Anyone holding a URL can use it until it expires. Multipart clients need retry and resume logic. Failures appear in storage access logs, not in your application logs.

When proxying is fine

Streaming through the app is often the correct choice when files are small, volume is low, or you must inspect the bytes before accepting them. Examples include malware scanning that gates the response, content transformation, and compliance checks that must pass before storage. Proxying only becomes a problem when uploads tie up workers long enough to starve other requests or push against body-size and timeout limits.

Engineering cost

The upfront work is real. You need CORS configuration, narrowly scoped credentials, lifecycle rules for abandoned multipart parts, and a reconciliation job. Plan it as a small project, not a single ticket.

In return, workers stop waiting on slow clients, you get fewer incidents from body limits and gateway timeouts, and upload capacity scales with the storage service instead of your app fleet. If you aren't seeing those incidents today, the migration can probably wait.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to presigned url uploads, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.