Presigned URL File Upload: Offload Uploads from Your App

Q: How OneEnv can help?

Disclosure: OneEnv is the product behind this blog. OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Why proxying uploads through the monolith breaks down at scale

When a monolith accepts file uploads directly, every byte travels from the client through the load balancer and into an application process before it reaches storage. That works at low volume, but several problems grow with traffic.

Worker occupancy. A user on a slow mobile connection uploading a 200 MB video ties up a request worker or thread for the entire transfer. Most of that time the worker does nothing but wait on the network. Prefork servers with a fixed worker pool feel this first: a handful of slow uploads can starve ordinary page requests.

Memory and temp-disk pressure. Frameworks buffer request bodies in RAM or spool them to local temp files. Many concurrent large uploads can exhaust either resource and take down unrelated requests on the same host.

Proxy limits and timeouts. Load balancers and reverse proxies enforce maximum body sizes and idle or request timeouts. Raising those limits to fit uploads also loosens them for every other endpoint.

Paying for bandwidth twice. The file enters your application tier and then leaves it again on the way to object storage. You pay for transfer and compute on both legs.

Presigned URL upload offloading removes the application from the data path. The monolith authenticates the user, validates intent, and signs a short-lived URL. The client then sends the bytes straight to object storage. The monolith remains the control plane, deciding who can upload what and where. Storage handles the bytes. You don't need to split anything into separate services.

How the presigned upload flow works

The flow has three steps. Each one crosses a trust boundary.

1. The client requests an upload intent

The client sends the monolith metadata only: filename, declared content type, declared size, and what the file is for (an avatar, an invoice attachment). The monolith authenticates the user, checks permissions and quotas, and writes a pending upload record. It generates the object key itself, typically from the record ID. Clients never choose keys, so they cannot overwrite another user's object or write outside their prefix.

2. The monolith returns a presigned PUT URL or POST form

The signature is computed locally with the monolith's credentials, so issuing a URL needs no network call to storage. A presigned PUT binds the key, the HTTP method, an expiry, and optionally the content type. A presigned POST policy can also enforce a content-length range, which is the better choice when size limits matter. Keep expiries short. Anyone holding the URL can use it until it expires.

3. The client uploads, then completion is signaled

The bytes go straight to the bucket. Afterward, either the client calls the monolith to say it's done, or the bucket emits an object-created event to a queue or webhook.

Neither signal is reliable. The client might close the tab after the upload succeeds, or it might call "complete" when nothing was uploaded. Storage events are usually delivered at least once, so duplicates and delays are expected.

The monolith should therefore treat any completion signal as a hint. It verifies the object with a HEAD request, compares the actual size and type against the pending record, and moves the record from pending to complete with a conditional update. Calling finalization twice, or calling it from both paths, must produce the same result.

+------------------+  (1) intent: name, type,   +--------------------------+
| CLIENT           |      size, purpose         | MONOLITH (trusted)       |
| (untrusted)      |--------------------------->| authn, authz, quota      |
|                  |                            | write PENDING record     |
|                  |  (2) presigned PUT URL or  | generate object key      |
|                  |      POST form, short TTL  | sign locally (no I/O)    |
|                  |<---------------------------|                          |
|                  |                            |--------------------------|
|                  |  (4a) "complete" (hint)    | FINALIZE (idempotent)    |
|                  |--------------------------->| HEAD object              |
|                  |                            | compare size/type        |
+------------------+                            | PENDING -> COMPLETE      |
         |                                      | via conditional update   |
         | (3) PUT/POST bytes directly          +--------------------------+
         |     (bypasses monolith)                            ^
         |                                                    |
         v                                                    |
+------------------------------------+   (4b) object-created  |
| OBJECT STORAGE BUCKET              |        event (hint,    |
| checks signature, expiry, key,     |        at-least-once)  |
| POST policy content-length range    |------------------------+
+------------------------------------+

Presigned upload flow: the monolith authorizes and signs, bytes go straight to the bucket, and both completion signals are hints that trigger an idempotent, verified finalize.

Implementing it: data model, signing, and bucket CORS

The uploads table

Insert the row before signing anything. It records the owner, the declared size and content type, and a server-generated object key such as tenants/{tenant_id}/uploads/{uuid}. Never derive the key from the client's filename. Store the original name as metadata only. Rows move through four states: pending when the URL is issued, uploaded when the object exists, verified once checks pass, and rejected otherwise. Because the declared size exists before any bytes do, finalization has something to compare against.

Signing

Keep expiry short, a few minutes, so a leaked URL has limited value. Sign with a dedicated role whose write permission covers only the upload prefix. A presigned URL carries the signer's permissions, so this caps the damage from a key-construction bug. Presigned URLs are bearer credentials. Redact them from application logs, proxy logs, and error trackers.

	Presigned PUT	Presigned POST policy
Size limit	Cannot enforce a range	`content-length-range` condition
Key control	Fixed by the signature	Exact-match condition on `key`
Content type	Can be bound as a signed header	Policy condition
Client request	Raw body	Multipart form

Use POST when you need a hard size cap at the storage layer. With PUT, size is only checked at finalization.

Bucket CORS

Allow only your application's exact origins and only the methods you use (PUT or POST). Allow every header the client sends, including any signed headers such as Content-Type. Expose ETag in the response headers. Browsers hide it from JavaScript otherwise, and multipart completion needs it.

import uuid
import boto3

BUCKET = "example-app-uploads"
MAX_BYTES = 25 * 1024 * 1024  # hard cap enforced by S3, not just at finalization

# Client built from credentials of a role whose s3:PutObject covers only tenants/*/uploads/*
s3 = boto3.client("s3")

def issue_upload(db, tenant_id, user_id, filename, declared_size, content_type):
    if declared_size <= 0 or declared_size > MAX_BYTES:
        raise ValueError("declared size out of range")

    # Server-generated key; the client's filename is stored as metadata only
    key = f"tenants/{tenant_id}/uploads/{uuid.uuid4()}"

    # Record owner, size and type before any URL exists
    db.execute(
        "INSERT INTO uploads (object_key, tenant_id, owner_id, original_name,"
        " declared_size, content_type, state) VALUES (%s, %s, %s, %s, %s, %s, 'pending')",
        (key, tenant_id, user_id, filename, declared_size, content_type),
    )

    post = s3.generate_presigned_post(
        Bucket=BUCKET,
        Key=key,  # boto3 also adds an exact-match condition on this key
        Fields={"Content-Type": content_type},
        Conditions=[
            {"Content-Type": content_type},
            # Reject bodies larger than the declared size at the storage layer
            ["content-length-range", 1, declared_size],
        ],
        ExpiresIn=300,  # five minutes limits the value of a leaked policy
    )

    # post["url"] and post["fields"] are bearer credentials: return them, never log them
    return {"object_key": key, "url": post["url"], "fields": post["fields"]}

Inserting a pending uploads row with a server-generated, tenant-prefixed key, then issuing a short-lived presigned POST policy with boto3 that pins the key and content type and caps the size with content-length-range.

Finalizing, serving, and cleaning up uploads

The client signals completion by calling a finalize endpoint with the upload ID. Never trust that call alone. Issue a HEAD on the object key to confirm it exists and that its Content-Length is within the limit you recorded when signing. Alternatively, drive finalization from a bucket event notification so a client that crashes after uploading still gets processed. Then fetch the first few kilobytes with a ranged GET and sniff the magic bytes. The Content-Type the client declared describes what it claimed, not what it sent.

Validated objects move from the quarantine prefix to a trusted prefix. Object stores generally have no rename, so this is a copy followed by a delete. Tagging the object in place avoids the copy, but every read path must then check the tag.

Scanning and image processing run after this step, so a file can exist in storage without being trusted yet. Model this as explicit states on the record: pending, uploaded, processing, ready, rejected. Serve only ready files, and have the UI show "processing" rather than a broken link.

Serve files through short-lived presigned GET URLs or a CDN with signed access. Set Content-Disposition: attachment and X-Content-Type-Options: nosniff so uploaded HTML or SVG downloads instead of rendering inline. Ideally, serve from a separate domain as well.

For cleanup, run a reconciliation job that expires pending records older than the signing TTL plus a grace period and deletes any matching objects. Back it with a lifecycle rule on the quarantine prefix that expires objects after a few days and aborts incomplete multipart uploads.

import boto3
from botocore.exceptions import ClientError

s3 = boto3.client("s3")
BUCKET = "example-uploads"
# Magic-byte prefixes for the types this endpoint accepts.
MAGIC = {b"\x89PNG\r\n\x1a\n": "image/png", b"\xff\xd8\xff": "image/jpeg", b"%PDF-": "application/pdf"}

def finalize(upload):
    """Called from the finalize endpoint or a bucket event; never trusts the client."""
    src = upload.quarantine_key  # e.g. "quarantine/<upload_id>"
    try:
        head = s3.head_object(Bucket=BUCKET, Key=src)
    except ClientError:
        return upload.set_state("pending")  # not there yet; reconciliation will expire it
    # Enforce the size limit recorded at signing time, using the stored size.
    if head["ContentLength"] > upload.max_bytes:
        return reject(upload, src)
    # Sniff the real type from the first bytes, ignoring the declared Content-Type.
    first = s3.get_object(Bucket=BUCKET, Key=src, Range="bytes=0-4095")["Body"].read()
    sniffed = next((t for sig, t in MAGIC.items() if first.startswith(sig)), None)
    if sniffed is None or sniffed not in upload.allowed_types:
        return reject(upload, src)
    # No rename in object storage: copy to the trusted prefix, then delete.
    # Single-request copy_object is limited to 5 GB; larger objects need multipart copy.
    dst = f"trusted/{upload.id}"
    s3.copy_object(
        Bucket=BUCKET, Key=dst, CopySource={"Bucket": BUCKET, "Key": src},
        MetadataDirective="REPLACE", ContentType=sniffed,
        ContentDisposition="attachment",  # download, never render inline
    )
    s3.delete_object(Bucket=BUCKET, Key=src)
    # Exists but not trusted yet: scanning/image processing moves it to "ready".
    upload.set_state("processing", key=dst, content_type=sniffed)

def reject(upload, key):
    s3.delete_object(Bucket=BUCKET, Key=key)
    upload.set_state("rejected")

A boto3 finalize handler that checks existence and real size with HEAD, sniffs magic bytes from a ranged GET, then copies the object to a trusted prefix with Content-Disposition: attachment and marks it processing.

Large files: multipart uploads with per-part presigned URLs

A single presigned PUT works until uploads get slow. A 4 GB video on a hotel connection can take longer than any reasonable URL lifetime, and the signature is checked when the request starts, so a long transfer that starts on time can still lose its connection and need a retry after expiry. Clock skew makes this worse. Validity is measured from the timestamp embedded at signing, so an app server whose clock runs ahead or behind can issue URLs that are effectively shorter than intended or rejected as not yet valid. Keep NTP healthy on signing hosts. Do not shrink TTLs so far that a few minutes of drift breaks them.

Multipart upload fixes both problems by splitting the risk into parts. The flow:

Initiate server-side. The monolith calls CreateMultipartUpload, stores the returned upload ID on the upload record, and picks a part size. On S3, parts must be at least 5 MiB except the last, and an upload can have at most 10,000 parts. Size large files accordingly.
Sign per part. Issue a short-lived URL for each part number, in batches as the client asks for them, not all up front.
Re-sign on expiry. When a part fails with an expired-signature error, the client requests a fresh URL for that part number and retries only that part.
Collect ETags. Each part response returns an ETag header. Your bucket CORS config must expose ETag, or the browser cannot read it.
Complete server-side. The client sends the part number and ETag pairs to the monolith. The monolith validates them against the record and calls CompleteMultipartUpload.

Parts from uploads that are never completed or aborted are invisible in normal listings but stay in storage and are billed. Add a lifecycle rule that aborts incomplete multipart uploads a day or two after initiation.

import boto3

s3 = boto3.client("s3")
BUCKET = "example-uploads-bucket"
PART_TTL = 900  # 15 min per part; generous enough to absorb minor clock drift

def initiate(key):
    # Server-side: store UploadId on the upload record; the client never creates uploads
    resp = s3.create_multipart_upload(Bucket=BUCKET, Key=key)
    return resp["UploadId"]

def sign_parts(key, upload_id, part_numbers):
    # Sign only the batch the client asked for; call again for a part whose URL expired
    return {
        n: s3.generate_presigned_url(
            "upload_part",
            Params={"Bucket": BUCKET, "Key": key, "UploadId": upload_id, "PartNumber": n},
            ExpiresIn=PART_TTL,
        )
        for n in part_numbers
    }

def complete(key, upload_id, parts):
    # parts: [{"PartNumber": 1, "ETag": '"abc..."'}, ...] after validating against the record
    parts = sorted(parts, key=lambda p: p["PartNumber"])
    s3.complete_multipart_upload(
        Bucket=BUCKET, Key=key, UploadId=upload_id,
        MultipartUpload={"Parts": parts},
    )

# One-time setup: abort uploads never completed so orphaned parts stop being billed.
# Note: this call replaces the bucket's entire lifecycle configuration.
s3.put_bucket_lifecycle_configuration(
    Bucket=BUCKET,
    LifecycleConfiguration={"Rules": [{
        "ID": "abort-incomplete-multipart",
        "Filter": {"Prefix": "uploads/"},
        "Status": "Enabled",
        "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 2},
    }]},
)

A boto3 sketch of the server side: initiating a multipart upload, signing per-part URLs on demand, completing with client-reported ETags, and a lifecycle rule that aborts incomplete uploads.

Who should adopt this, when to avoid it, and the alternatives

The pattern pays off when the monolith's upload path is under strain: upload volume is growing, files are large, or clients are slow. Slow mobile connections hold a request worker for the whole transfer, so a few hundred concurrent uploads can exhaust a worker pool that handles thousands of ordinary requests. Presigned uploads move that waiting to the storage service.

Avoid it if your volume is low and the proxy is not causing problems, because you add CORS configuration, a finalize step, and orphan cleanup for no gain. Avoid it if you must inspect bytes in-stream before they are stored, such as for compliance scanning or rejection on content, because the server never sees the payload until afterward. Hold off if your team has no local emulator or test bucket. Signing and CORS bugs show up only against a real endpoint.

"S3-compatible" does not mean identical. Before you commit, check whether your store supports POST policies, checksum headers, lifecycle rules that abort incomplete multipart uploads, and its multipart part-size limits.

Option	Fits when	Trade-off
Proxied upload with streaming and tuned limits	Moderate volume, inline inspection needed	Workers still held for the full transfer
Dedicated upload service	Many apps share upload logic	Another service to deploy and run
Resumable upload protocol	Unreliable networks, very large files	Client and server protocol support
Managed upload widget	Fast delivery, standard needs	Vendor coupling, less control

Migration path and impact on cost and engineering time

Ship the new flow behind a feature flag, scoped per tenant or per client version, while the proxied endpoint keeps serving everyone else. Both paths should write the same upload record, so downstream code never needs to know which path produced a file. Route every read through a single storage abstraction in the monolith that resolves a record to its location, whether that is a legacy disk path or an object key. You can then backfill old files with a background job, or leave them in place and keep them readable. Delete the proxied endpoint only after flag-off traffic has dropped to near zero.

Cost you take on	What it buys back
CORS rules per environment	Upload bytes no longer pass through app servers
Client upload and retry logic	Workers are not held open by slow clients
Finalize callback and reconciliation job	Fewer replicas, or smaller ones
Observability moved to client error reports and storage access logs or metrics	Fewer incidents from exhausted connections or full disks

Illustrative estimate (assumed numbers, not a benchmark). Worker-hours freed per month = uploads per month × average upload duration ÷ 3,600. Assume 300,000 uploads per month at an average of 40 seconds each: 300,000 × 40 ÷ 3,600 ≈ 3,333 worker-hours. Payback period = build cost ÷ (monthly compute savings + monthly incident time saved). Plug in your own engineer-week rate and incident history. In practice, incident time often outweighs the compute savings.

Keeping this inside the monolith saves real engineering time. Signing is a few endpoints that reuse your existing authentication, permission checks, and data model. A separate upload service would add a new deployable, service-to-service authentication, an extra network hop, and another on-call rotation.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to presigned url file upload, this is where it helps:

Saves time

Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
Less duplicated configuration to maintain: one shared definition inherited by many services.
A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.

Presigned URL File Upload: Offload Monolith Uploads to Object Storage

Why proxying uploads through the monolith breaks down at scale

How the presigned upload flow works

1. The client requests an upload intent

2. The monolith returns a presigned PUT URL or POST form

3. The client uploads, then completion is signaled

Implementing it: data model, signing, and bucket CORS

The uploads table

Signing

Bucket CORS

Finalizing, serving, and cleaning up uploads

Large files: multipart uploads with per-part presigned URLs

Who should adopt this, when to avoid it, and the alternatives

Migration path and impact on cost and engineering time

How OneEnv can help

Saves time

Saves cost

Sources

Presigned URL Uploads for Monoliths: Large Files Direct to S3

Presigned URL Uploads: Moving Large Files Out of Your Monolith

How to Migrate File Uploads to S3 in a Monolith Without Downtime

Why proxying uploads through the monolith breaks down at scale

How the presigned upload flow works

1. The client requests an upload intent

2. The monolith returns a presigned PUT URL or POST form

3. The client uploads, then completion is signaled

Implementing it: data model, signing, and bucket CORS

The uploads table

Signing

Bucket CORS

Finalizing, serving, and cleaning up uploads

Large files: multipart uploads with per-part presigned URLs

Who should adopt this, when to avoid it, and the alternatives

Migration path and impact on cost and engineering time

How OneEnv can help

Saves time

Saves cost

Sources

Keep reading

Presigned URL Uploads for Monoliths: Large Files Direct to S3

Presigned URL Uploads: Moving Large Files Out of Your Monolith

How to Migrate File Uploads to S3 in a Monolith Without Downtime