Presigned URL Uploads for Monoliths: Large Files Direct to S3

Why do file uploads through app servers become a scaling bottleneck?

When a user uploads a file to your monolith, the bytes travel through every layer you run: the load balancer, the reverse proxy, and an application worker. That worker is tied up for the entire transfer. A 2 GB video from a user on a slow connection can occupy a thread or process for many minutes. Most of that time is spent waiting on the network, not doing useful work. In a fixed-size worker pool, a handful of concurrent uploads can starve ordinary page loads and API calls.

Each layer also adds limits. Load balancers and reverse proxies enforce maximum request body sizes, idle timeouts and read timeouts. These defaults are tuned for normal requests, not long uploads. Raising them globally to fit uploads loosens protection for every other endpoint. Proxies that buffer request bodies spill large uploads to temporary disk. Frameworks that parse multipart bodies can hold parts in memory or write them to temp files again. A single upload may therefore be written to disk twice before your code ever sends it to S3.

Symptoms you have hit the limit

• Uploads fail intermittently with 413 or 504 errors, or connections reset partway through.
• Latency on unrelated endpoints spikes when several users upload at once.
• Temp directories fill up, or memory jumps during uploads.
• Autoscaling adds instances because workers are busy, even though CPU usage is low.

If these sound familiar, presigned URL uploads remove your servers from the data path. The client sends the bytes directly to S3, and your application only handles authorization and metadata.

What are presigned URL uploads and how does the direct upload flow work?

A presigned URL is an ordinary S3 request, such as a PUT to a specific bucket and key, that your server has already signed with its own credentials. The signature and an expiry time travel in the query string, so S3 can check the request without knowing who sent it. That makes the URL a bearer token: whoever holds it can perform that one operation until it expires. It also stops working early if the credentials that signed it expire or are revoked. Keep lifetimes short and scope each URL to a single object key.

The direct upload flow has five steps:

1. The client requests permission. It sends the monolith the filename, declared size and content type. No file bytes are sent yet.
2. The monolith authorizes the request and records it. It runs the same permission checks as any other endpoint and rejects disallowed types or sizes. It then inserts a pending upload row that ties a server-generated object key to the user and the owning entity.
3. The monolith returns upload instructions. This is either a presigned PUT URL or a presigned POST form (policy plus fields) for that key.
4. The client uploads directly to S3. Your servers never handle the byte stream.
5. Completion is confirmed. Either the client calls back to say the upload finished, or an S3 event notification reports the new object. The monolith then marks the row uploaded.

Authorization, validation and the system of record all stay in the monolith, inside your existing transactions and permission model. Only the transfer of bytes moves to S3, so you do not need to split out a separate upload service.

Copy +--------------+                               +------------------------+
 |    CLIENT    | (1) filename, size, type      |        MONOLITH        |
 |  (browser,   |------------------------------>|  authz + validation    |
 |   mobile)    |                               |  (2) create pending    |
 |              | (3) presigned PUT URL         |      row + object key  |
 |              |     or POST form + policy     |                        |
 |              |<------------------------------|  (5) mark uploaded     |
 |              |                               |                        |
 |              | (5a) client callback: done    |                        |
 |              |------------------------------>|                        |
 +--------------+                               +------------------------+
        |                                           |                 ^
        | (4) PUT/POST file bytes                   v                 |
        |     straight to S3                 +-------------+          |
        v                                    |  DATABASE   |          |
 +----------------------------+              | upload rows |          |
 |         S3 BUCKET          |              | pending ->  |          |
 | verifies signature, expiry |              | uploaded    |          |
 | stores object at the key   |              +-------------+          |
 +----------------------------+                                       |
        | (5b) S3 event notification: object created                  |
        +-------------------------------------------------------------+

The monolith authorizes the upload, records it as pending, and hands back a short-lived presigned PUT URL or POST form. The client sends the file bytes straight to S3, and completion is reported by either a client callback (5a) or an S3 event notification (5b).

Should you use a presigned PUT or a presigned POST for direct uploads?

Each option gives the server a different kind of control, and that difference determines when the server can reject a bad upload.

A presigned POST carries a signed policy document. S3 evaluates the policy when the upload arrives and rejects a request that breaks any condition. The policy can set a content-length-range, require Content-Type to start with image/, and restrict the key to a prefix such as uploads/user-123/. A 4 GB file sent to an avatar endpoint fails at S3 and never becomes an object.

A presigned PUT signs the method, the key, the expiry and any headers you choose to include, such as Content-Type. It has no way to express a size range. A client holding the URL can upload any size up to the single-request limit. You therefore need a check after the upload: a HEAD on the object, a size comparison, and deletion if the object is out of bounds.

Recommendation: Use POST for browser and SPA uploads of user-supplied files. FormData makes the form body easy to build, and the size and type limits are enforced before any bytes are stored. Use PUT for native mobile clients, internal tools and CLIs, where streaming a raw body is simpler, and pair it with a post-upload check. Large files will use presigned PUTs for each part, which the next section covers.

How do you implement presigned URL uploads safely in a monolith?

Start in the database, not in S3. Before issuing any URL, insert an uploads row with status pending, the owner, declared size and content type, and an object key the server builds itself, such as pending/{user_id}/{uuid}. That key is the only link between the row and the object. Never let the client choose it, or users can overwrite each other's files or write outside their prefix. The row moves to uploaded when the client reports completion and to verified only after the server has inspected the object.

The endpoint then signs a URL for exactly that key, with a short expiry of a few minutes. A URL is a bearer credential: anyone holding it can use it until it expires. Short lifetimes limit how long a leaked URL stays useful.

Scope the signer's credentials

The role that signs URLs should allow s3:PutObject only on arn:aws:s3:::your-bucket/pending/*. A presigned URL carries the signer's permissions, so a narrow policy caps what any URL can do, even if the signing code has a bug.

Configure CORS for browsers

Allow only your own origins, the PUT and POST methods, and the request headers you sign, such as Content-Type. Add ETag to ExposeHeaders. Without it, browser JavaScript cannot read the ETag that multipart uploads need when completing.

Move controls to issuance time

Upload bytes no longer pass through your app, so the URL request is your only enforcement point. Apply rate limits, per-user quotas and audit logging there, recording who requested which key, the declared size and the time.

Copyimport uuid
import boto3

s3 = boto3.client("s3", region_name="us-east-1")
BUCKET = "example-uploads-bucket"
URL_TTL_SECONDS = 300  # a leaked URL is only usable for 5 minutes
MAX_BYTES = 2 * 1024**3
USER_QUOTA_BYTES = 20 * 1024**3
ALLOWED_TYPES = {"image/png", "image/jpeg", "application/pdf"}

def issue_upload_url(db, rate_limiter, audit_log, user_id, size, content_type):
    # Bytes never reach this app, so every control runs here, at issuance.
    rate_limiter.check(f"upload-url:{user_id}")
    if content_type not in ALLOWED_TYPES or not 0 < size <= MAX_BYTES:
        raise ValueError("unsupported size or content type")
    used = db.scalar(
        "SELECT COALESCE(SUM(declared_size), 0) FROM uploads WHERE owner_id = %s",
        (user_id,))
    if used + size > USER_QUOTA_BYTES:
        raise PermissionError("quota exceeded")

    # The server builds the key; the client never chooses it.
    key = f"pending/{user_id}/{uuid.uuid4()}"
    # Record the pending row BEFORE any URL exists.
    db.execute(
        "INSERT INTO uploads (object_key, owner_id, declared_size, content_type, status)"
        " VALUES (%s, %s, %s, %s, 'pending')",
        (key, user_id, size, content_type))

    # Signed for exactly one key; the client must send the same Content-Type.
    url = s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
        ExpiresIn=URL_TTL_SECONDS)

    audit_log.info("upload_url_issued", user_id=user_id, key=key,
                   declared_size=size, ttl=URL_TTL_SECONDS)
    # Declared size is a claim; verify the real object size before 'verified'.
    return {"url": url, "key": key}

An upload-issuance function that enforces rate limits and quotas, inserts a pending row with a server-built key under pending/, signs a five-minute PUT URL for that single key with boto3, and writes an audit record.

How do you handle large files with S3 multipart presigned uploads?

A single presigned PUT works until files reach the hundreds of megabytes. Past that point, one dropped connection discards the whole transfer. Multipart upload splits the object into independently uploaded parts.

The flow

1. Start. The client asks your app to begin an upload. The server calls CreateMultipartUpload for a key it generates and stores the returned upload ID against the user's record.
2. Presign parts on demand. Presign UploadPart URLs in small batches as the client requests them, rather than all at once. A slow mobile client that receives every URL upfront can watch the later ones expire before it reaches them.
3. Upload and retry. The client uploads a few parts in parallel and retries failed parts individually. It records each part's ETag response header. Your bucket's CORS configuration must expose ETag, or browsers cannot read it.
4. Complete. The client sends the part numbers and ETags to the server. The server calls CompleteMultipartUpload itself.

Constraints and limits

S3 sets a minimum size for every part except the last. It also limits the number of parts per upload. Pick a part size large enough that your largest permitted file fits within that limit.

Resumability

Because the upload ID persists, an interrupted client can call ListParts, skip the parts S3 already has, and continue from there. On mobile and unreliable networks, this is often the difference between a slow upload and one that never finishes.

Enforce limits on the server

Store the declared size when you create the upload. Refuse to presign part numbers beyond what that size requires. After completion, check the object's actual size and delete it if it doesn't match.

Copyimport math, uuid
import boto3

s3 = boto3.client("s3")
BUCKET = "example-app-uploads"
PART_SIZE = 64 * 1024 * 1024   # above S3's 5 MiB minimum; at 10,000 parts this allows ~640 GiB
MAX_SIZE = 20 * 1024 ** 3       # app policy: 20 GiB, so at most 320 parts
MAX_BATCH = 20                  # presign a few URLs at a time so slow clients don't hit expiry

# `db` is your app's persistence layer (placeholder).
def start_upload(user_id, declared_size, content_type):
    if not 0 < declared_size <= MAX_SIZE:
        raise ValueError("size not allowed")
    key = f"uploads/{user_id}/{uuid.uuid4()}"  # server picks the key, never the client
    resp = s3.create_multipart_upload(Bucket=BUCKET, Key=key, ContentType=content_type)
    db.save_upload(resp["UploadId"], user_id=user_id, key=key, size=declared_size,
                   max_parts=math.ceil(declared_size / PART_SIZE))
    return {"upload_id": resp["UploadId"], "part_size": PART_SIZE}

def presign_parts(user_id, upload_id, part_numbers):
    up = db.get_upload(upload_id, user_id=user_id)  # lookup scoped to the owner
    if len(part_numbers) > MAX_BATCH or any(not 1 <= n <= up.max_parts for n in part_numbers):
        raise ValueError("part numbers outside the declared size")
    return {n: s3.generate_presigned_url("upload_part", ExpiresIn=900, Params={
                "Bucket": BUCKET, "Key": up.key, "UploadId": upload_id, "PartNumber": n})
            for n in part_numbers}

def complete(user_id, upload_id, parts):  # parts: [{"part": 1, "etag": "..."}, ...] from the client
    up = db.get_upload(upload_id, user_id=user_id)
    s3.complete_multipart_upload(Bucket=BUCKET, Key=up.key, UploadId=upload_id, MultipartUpload={
        "Parts": sorted(({"PartNumber": p["part"], "ETag": p["etag"]} for p in parts),
                        key=lambda p: p["PartNumber"])})
    actual = s3.head_object(Bucket=BUCKET, Key=up.key)["ContentLength"]
    if actual != up.size:  # the client lied about the size or sent short parts
        s3.delete_object(Bucket=BUCKET, Key=up.key)
        raise ValueError("uploaded size does not match declared size")
    db.mark_complete(upload_id)

A boto3 server sketch that starts a multipart upload, presigns UploadPart URLs in small batches limited to the part count implied by the declared size, then completes the upload and deletes the object if its actual size does not match.

How do you verify uploads, serve downloads and clean up orphaned objects?

Confirmation can come from two places. A client callback ("upload finished, here is the key") is simple, but the client may crash, lose connectivity or never call. S3 event notifications fire whenever an object is created, whether or not the client cooperates. Delivery is at-least-once and can be delayed, so make the handler idempotent. Many teams use both. The callback gives fast UI feedback, and the event catches anything the callback missed.

In either path, call HeadObject before trusting anything. Compare ContentLength with the size you recorded. The Content-Type that HeadObject returns is whatever the client sent, so treat it as untrusted. Read the first bytes of the object to detect the real format.

Leave the object in quarantine under pending/. A background job runs virus scanning or processing, then copies the object to its permanent prefix and marks the database row usable. S3 has no rename, so this step is a copy followed by a delete.

For cleanup, the lifecycle configuration shown below does two things:

• Abort incomplete multipart uploads after a few days. Their parts are stored and billed but are not visible in normal object listings.
• Expire objects under pending/ after a short window. This removes uploads that were never confirmed.

Lifecycle rules work in whole days and do not know about your database. Run a periodic reconciliation job that marks stale pending rows as failed.

Serve reads with short-lived presigned GET URLs, or through a CDN with signed URLs or cookies. Downloads then bypass your app servers just as uploads do.

Copy#!/usr/bin/env bash
set -euo pipefail

BUCKET="example-app-uploads"

# Two rules:
# 1. Abort multipart uploads that were never completed. Their parts are
#    billed but do not appear in normal object listings.
# 2. Expire anything still under pending/ after the confirmation window.
#    Confirmed objects have already been copied to their permanent prefix,
#    so only abandoned or unconfirmed uploads remain here.
#    Keep this window longer than your slowest scan or processing job.
cat > lifecycle.json <<'EOF'
{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart",
      "Filter": { "Prefix": "" },
      "Status": "Enabled",
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 3 }
    },
    {
      "ID": "expire-unconfirmed-pending",
      "Filter": { "Prefix": "pending/" },
      "Status": "Enabled",
      "Expiration": { "Days": 2 }
    }
  ]
}
EOF

# Replaces the bucket's entire lifecycle configuration. Merge with any existing rules first.
aws s3api put-bucket-lifecycle-configuration \
  --bucket "$BUCKET" \
  --lifecycle-configuration file://lifecycle.json

# Confirm what S3 actually stored.
aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET"

A lifecycle configuration that aborts incomplete multipart uploads across the bucket and expires unconfirmed objects under pending/, applied and verified with the AWS CLI.

When should you avoid presigned URL uploads, and what are the alternatives?

Presigned uploads move complexity rather than remove it. The client now runs a protocol: request a URL, upload, retry, and confirm. Each step can fail independently. Validation of content type, magic bytes and malware can only happen after the bytes land. The URL is a bearer credential, so it can leak through access logs, analytics events or a Referer header and be replayed until it expires.

The database row and the object can disagree until reconciliation runs. A presigned POST can enforce a size range, but a presigned PUT or a multipart part cannot easily cap total size, so you check it afterwards. Short expirations limit replay, but they also fail when the signing server's clock drifts or a slow client starts late. Long expirations widen the leak window. You also couple your code to S3 semantics, which means running an S3-compatible emulator locally. Any auth, quota or audit middleware that used to wrap the upload route has to be rebuilt as checks before signing and after confirmation.

For most monoliths, presigned uploads are a contained change: one signing endpoint, one confirmation endpoint and a cleanup job. That is far less engineering time than extracting an upload service. Choose the proxy if inline validation is non-negotiable.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to presigned url uploads, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.