What is an S3 presigned URL upload, and why not proxy files through app servers?
An S3 presigned URL is a normal S3 request, such as "PUT this object at this key," that your application signs with its own AWS credentials and hands to a client. The signature encodes the operation, the bucket and key, and an expiry time. Anyone holding the URL can perform that one request until it expires, with no AWS credentials of their own. A direct-to-S3 upload builds on this. The browser or mobile client asks your app for a URL, then sends the file bytes straight to S3.
The alternative is proxying uploads through the monolith, which costs more than it first appears:
- Held capacity. A user on a slow connection uploading a 2 GB video holds a worker process or thread for the entire transfer. A few dozen of these can starve ordinary page requests.
- Limits and timeouts. Reverse proxies and load balancers enforce request body size caps and idle or request timeouts. Raising them for one upload route weakens protection everywhere else.
- Memory and disk pressure. Frameworks often buffer bodies in memory or spill them to temp disk before your code sees them.
- Double transfer. Every byte crosses your network twice, once inbound from the client and again outbound to S3.
With presigned URLs, the upload bytes leave your app servers, but the monolith still makes every decision. It authorizes the request, picks the key, sets constraints, validates the result, and records the file. This suits teams running a monolith that accepts user files and don't want to build a separate upload service. It pays off once uploads are large or frequent enough to tie up real app capacity.
Who should use S3 presigned URL uploads?
Presigned uploads fit teams that meet most of these conditions:
- You run a monolith, or a small number of services, that already stores files in S3. The pattern adds one or two endpoints to the existing app rather than a new deployable.
- Users upload files that are large relative to your normal request size. Videos, raw images, datasets, backups, design files and long PDFs are typical. Megabyte-to-gigabyte uploads are where held workers and proxy limits start to hurt.
- Clients are browsers or mobile apps you control. You can ship the client code that requests a URL, performs the PUT or POST, and reports completion.
- Post-upload validation is acceptable. Your product can show a "processing" state while a background job checks type, scans for malware and generates derivatives.
- You want the monolith to remain the authority. Permissions, tenancy, key naming and file state all stay in the app and its database, not in a separate system.
It is a poor fit for teams whose clients cannot reach S3 endpoints directly, whose compliance rules require inspecting bytes before anything is stored, or whose uploads are small and rare enough that proxying causes no measurable pressure. The final section covers these cases in more detail.
How does it work?
A presigned URL is an ordinary S3 request with its authentication moved from headers into the query string. When your app calls an SDK method such as boto3's generate_presigned_url, the SDK computes an AWS Signature Version 4 signature locally, using the credentials the app already has. No network call to AWS is made at signing time. The resulting URL carries query parameters including:
X-Amz-Algorithm, the signing algorithm.X-Amz-Credential, the access key ID plus the date, region and service scope.X-Amz-Date, the time the URL was signed.X-Amz-Expires, how many seconds the URL stays valid after that time.X-Amz-SignedHeaders, the headers the client must send with the exact signed values, such ashostand, if you included it,content-type.X-Amz-Security-Token, present when the signer used temporary credentials.X-Amz-Signature, the HMAC computed over the method, path, query parameters and signed headers.
When the request arrives, S3 recomputes the signature from what the client actually sent and compares it. If the method, key, signed headers or any signed parameter differ, or if the current time is past X-Amz-Date plus X-Amz-Expires, S3 rejects the request. S3 also evaluates the request against the signer's IAM permissions at that moment, so a URL can never do more than the signing identity is allowed to do. Revoking or narrowing that identity's permissions invalidates URLs it has already issued.
A POST policy works the same way at the signature level, but the thing being signed is a base64-encoded JSON policy document listing conditions. The client submits that policy and its signature as form fields alongside the file, and S3 checks both the signature and every condition before writing the object.
The important consequence for a monolith is that signing is cheap and stateless. Issuing a URL costs a few cryptographic operations in your process, not a held connection, so the endpoint that hands out URLs behaves like any other short API request.
How does a direct-to-S3 upload flow work end to end?
The flow has five steps. Only the last one decides whether a file exists as far as your application is concerned.
- The client asks for permission. It sends the filename, declared size and content type to an endpoint on your monolith. No file bytes are sent yet.
- The app authorizes and signs. The server checks that the user may upload, rejects disallowed sizes or types, and generates the object key itself, for example from a tenant ID and a random UUID. Never accept a client-supplied path. The app then inserts a database row in the
pendingstate and returns a short-lived presigned URL bound to that key. - The client uploads straight to S3. The bytes go from the browser or mobile app to the bucket. Your app servers never handle the payload, so slow connections no longer hold your workers or request threads.
- The client reports completion. It calls back with the upload ID, and the row moves to
uploaded. - The app verifies. The server checks the object in S3, including its existence, actual size and content type. It then marks the row
verified, orfailedif something doesn't match.
The client's "done" message is only a claim. A client can crash before reporting, lie about what it sent, or upload a different file than the one it declared. The source of truth is the server-side check in step 5. Your application should serve, attach or process a file only when its row is verified, and treat every other state as not yet usable.
+----------+ +----------+ +----------+ +----------+
| Client | | Monolith | | Database | | S3 |
|(browser) | | app | | (rows) | | bucket |
+----------+ +----------+ +----------+ +----------+
| | | |
| 1 name,size,type | | |
|-------------------->| | |
| | 2 INSERT 'pending' | |
| |-------------------->| |
| 2 presigned PUT URL | | |
|<--------------------| | |
| 3 PUT bytes directly to S3 (app never sees payload) |
|-------------------------------------------------------------->|
| 4 done, upload ID | | |
|-------------------->| | |
| | 4 set 'uploaded' | |
| |-------------------->| |
| | 5 HEAD object: exists? size? type? |
| |---------------------------------------->|
| | metadata (size, type) |
| |<----------------------------------------|
| | 5 verified / failed | |
| |-------------------->| |
| | | |
+--------------------------------------------------------------------------+
| Step 4 is only the client's claim. Step 5 is the source of truth: |
| serve, attach or process a file only when its row is 'verified'. |
+--------------------------------------------------------------------------+
The five-step direct-to-S3 upload flow: the monolith signs the request and tracks each upload's state, the file bytes go straight to S3, and only the server-side check moves a row to 'verified' or 'failed'.
Should you use a presigned PUT or a presigned POST policy?
A presigned PUT and a presigned POST policy both let the client upload without your credentials, but they differ in what S3 will refuse at upload time.
A presigned PUT signs a specific request: method, bucket, key, expiry, and any headers you choose to include in the signature, such as Content-Type. S3 checks that the request matches what was signed and nothing more. A PUT has no way to express a range. You cannot say "between 1 KB and 20 MB" or "any key under uploads/user-42/."
A POST policy is a signed JSON document of conditions that S3 evaluates against a multipart/form-data upload. Conditions include content-length-range, starts-with on $key, and exact or prefix matches on fields like Content-Type. A request that violates any of them is rejected before an object is written.
| Concern | Presigned PUT | Presigned POST policy |
|---|---|---|
| Size limit enforced by S3 | No, only the single-request maximum | Yes, via content-length-range |
| Key constraint | One exact key | Exact key or prefix |
| Client format | Raw body, simple to use from any HTTP client | HTML form fields plus file |
| Works with multipart upload | Yes, per part | No |
Decision rule:
- List the constraints S3 must reject at upload time. These are the ones whose violation costs you money or capacity, such as size and key location.
- List the constraints the app can check after upload, such as file type by content sniffing, malware scanning, and image dimensions.
- If the first list includes size or a key prefix, use POST. Otherwise, PUT is simpler.
Be careful with an unconstrained PUT. If the URL leaks or a client is malicious, anyone holding it can push objects up to S3's single-request limit until the URL expires. If the bucket has versioning enabled, each overwrite is retained as a separate version, and you pay to store every one. Short expiries limit the window, but they do not limit the size of each upload.
How do you generate presigned URLs and configure CORS safely?
The endpoint that signs the URL is where those limits are actually enforced. It should authenticate the user, check that they may upload to this resource, and build the object key itself, for example tenants/{tenantId}/uploads/{uuid}. Never accept a path from the client. Then call a standard SDK method such as boto3's generate_presigned_url for put_object, with an expiry of a few minutes. If the content type is included in the signature, the client must send that exact Content-Type header.
Sign with a dedicated role that can only call s3:PutObject on that tenant prefix. A presigned URL can do no more than its signer could, so a narrow role limits the damage from a leaked URL. A presigned URL is a bearer token: anyone who holds it can use it. Keep it out of access logs, analytics events, error reporters and browser history. The URL also stops working when the signing credentials expire. A URL signed with temporary role credentials dies when the session does, whatever expiry you requested.
CORS
Configure CORS on the bucket as follows:
- Allowed origins: your exact app origins, not
*. - Allowed methods:
PUT, orPOSTif you use POST policies. - Allowed headers: the headers you sign, such as
Content-Type. - Expose headers:
ETag, which the browser needs to complete a multipart upload.
curl does not enforce CORS, which is why an upload can succeed from curl and fail in the browser.
S3 checks expiry against the time the server signed the URL, so keep server clocks NTP-synced. The check happens when S3 receives the request, so a slow upload only has to start in time. For long multipart uploads, issue part URLs on demand rather than all at once with a long expiry.
import uuid
import boto3
from flask import Flask, abort, jsonify, request
app = Flask(__name__)
BUCKET = "example-app-uploads"
ALLOWED_TYPES = {"image/png", "image/jpeg", "application/pdf"}
s3 = boto3.client("s3", region_name="eu-west-1")
@app.post("/uploads/presign")
def presign_upload():
user = current_user() # your existing auth; returns None if unauthenticated
if user is None or not user.can_upload():
abort(403)
content_type = (request.get_json(silent=True) or {}).get("contentType")
if content_type not in ALLOWED_TYPES:
abort(400)
# Server builds the key under the tenant prefix; no client-supplied path is used.
key = f"tenants/{user.tenant_id}/uploads/{uuid.uuid4()}"
url = s3.generate_presigned_url(
"put_object",
Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
ExpiresIn=300, # a few minutes; S3 only checks this when the request arrives
)
# Return the URL in the body only. Do not log it: it is a bearer token.
return jsonify({"url": url, "key": key, "contentType": content_type})
def configure_cors():
s3.put_bucket_cors(Bucket=BUCKET, CORSConfiguration={"CORSRules": [{
"AllowedOrigins": [""], # exact origins, not *
"AllowedMethods": ["PUT"],
"AllowedHeaders": ["Content-Type"], # headers the browser sends and you sign
"ExposeHeaders": ["ETag"], # browser JS needs ETag per part for multipart
"MaxAgeSeconds": 3000,
}]})
A Flask endpoint that authorizes the user, builds a tenant-scoped object key on the server, and returns a five-minute boto3 presigned PUT URL, plus the matching bucket CORS rule that allows exact origins and exposes ETag for multipart uploads.
How do you handle large file uploads with S3 multipart upload?
Above a certain size, a single PUT becomes fragile. One dropped connection at 90% means starting over, and a single request can only carry 5 GB. Multipart upload splits the file so each piece succeeds or fails on its own.
The flow
- The client asks your app to start an upload. The app calls
CreateMultipartUpload, stores the returnedUploadIdagainst the user and object key, and returns the ID. - For each part, the app presigns an
UploadPartrequest bound to thatUploadIdand part number. Part numbers run from 1 to 10,000. Every part except the last must be at least 5 MiB. - The client PUTs each chunk to its URL, in parallel if bandwidth allows, and records the
ETagheader from each response. Your CORS configuration must listETagunder exposed headers, or browser JavaScript cannot read it. - The client sends the part numbers and ETags to the app. The app calls
CompleteMultipartUpload, or callsAbortMultipartUploadif the user cancels.
Failure and resumption
If one part fails, retry only that part. Uploading the same part number again replaces the earlier attempt. To resume after a crash or a dropped connection, the app calls ListParts for the stored UploadId. The client then skips the parts already uploaded. Because the client requests part URLs in small batches as it needs them, each URL can expire within minutes. A long upload never depends on a credential that has been valid for hours.
The cost
The client now has to handle chunking, a concurrency limit, retries with backoff, and ETag bookkeeping. This is more work than a single PUT. It is only worth it once file sizes make restarting from zero a real problem.
import boto3
s3 = boto3.client("s3")
BUCKET = "example-uploads-bucket"
def start_upload(key: str) -> str:
# Store the returned UploadId against the user and key so the upload can be resumed later.
resp = s3.create_multipart_upload(Bucket=BUCKET, Key=key)
return resp["UploadId"]
def presign_parts(key: str, upload_id: str, part_numbers: list[int]) -> dict[int, str]:
# Hand out URLs in small batches with a short expiry. The client asks again when it needs more.
return {
n: s3.generate_presigned_url(
"upload_part",
Params={"Bucket": BUCKET, "Key": key, "UploadId": upload_id, "PartNumber": n},
ExpiresIn=300,
)
for n in part_numbers
}
def uploaded_parts(key: str, upload_id: str) -> dict[int, str]:
# On resume, report which parts S3 already holds so the client can skip them.
parts = {}
for page in s3.get_paginator("list_parts").paginate(Bucket=BUCKET, Key=key, UploadId=upload_id):
for p in page.get("Parts", []):
parts[p["PartNumber"]] = p["ETag"]
return parts
def complete(key: str, upload_id: str, etags: dict[int, str]) -> None:
# The client sends the part number and ETag pairs it collected. S3 requires them in ascending order.
parts = [{"PartNumber": n, "ETag": etags[n]} for n in sorted(etags)]
s3.complete_multipart_upload(
Bucket=BUCKET, Key=key, UploadId=upload_id, MultipartUpload={"Parts": parts}
)
def cancel(key: str, upload_id: str) -> None:
# Abort frees the stored parts. Otherwise they stay in the bucket and are billed until removed.
s3.abort_multipart_upload(Bucket=BUCKET, Key=key, UploadId=upload_id)
Server-side boto3 handlers that start a multipart upload, presign short-lived URLs for batches of parts, list the parts already uploaded so a client can resume, and complete or abort the upload.
How do you confirm, validate, serve and clean up uploaded files?
When the client reports "done", the database row should move to a pending state, not to complete. Before accepting the claim, the server calls HeadObject and compares the object's size and content type against what it issued in the presigned request. S3 event notifications avoid relying on the client, but they arrive asynchronously and can be duplicated, so the handler must be idempotent.
| Signal | Latency | Trust | Failure mode |
|---|---|---|---|
| Client callback | Immediate | Low, so verify with HeadObject |
Client closes the tab and the callback never arrives |
| S3 event notification | Seconds, sometimes more | High | Duplicate or delayed delivery |
The declared Content-Type is only a client claim. A background job should read the first bytes of the object to check its magic numbers, run a malware scan, and generate thumbnails or transcodes. Uploads should land under a quarantine/ prefix that nothing serves from. The job copies the object to its final key only after every check passes. Until then, the UI shows a "processing" state, so users understand why the file isn't available yet.
Add a lifecycle rule with AbortIncompleteMultipartUpload set to a few days after initiation. Without it, you keep paying to store parts from abandoned uploads, even though they never appear as objects. A nightly reconciliation job covers the gaps the other mechanisms miss. It matches stale pending rows against S3 and marks them failed when the object is missing. It also lists objects that have no matching row and deletes them.
For downloads, issue short-lived presigned GET URLs, or put a CDN in front of a private bucket. Either way, file bytes never pass through the app servers, and the bucket stays non-public.
#!/usr/bin/env bash
set -euo pipefail
BUCKET="example-app-uploads"
cat > lifecycle.json <<'EOF'
{
"Rules": [
{
"ID": "abort-incomplete-multipart-uploads",
"Status": "Enabled",
"Filter": { "Prefix": "quarantine/" },
"AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 3 }
}
]
}
EOF
aws s3api put-bucket-lifecycle-configuration \
--bucket "$BUCKET" \
--lifecycle-configuration file://lifecycle.json
aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET"
aws s3api list-multipart-uploads --bucket "$BUCKET" --prefix "quarantine/"
Applies an S3 lifecycle rule that aborts incomplete multipart uploads under the quarantine/ prefix three days after they start, then verifies the rule and lists uploads still in progress.
How can presigned uploads save engineering time?
The savings come from work you stop doing, not from the upload code itself, which is extra work up front.
- No upload service to build and run. The alternative to proxying is often a dedicated upload service with its own deployment, scaling, monitoring and on-call. Presigned uploads keep that logic as two or three endpoints inside the monolith, reviewed and deployed with everything else.
- No per-route tuning of proxy and load balancer limits. You no longer need special body size caps, longer timeouts or separate worker pools for one upload route, and you avoid the security review that loosening those limits deserves.
- Fewer capacity incidents to investigate. When uploads share workers with page requests, a burst of slow uploads shows up as general latency or timeouts, which takes time to trace back. Moving the bytes to S3 removes that failure mode from the app tier.
- Retry and resumption logic you don't write on the server. With multipart upload, S3 stores parts and reports them through
ListParts. The server only tracks theUploadIdand hands out URLs. It does not have to buffer chunks or stitch files together. - Standard, well-documented building blocks. Signing is a single SDK call, and CORS, lifecycle rules and event notifications are bucket configuration. New team members can read the AWS documentation rather than learning a custom upload protocol.
- Simpler capacity planning. App servers can be sized for request rate and CPU, not for the number of concurrent slow transfers.
An illustrative estimate shows the capacity side. Assume 50 users upload 500 MB files at the same time over an assumed 10 Mbit/s uplink each. Each transfer takes about 400 seconds (500 MB is roughly 4,000 Mbit, divided by 10 Mbit/s). Proxied through the monolith, that is 50 workers or threads held for nearly seven minutes, plus 25 GB passing through app servers twice. With presigned URLs, the same users cost 50 short signing requests and 50 short completion callbacks. These inputs are assumed for illustration, not measured. Substitute your own file sizes, concurrency and client bandwidth to see whether the difference matters for your workload.
The time saved is offset by the work listed in the next section: CORS setup, state modelling, reconciliation and multipart clients. The trade is favourable when the ongoing cost of proxy tuning, capacity incidents or a separate service is higher than that one-time build plus the reconciliation job.
When should you avoid presigned uploads, and what are the alternatives?
Presigned uploads trade control for capacity. The app never sees the byte stream, so it cannot reject a file halfway through or scan it inline. Validation always happens after the object already exists. The database and the bucket can drift apart, which means reconciliation is a permanent job, not a one-off cleanup. A presigned URL is a bearer token: anyone who obtains it can use it until it expires. Size and type limits are only as strong as the conditions you signed. Abandoned multipart uploads keep billing for stored parts unless a lifecycle rule aborts them. Some corporate networks and proxies block direct connections to S3 endpoints. Local development depends on an S3-compatible stand-in or a dev bucket, and both can differ subtly from production in CORS, signing and error behaviour.
| Option | Byte-level control | Ops burden | Best fit |
|---|---|---|---|
| Streaming proxy through the app | Full, inline | Tuning body limits, timeouts, worker capacity | Small files, inline inspection |
| Dedicated upload service | Full, isolated from the monolith | A separate service to build, deploy and operate | High volume with inline scanning |
| Resumable-upload server | Full, with chunk resumption | Session state and storage to manage | Unreliable client networks |
Presigned uploads carry upfront work: CORS, upload state modelling, reconciliation, and multipart clients. That cost pays off when it replaces ongoing proxy tuning or running a separate upload service.
Proxying is still the simpler choice when:
- files are small, such as avatars or short CSVs
- volume is low
- compliance requires inspecting bytes before anything is stored
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to s3 presigned url upload, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.