Why does proxying file uploads through your app servers hurt?
When a user uploads a file to your monolith, your application has to handle every byte of it. That causes problems at several layers.
Workers are held by slow clients. A request worker or thread stays occupied until the last byte arrives. Your server's speed doesn't matter here. The client's uplink sets the pace, and mobile or home connections are often slow.
Illustrative estimate. Assume these inputs:
- 16 request workers
- 200 MB average file size
- 5 Mbit/s client upload bandwidth
200 MB is about 1,600 Mbit, so each upload takes roughly 320 seconds, a little over five minutes. Sixteen concurrent uploads would occupy every worker for that whole time. Ordinary page loads and API calls would queue behind them or time out. These numbers are assumed inputs, not measurements.
Memory and disk pressure. Many frameworks buffer request bodies in memory or spool them to temporary disk before your handler runs. Concurrent large uploads can exhaust RAM or fill the temp volume, and that affects unrelated requests on the same host.
Limits at every hop. Load balancers and reverse proxies enforce maximum body sizes and idle or request timeouts. Each layer needs tuning, and raising those limits for one upload route often loosens them everywhere else too.
Double transfer. The file crosses your network twice: once from the client to the app, then from the app to S3. Depending on the route, such as through a NAT gateway rather than a VPC endpoint, you may also pay for that second hop.
This pattern helps most for monoliths that accept user files of meaningful size.
Who should use presigned uploads?
Presigned uploads suit teams that already store user files in S3 and run an application tier that should not be in the data path. They are a good fit if one or more of these describes you:
- You run a monolith or a small fleet of app servers with a fixed pool of request workers, where a handful of long uploads can starve ordinary traffic.
- Your users upload files of meaningful size. Examples include videos, design files, datasets, scanned documents and backups, rather than only small avatars.
- Your clients are on slow or unreliable networks, such as mobile apps or users on home connections, where uploads take minutes and connections drop.
- You are multi-tenant and need each tenant's files kept under its own prefix, with the server deciding where every object lands.
- You want upload capacity to scale separately from app capacity, so a burst of uploads doesn't force you to add app servers.
They are a weaker fit for apps where uploads are small and rare, where every file must be rejected or transformed synchronously before the response returns, or where files must never be stored unverified, even briefly in a quarantine prefix. The last section covers those cases and the alternatives.
How does a direct to S3 upload work end to end?
The pattern splits responsibilities: the app authorizes, signs and records, and the client sends the bytes to S3. Your servers handle only small JSON requests. The file body never passes through them.
Client App S3
|-- 1. request upload -->| |
| (name, size, type) |-- 2. authenticate, |
| | authorize |
| |-- 3. insert row: pending |
| |-- 4. sign URL for a |
| | server-chosen key |
|<-- 5. URL + upload id -| |
|-- 6. upload bytes ------------------------------->|
|<-- 7. success ------------------------------------|
|-- 8a. finalize(id) --->| |
| |<-- 8b. event notification |
The server picks the object key, for example a tenant prefix plus a random ID. The client never chooses it. This stops one user from overwriting another user's object or writing into a path they shouldn't reach. The client's filename becomes metadata in your database. It is not part of the key.
Finalization has two possible triggers, and each has a weakness. The client's finalize call (8a) is immediate, but it never arrives if the user closes the tab after the upload succeeds. An S3 event notification (8b) does not depend on the client, but it is asynchronous and can be delivered more than once. Many teams use both and make finalize idempotent, keyed on the upload ID.
The trade-off is new state. Before, an upload was a single request that either succeeded or failed. Now each upload is a record with an owner, key, expected size and type, expiry, and a status that moves from pending to uploaded or expired. The two halves can drift apart. You can end up with pending rows that never complete, or with objects in S3 that have no matching row. The section on verifying, processing and reconciling uploads covers how to reconcile them.
+---------+ +-------------+ +----------+
| Client |----1. request upload--->| App |--3. insert-->| DB |
|(browser)| (name, size, type) | 2. authn/z | (pending) | uploads |
| |<---5. URL + upload id---| 4. sign URL | | record |
+---------+ | for server- | +----------+
| ^ | | chosen key | ^
| | | +-------------+ |
| | | ^ ^ |
| | +------8a. finalize(id)---------+ | 8. finalize sets |
| | | status = uploaded |
| | | idempotent on id ---+
| | | 8b. event notification
| | |
| | +------+------+
| +-------7. 200 OK-----------------| S3 |
| 6. PUT bytes direct | bucket |
+----------------------------------->| tenant/<id> |
+-------------+
Upload record: pending --(finalize, 8a or 8b)--> uploaded
pending --(expiry passes)-------> expired
The app authenticates, records a pending upload and signs a URL for a server-chosen key; the client PUTs bytes directly to S3, and finalize runs from the client (8a), an S3 event notification (8b), or both, idempotently keyed on the upload ID.
How does a presigned URL work?
A presigned URL is an ordinary S3 request with its authentication moved into the query string. Normally an SDK signs each request with Signature Version 4 and sends the signature in the Authorization header. When you presign, the SDK computes the same kind of signature over the method, bucket, key, any headers you chose to sign, and an expiry. It then encodes everything as query parameters instead, such as X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders and X-Amz-Signature. If the signer uses temporary credentials, X-Amz-Security-Token is included as well.
Three properties follow from this:
- Signing is local. Generating the URL is a cryptographic computation with the signer's credentials. It does not call S3, so the signing endpoint is cheap and fast.
- S3 checks everything when the request arrives. S3 recomputes the signature from the incoming request. It rejects the request if anything signed has changed, such as the key, the method or a signed header. It also rejects the request if the expiry has passed or the signer's identity lacks permission for the operation at that moment.
- The URL carries the signer's authority, and only for one operation. The client never sees your credentials. It gets a URL that performs exactly the signed request, as the signing identity, until it expires. This is why the signing role should be narrowly scoped and why the URL must be handled as a secret.
A presigned POST works on the same principle. The difference is that the signature covers a policy document of conditions rather than one fixed request.
How do you generate an S3 presigned URL safely?
The signing endpoint should be small and boring. It authenticates the user, decides where the object will live, records the pending row, and returns a URL. The signing credentials should come from a role whose policy allows only s3:PutObject on one prefix, such as uploads/pending/*. The role should not allow GetObject, DeleteObject, or access to the rest of the bucket.
That scoping matters because a presigned URL is a bearer token. Anyone holding it can perform that one operation with the signer's permissions until it expires. Treat it like a password:
- Keep it out of application logs, error trackers, and analytics events.
- Keep it out of anything that ends up in browser history or
Refererheaders. - Keep expirations short, minutes rather than hours, since the client normally uses the URL immediately.
- If you sign with temporary credentials, the URL also stops working when those credentials expire, whichever comes first.
Choose the object key on the server, for example uploads/pending/{tenant_id}/{uuid}. Never build keys from client filenames. They invite path tricks, collisions, encoding problems, and overwrites of other users' objects. Store the original filename in your database or as x-amz-meta- metadata. If you sign metadata or Content-Type headers, the client must send exactly those values, or S3 rejects the request.
Signature Version 4 embeds the signer's timestamp in the URL. If the signing host's clock drifts, S3 can reject freshly issued URLs with errors that look like expiry or access-denied problems. That sends people chasing IAM policies instead of NTP, so keep server clocks synchronized.
import uuid
import boto3
BUCKET = "example-uploads-bucket"
ALLOWED_TYPES = {"image/png", "image/jpeg", "application/pdf"}
s3 = boto3.client("s3", region_name="eu-west-1")
def create_upload(user, filename: str, content_type: str, db) -> dict:
if content_type not in ALLOWED_TYPES:
raise ValueError("unsupported content type")
# The server chooses the key. The client filename never becomes part of the path.
key = f"uploads/pending/{user.tenant_id}/{uuid.uuid4()}"
# Store the original filename as data, alongside the pending row.
db.insert_upload(key=key, owner_id=user.id,
original_filename=filename, status="pending")
url = s3.generate_presigned_url(
"put_object",
# ContentType is signed, so the client must send exactly this header.
Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type},
ExpiresIn=300, # Minutes, not hours, because the client uploads right away.
)
# Never log `url`. It is a bearer token until it expires.
return {"upload_id": key, "url": url,
"headers": {"Content-Type": content_type}}
A minimal boto3 signing handler picks the object key on the server, records the pending upload, and returns a short-lived PUT URL signed by a role that can only write to uploads/pending/.*
Should you use a presigned PUT or a presigned POST?
A presigned PUT URL authorizes one operation: write a body to one exact key. It says nothing about how large that body may be. If a client receives a URL for uploads/123/avatar.png, it can send a 4 GB file to that key and S3 will store it. You will only find out afterwards, during verification.
A presigned POST carries a signed policy document. S3 checks every condition in the policy before it accepts the upload. Conditions useful for uploads:
["content-length-range", 1, 10485760]rejects bodies under 1 byte or over 10 MiB.["starts-with", "$key", "uploads/123/"]lets the client pick a filename but keeps the object under that user's prefix.["starts-with", "$Content-Type", "image/"]limits uploads to image types.{"bucket": "my-uploads"}pins the target bucket.
| Concern | Presigned PUT | Presigned POST |
|---|---|---|
| Size cap at upload time | No | Yes, via content-length-range |
| Key flexibility | Exact key only | Exact key or prefix |
| Content type | Exact value, if signed | Exact or prefix match |
| Request shape | Raw body | multipart/form-data |
| Plain HTML form | No | Yes |
| Multipart upload parts | Yes, per-part URLs | No |
Use POST when the limits must be enforced before storage, such as public-facing uploads where an oversized file costs you money. It is also the natural choice for browser form uploads.
Use PUT for trusted or programmatic clients, which can send a raw body with less ceremony. Treat its lack of a size cap as a risk you handle in post-upload verification.
The Content-Type row in the table assumes you included Content-Type in the signed headers when generating the PUT URL. If you did not, the client can send any type.
import boto3
s3 = boto3.client("s3", region_name="us-east-1")
user_id = "123"
prefix = f"uploads/{user_id}/"
conditions = [
# Reject empty bodies and anything over 10 MiB.
["content-length-range", 1, 10 * 1024 * 1024],
# The client chooses the filename, but the object stays under this user's prefix.
["starts-with", "$key", prefix],
# Only image types. The form must send a Content-Type field.
["starts-with", "$Content-Type", "image/"],
]
post = s3.generate_presigned_post(
Bucket="my-uploads",
# S3 replaces ${filename} with the name of the uploaded file.
Key=prefix + "${filename}",
Conditions=conditions,
ExpiresIn=300, # 5 minutes
)
print(post["url"])
for name, value in post["fields"].items():
print(name, "=", value)
Uses boto3 to generate a presigned POST whose signed policy caps uploads at 10 MiB, keeps keys under the user's prefix and allows only image/ content types.*
How do you handle large file uploads with multipart upload?
A single presigned PUT stops working well for large files. One dropped connection restarts the whole transfer, and S3 caps a single PUT at 5 GB. Multipart upload splits the file into parts that upload, retry, and resume independently.
The flow
- Initiate. The client asks your app to start an upload. The app authorizes the request, picks the object key, and calls
CreateMultipartUpload. It stores the returnedUploadIdagainst the user and key, then returns theUploadIdto the client. - Presign parts lazily. The client requests URLs for a batch of part numbers. The app presigns
UploadPartfor each one. Do not presign every part up front. A multi-hour upload would outlive short expiries, and long expiries widen the window in which a leaked URL can be used. Issue small batches, keep expiries short, and let the client fetch fresh URLs as it goes. - Upload in parallel. The client sends a few parts at a time and retries failures with backoff. It records each part's
ETagresponse header, which lets it resume after a page reload. Every part except the last must be at least 5 MiB, and an upload can have at most 10,000 parts. Choose a part size that fits your largest expected file within that limit. - Complete. The client sends the part number and ETag pairs to the app. The app checks that they belong to the stored
UploadIdand callsCompleteMultipartUpload.
Bucket configuration
Browsers can't read ETag unless CORS exposes it. Your CORS rule needs:
- your app's origins in
AllowedOrigins PUTinAllowedMethods- the headers you send in
AllowedHeaders ETaginExposeHeaders
Abandoned uploads leave parts that you keep paying for but never see in normal object listings. Add a lifecycle rule with AbortIncompleteMultipartUpload and a DaysAfterInitiation threshold that exceeds your longest realistic upload.
import { randomUUID } from "node:crypto";
import { S3Client, CreateMultipartUploadCommand, UploadPartCommand,
CompleteMultipartUploadCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
const s3 = new S3Client({ region: "us-east-1" });
const Bucket = "example-uploads-bucket";
// 1. Initiate: the server picks the key, never the client.
export async function initiate(userId: string) {
const Key = `uploads/${userId}/${randomUUID()}`;
const { UploadId } = await s3.send(new CreateMultipartUploadCommand({ Bucket, Key }));
// Persist { userId, Key, UploadId } so later calls can be checked against it.
return { Key, UploadId };
}
// 2. Presign lazily: small batches with short expiries. The client asks again as it goes.
export async function presignParts(Key: string, UploadId: string, partNumbers: number[]) {
// Before signing, check that this user owns (Key, UploadId).
if (partNumbers.length > 20) throw new Error("batch too large");
if (partNumbers.some((n) => !Number.isInteger(n) || n < 1 || n > 10000)) {
throw new Error("part number out of range");
}
return Promise.all(partNumbers.map(async (PartNumber) => ({
PartNumber,
url: await getSignedUrl(s3,
new UploadPartCommand({ Bucket, Key, UploadId, PartNumber }),
{ expiresIn: 900 }), // 15 minutes per URL
})));
}
// 3. Complete: the client reports the ETag it read from each part's PUT response.
export async function complete(Key: string, UploadId: string,
parts: { PartNumber: number; ETag: string }[]) {
// Check ownership again, then send the parts in ascending part-number order.
const Parts = [...parts].sort((a, b) => a.PartNumber - b.PartNumber);
return s3.send(new CompleteMultipartUploadCommand({
Bucket, Key, UploadId, MultipartUpload: { Parts } }));
}
A server-side outline using the AWS SDK for JavaScript v3: the server starts the multipart upload, presigns UploadPart URLs in small short-lived batches on request, and completes the upload from the part numbers and ETags the client reports.
How should the backend verify, process and reconcile uploads?
The browser's "upload complete" call is only a hint. The client may lie, retry, or never send it, so the backend has to confirm against S3 itself. On finalize, call HeadObject on the expected key and compare ContentLength against the size you signed for. Check ContentType as well, keeping in mind that it is whatever the client declared. If the upload included a checksum, compare it with the value you recorded when issuing the URL. Only then move the record from pending to available.
Make finalize idempotent. Retries and duplicate requests are normal. A conditional update such as UPDATE ... SET status = 'available' WHERE id = ? AND status = 'pending' means a second call becomes a no-op rather than a second round of side effects. The handler sketch below follows this pattern.
Process asynchronously, from quarantine
Have clients upload under a quarantine prefix such as incoming/. Use S3 event notifications or your own job queue to trigger:
- virus scanning
- content sniffing, which checks the actual bytes, not the declared type
- image resizing
Copy the object to its permanent prefix only after these checks pass. Keeping this work out of the request path stops a slow scanner from tying up web workers, and nothing unverified is ever served.
Reconcile on a schedule
Events get dropped and clients disappear, so run a periodic job that:
- expires
pendingrecords older than the URL lifetime plus a margin - deletes objects in quarantine with no matching record
- finalizes objects that exist and match a
pendingrecord but never received a finalize call
How can presigned uploads save engineering time?
The time savings come from removing a category of recurring operational work, not from making the first implementation faster. Building the flow above takes deliberate effort. Once it is in place, several kinds of work largely go away:
- No more per-hop limit tuning. Body-size limits and timeouts on load balancers, reverse proxies and the app framework no longer have to be raised for an upload route and then re-checked whenever infrastructure changes. Those layers only see small JSON requests.
- Fewer worker-starvation incidents to debug. When uploads share workers with ordinary traffic, slow uploads show up as unexplained latency spikes and timeouts on unrelated endpoints. Those incidents are time-consuming to diagnose because the cause is a client's network, not your code. With the bytes going to S3, that failure mode is gone.
- Capacity planning gets simpler. Upload volume no longer feeds into how many app servers or workers you need, and you don't have to size temp disks or memory for concurrent upload bodies.
- Retry and resume logic is mostly S3's protocol, not yours. With multipart upload, resuming after a dropped connection means re-sending failed parts against an existing
UploadId. You don't have to design a chunked-upload protocol of your own on the server. - Signing is SDK code you don't maintain. The official SDKs implement Signature Version 4 and presigning. Your code is the small authorization and bookkeeping layer around it.
- Security review has a smaller surface. The upload path's privileges are captured in one narrowly scoped IAM policy and a short signing handler, which is easier to review than a streaming file handler with its own buffering and limits.
Count the new work against these savings: the pending state machine, CORS and lifecycle configuration, quarantine processing and the reconciliation job. If uploads are a minor feature with small files, the proxy approach may cost less engineering time overall. The next section covers that case.
When should you avoid presigned uploads, and what are the alternatives?
Presigned uploads move complexity rather than remove it. The single request becomes a multi-step protocol with pending state, expiry windows and a reconciliation job. Inline validation is gone: a malicious or malformed file sits in the bucket until your verifier runs, so quarantine prefixes and deny-by-default reads are required. A presigned URL is a bearer token, so anyone who obtains one from logs, browser history or a shared link can use it until it expires. You also take on CORS configuration, client-side chunking and retry logic, and a local setup that needs a real bucket or an S3-compatible emulator.
The payoff is operational. Worker processes stop being held by slow clients, proxy body-size and timeout limits no longer need tuning, and upload volume scales independently of app capacity. If uploads are large, frequent or come from slow networks, that trade usually favors presigning. If uploads are small, infrequent and need synchronous rejection or transformation, such as an avatar cropped before the response returns, proxying through the app is still a reasonable choice.
| Approach | Best for | Main cost |
|---|---|---|
| App proxy (buffered) | Small, rare files needing inline validation | Ties up workers and memory |
| Streaming proxy | Medium files that must pass through your code | Still holds connections for the full transfer |
| Presigned PUT | Single objects with a known size and type | Weak constraints on size |
| Presigned POST | Browser forms that need policy limits on size and key | Form-encoding quirks, single request only |
| Multipart presigned | Large files and resumable uploads | Most client and cleanup logic |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to s3 presigned url upload, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.