Why do local-disk uploads stop a monolith from scaling horizontally?
When a monolith writes uploads to its own filesystem, the file exists on exactly one machine. That is fine with one instance. With two behind a load balancer, a user uploads an avatar to instance A, and the next request to view it lands on instance B, which returns a 404.
Teams usually patch this in one of two ways. Sticky sessions pin each user to one instance. That skews load and breaks the moment that instance is drained or crashes. A shared network volume keeps the files in one place, but it adds a new single point of failure, its own throughput limits, and locking problems when two writers touch the same path. Containers and autoscaling groups make the problem worse. Instances are disposable, so anything written to the local disk disappears when the instance is replaced.
There is also a cost in compute. A 2 GB video upload over a slow mobile connection holds an application worker or request thread for the entire transfer. That worker does nothing but copy bytes. A handful of such uploads can starve the pool that serves ordinary requests.
Presigned URLs change who moves the bytes. The application becomes a control plane. It authenticates the user, decides what they may upload, issues a short-lived signed permission, and records the upload's state. The client sends the data directly to S3. The application never touches the body, so every instance is interchangeable.
This change suits teams that are adding instances, moving to containers, or enabling autoscaling. Make it before you scale out. Migrating files that are already split across machines is much harder.
How do S3 presigned URL uploads work end to end?
The flow has four steps. The client asks the app for an upload slot and sends the filename, declared size, and content type. The app checks that the user is allowed to upload, applies its own limits, and records a pending upload row. It then generates an object key server-side, such as uploads/{tenant}/{uuid}, and signs a PUT URL that is valid only for that key and expires within minutes. The client sends the bytes directly to S3, so no app instance handles the file body. Finally, something marks the upload as complete.
That last step is where trust matters. A client that says "done" may be buggy, may be retrying, or may be malicious. It can report success for a PUT that failed, or claim a different size and type than what it actually sent. Before you mark the row active, the server should issue a HEAD request on the key and compare the actual size and content type against the pending record. S3 offers strong read-after-write consistency for new objects, so if the PUT succeeded, the HEAD request will see the object.
| Aspect | Client callback | S3 event notification |
|---|---|---|
| Latency | Immediate after PUT returns | Usually seconds, occasionally longer |
| Reliability | Lost if the client closes or crashes | Delivered even if the client disappears |
| Delivery semantics | Whatever the client retries | At-least-once; duplicates possible |
| Ordering | Follows the client's sequence | Not guaranteed; use the sequencer field per key |
| Extra infrastructure | None | Queue or function consumer |
Many teams run both. The callback gives the user fast feedback, and the event catches uploads where the client never called back. Make finalization idempotent, keyed on the object key, so that duplicate or late events do nothing.
+----------+ 1. request slot (name, size, type) +-----------------------+
| |------------------------------------->| App |
| Client | 2. signed PUT URL (1 key, ~minutes) | authz + limits |
| |<-------------------------------------| key: uploads/t/uuid |
| | 4a. callback "done" (untrusted) | pending row in DB |
| |------------------------------------->| |
+----------+ +-----------------------+
| |
| 3. PUT bytes direct to S3 | 4a relayed
| (app never sees the body) v
v +-----------------------+
+----------+ +--------------+ | Finalizer |
| S3 | 4b | event queue | | idempotent on key |
| bucket |--------->| (or function)|----------->| HEAD key, compare |
| | +--------------+ | size/type vs pending |
| |<-------------------------------------| then mark active |
+----------+ 5. HEAD key (read-after-write) +-----------------------+
4b is at-least-once and unordered: dedupe on object key, check sequencer.
4a is fast but untrusted; 4b catches clients that never call back.
Presigned PUT flow: the app signs a single-key URL, the client uploads bytes directly to S3, and both the client callback and the S3 event feed one idempotent finalizer that verifies the object with a HEAD request before marking it active.
How do you implement presigned uploads in a monolith?
Data model and key generation
Start with an uploads table: id, object_key, status (pending, complete, failed), original_filename, declared_content_type, declared_size, owner_id, and timestamps. The server generates the object key, for example uploads/{owner_id}/{uuid}, and never derives it from the user's filename. User filenames collide, contain path separators and odd Unicode, and can leak personal information into logs and URLs. Store the original filename as a column, and set it as the Content-Disposition header when you serve the file.
Issuing the URL
The issue endpoint authorizes the user, validates the declared type and size against your allowlist, inserts a pending row, and returns a presigned PUT. Include Content-Type in the signed headers so the client must send that exact value or S3 rejects the request. Keep the expiry short, a few minutes, so a leaked URL is of little use.
Finalizing
The finalize endpoint does not trust the client's word. It issues a HEAD on the object and compares Content-Length, Content-Type, and the checksum (if the client uploaded with a checksum header, request it back in the HEAD) against the row. On a match it marks the row complete. On a mismatch it marks the row failed and deletes the object.
Operational details
Signatures are time-bound, so a signing host with a drifting clock produces URLs that S3 rejects as expired or not yet valid. Run time synchronization on every app server. For local development and CI, point the SDK at an S3-compatible emulator so tests exercise real signing and HEAD behavior instead of mocks.
import os, uuid
import boto3
from botocore.exceptions import ClientError
# In dev/CI, S3_ENDPOINT_URL points at an S3-compatible emulator; unset means real S3
s3 = boto3.client("s3", endpoint_url=os.environ.get("S3_ENDPOINT_URL"))
BUCKET = "example-app-uploads"
ALLOWED_TYPES = {"image/png", "image/jpeg", "application/pdf"}
MAX_SIZE = 25 * 1024 * 1024
def issue_upload(db, owner_id, filename, content_type, size):
if content_type not in ALLOWED_TYPES or not 0 < size <= MAX_SIZE:
raise ValueError("type or size not allowed")
key = f"uploads/{owner_id}/{uuid.uuid4()}" # server-generated, never from filename
upload_id = db.insert_upload(object_key=key, status="pending", owner_id=owner_id,
original_filename=filename, declared_content_type=content_type, declared_size=size)
# ContentType is signed: the client must send exactly this header or S3 rejects the PUT
url = s3.generate_presigned_url("put_object", ExpiresIn=300,
Params={"Bucket": BUCKET, "Key": key, "ContentType": content_type})
return {"upload_id": upload_id, "url": url, "headers": {"Content-Type": content_type}}
def finalize_upload(db, upload_id, owner_id):
row = db.get_pending_upload(upload_id, owner_id=owner_id)
try:
# ChecksumMode returns stored checksums if the client uploaded with one
head = s3.head_object(Bucket=BUCKET, Key=row.object_key, ChecksumMode="ENABLED")
except ClientError:
db.set_status(upload_id, "failed") # object never arrived
return False
ok = (head["ContentLength"] == row.declared_size
and head.get("ContentType") == row.declared_content_type)
if row.expected_sha256: # optional column: base64 SHA-256 the client declared
ok = ok and head.get("ChecksumSHA256") == row.expected_sha256
if not ok:
s3.delete_object(Bucket=BUCKET, Key=row.object_key)
db.set_status(upload_id, "failed")
return False
db.set_status(upload_id, "complete")
return True
A boto3 sketch of the issue endpoint (server-generated key, pending row, presigned PUT with signed Content-Type and a 5-minute expiry) and the finalize endpoint that HEADs the object to verify size, type, and checksum before marking the row complete or failed.
How do you restrict what clients can upload and secure the bucket?
A presigned PUT cannot cap object size. It authorizes one key and method, and it can pin a few signed headers, but it has no way to express "at most 50 MB". A client holding the URL can upload as much data as S3 accepts. If you need a maximum, use a presigned POST. Its signed policy document can carry a content-length-range condition, and S3 rejects any upload outside that range.
| Presigned PUT | Presigned POST | |
|---|---|---|
| Size limit | Not enforceable | content-length-range condition |
| Content-type control | Exact value, if signed | Exact value or starts-with prefix |
| Client request | Raw body | multipart/form-data with policy fields |
Bucket configuration. Keep Block Public Access on. Clients write through signed requests, so nothing needs to be public. Add a CORS rule that lists your app's exact origin, allows only PUT and/or POST, permits the headers your client sends (such as Content-Type), and exposes ETag, which multipart uploads need. The signing role only needs s3:PutObject on the upload prefix. A presigned URL carries its signer's permissions, so any extra access on that role widens what a leaked URL can do.
Treat URLs as bearer credentials. Anyone holding the URL can use it. Use expiries measured in minutes. Strip query strings from access logs and analytics, and set Referrer-Policy: no-referrer on pages that handle these URLs.
Diagnosing failures. CORS misconfiguration is the most common failure. The browser reports a generic network error, so open devtools and inspect the preflight OPTIONS request. A 403 response, or a response with no Access-Control-Allow-Origin header, means no CORS rule matched the request's origin, method, or headers. Some corporate proxies also block direct S3 hostnames, so keep a server-relayed upload path for those users.
import uuid
import boto3
# Runs under the signing role, which only has s3:PutObject on uploads/*
s3 = boto3.client("s3", region_name="us-east-1")
BUCKET = "example-app-uploads"
MAX_BYTES = 50 * 1024 * 1024 # 50 MB cap, enforced by S3 itself
def create_upload(content_type: str) -> dict:
# Server picks the key so clients cannot write outside the upload prefix
key = f"uploads/{uuid.uuid4()}"
post = s3.generate_presigned_post(
Bucket=BUCKET,
Key=key,
# Fields are pre-filled form values the client must send unchanged
Fields={"Content-Type": content_type},
Conditions=[
# S3 rejects the upload if the file is outside this byte range
["content-length-range", 1, MAX_BYTES],
# Allow any image subtype rather than one exact value
["starts-with", "$Content-Type", "image/"],
],
ExpiresIn=300, # short-lived: the policy is a bearer credential
)
# Log the key only, never the URL or the signed policy fields
print(f"issued upload policy for key={key}")
# Client sends multipart/form-data: every field in post["fields"],
# then the file as the last form field named "file", to post["url"]
return {"key": key, "url": post["url"], "fields": post["fields"]}
Generating a presigned POST with boto3 whose signed policy caps uploads at 50 MB with content-length-range, restricts Content-Type to images, expires in five minutes, and logs only the object key.
When should you use multipart upload for large files?
A single presigned PUT can carry up to 5 GB, but you shouldn't push it that far. AWS recommends multipart upload once objects exceed roughly 100 MB. The reason is reliability more than speed: a dropped connection at 90% of a 2 GB single PUT means starting over. With multipart, you only resend the part that failed.
The flow has four steps:
- Initiate on the server. Your monolith calls
CreateMultipartUploadwith the final key and content type, gets anUploadId, and records it next to the pending file row. - Presign each part. For each part number, the server generates a presigned
UploadPartURL. Parts must be at least 5 MiB, except the last one, and an upload can have at most 10,000 parts. Choose a part size large enough that your biggest allowed file fits within that limit. - Upload parts in parallel. The client sends several parts at once, typically three to six. It retries each failed part with backoff and reads the
ETagresponse header from every successful part. The browser can only read that header if your CORS rule listsETagunderExposeHeaders. - Complete. The client returns the part numbers and ETags to the server. The server calls
CompleteMultipartUpload, and S3 assembles the object.
If a client never reaches the last step, the uploaded parts stay in the bucket. You pay for their storage, and they don't appear in normal object listings. Add a lifecycle rule with AbortIncompleteMultipartUpload set to a few days after initiation so S3 deletes them for you.
The trade-off is client complexity. The client now has to handle chunking, concurrency, retry logic, and ETag bookkeeping, and the abort rule has to be in place to avoid paying for abandoned parts. For files under about 100 MB, a single presigned PUT is simpler and usually enough.
import math
import boto3
s3 = boto3.client("s3")
BUCKET = "example-app-uploads"
PART_SIZE = 64 * 1024 * 1024 # 64 MiB: >= 5 MiB minimum, 10,000 parts covers ~625 GiB
def start_upload(key, content_type, file_size):
# Step 1: initiate; persist UploadId next to the pending file row
resp = s3.create_multipart_upload(Bucket=BUCKET, Key=key, ContentType=content_type)
upload_id = resp["UploadId"]
part_count = math.ceil(file_size / PART_SIZE)
# Step 2: presign one UploadPart URL per part number (1-based)
urls = [
s3.generate_presigned_url(
ClientMethod="upload_part",
Params={"Bucket": BUCKET, "Key": key,
"UploadId": upload_id, "PartNumber": n},
ExpiresIn=3600,
)
for n in range(1, part_count + 1)
]
return upload_id, PART_SIZE, urls
def finish_upload(key, upload_id, parts):
# Step 4: parts = [{"PartNumber": 1, "ETag": "\"abc...\""}, ...] sent back by the client
parts = sorted(parts, key=lambda p: p["PartNumber"])
s3.complete_multipart_upload(
Bucket=BUCKET, Key=key, UploadId=upload_id,
MultipartUpload={"Parts": parts},
)
# One-time setup: abort abandoned uploads so orphaned parts stop costing money.
# Note: this call replaces the bucket's entire lifecycle configuration.
s3.put_bucket_lifecycle_configuration(
Bucket=BUCKET,
LifecycleConfiguration={"Rules": [{
"ID": "abort-incomplete-multipart",
"Filter": {"Prefix": "uploads/"},
"Status": "Enabled",
"AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 3},
}]},
)
Server-side boto3 code that initiates a multipart upload, presigns a URL for each part, completes the upload with the client's part numbers and ETags, and adds a lifecycle rule that aborts incomplete uploads after three days.
How do you migrate existing files from local disk to S3 safely?
Migrate in phases so that every step can be rolled back and no user request depends on a half-copied bucket.
| Phase | Reads | Writes | Exit criterion |
|---|---|---|---|
| 1. Dual-read | S3 first, disk fallback | Disk | Fallback path deployed and logging hits |
| 2. Backfill | S3 first, disk fallback | Disk and S3 | Backfill job reports zero remaining rows |
| 3. Verify | S3 first, disk fallback | Disk and S3 | Counts and checksums match |
| 4. Cut over | S3 only | S3 only | Fallback hit rate is zero for an agreed window |
Dual-read lets you deploy before any data moves. When an object is missing from S3, the request is served from disk, so users see no change. Log every fallback hit. That metric shows when the fallback is safe to remove.
Backfill should iterate over the uploads table by primary key, not by walking the filesystem. Record the last processed ID so that a crashed job resumes where it stopped. Derive the object key deterministically from the row so that re-running the job overwrites rather than duplicates. While the backfill runs, write new uploads to both disk and S3. Otherwise anything uploaded after the job passes a given ID would be missed.
Verify by comparing the database row count with the S3 object count under the prefix. Then compare a SHA-256 you compute from the disk file with one you computed and stored at upload time. Do not rely on the ETag: it is only an MD5 for some single-part uploads.
Cut over: point writes at S3, then replace the old streaming download endpoint with one that returns a short-lived presigned GET. Use minutes for private documents. Use longer expiries only where the links are embedded in emails.
The dual-read path adds an extra request, and a slow 404, to every miss. Teams also tend to leave it in place permanently. Put a removal date on the migration ticket, and delete the disk mount when you remove the fallback code.
How do you clean up orphaned and abandoned uploads?
Every presigned flow leaks. Clients request a URL and never upload, uploads succeed but the confirm call never arrives, and multipart uploads stall halfway. Without a cleanup path, you pay to store bytes nobody references, and your database fills with records that will never resolve.
The reconciliation job
Run a scheduled job that checks both directions:
- Records without objects. Select
pendingrows older than the presigned URL's expiry plus a grace period. For each, issue a HEAD on the key. If the object exists, the confirm call was lost, so promote the record. If it does not exist, mark the recordabandoned. The HEAD check matters because deleting on age alone discards uploads that actually succeeded. - Objects without records. List the upload prefix and delete keys older than a threshold that have no matching row. Listing a large bucket is slow and costs money per request, so for big buckets use a bucket inventory report instead of live listing.
Lifecycle rules as the backstop
Write uploads to a staging prefix such as incoming/ and copy or move them to their permanent prefix only after confirmation. A lifecycle rule can then expire anything left in incoming/ after a few days. Add a rule that aborts incomplete multipart uploads. Their parts are billed but stay invisible in normal listings.
Asynchronous processing
Extend the status model to pending → uploaded → processing → ready, with rejected as a terminal state for files that fail virus scanning or validation. Serve a file only when it is ready. In the UI, show uploads in processing with a placeholder and a clear "still processing" label, and poll or push the status change. Users should not see a broken thumbnail while the scan is still running.
When should you avoid presigned uploads, and what are the alternatives?
Presigned uploads are not free. You take on an asynchronous pipeline (upload confirmation, background validation, the processing state above), integration tests that need an S3 emulator or a real test bucket, and client code that handles retries, CORS, and expired URLs. In return you drop shared volumes, upload-tuned instances with large disks and long request timeouts, and the coupling between request routing and file location. Whether that trade pays off depends on your traffic and your constraints.
| Approach | Build cost | Ongoing cost | Main weakness |
|---|---|---|---|
| Presigned URLs | High: async pipeline, client changes, emulator tests | Low: app servers never touch bytes | No synchronous validation before the object exists |
| App proxies stream to S3 | Low: server-side change only | Medium: app instances hold connections and bandwidth for each upload | Upload volume still drives app capacity |
| Shared network filesystem | Low: mount and repoint paths | Medium: filesystem throughput, locking, and mount failures to operate | Becomes a shared dependency for every node |
| Local disk with sticky sessions | None | High: scaling, deploys, and node loss all risk files | Not stateless; files die with the instance |
Presigned uploads are a poor fit in three cases:
- Inline synchronous validation is mandatory. If a malware scan, format check, or business rule must reject the file before the request returns, proxying lets you inspect the stream and fail the request directly.
- Clients cannot reach S3 directly. Locked-down corporate networks, egress allowlists, or on-prem clients may only be able to reach your domain.
- Files are small and upload volume is low. A proxy that streams to S3 already gives you stateless file storage. The async machinery adds work you may never need.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to s3 presigned url uploads, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.