How to Migrate File Uploads to S3 in a Monolith Without Downtime

What moving to object storage changes, and who needs to do it

This migration replaces the local directory your app writes uploads to with an object store such as S3. All reads and writes go through one storage interface, so the rest of the codebase stops depending on where the bytes live. The app server stops holding state. Any instance can serve any request, a redeploy or a replaced host no longer deletes user files, and durability becomes the storage service's job instead of your backup script's. You also stop depending on a shared volume that every instance has to mount.

How it works

An object store keeps each file as an object addressed by a key inside a bucket. The namespace is flat. A key like tenants/42/attachments/7f3c… looks like a path, but the slashes are just characters in the key. There are no real directories, renames or partial writes. The app reads and writes objects over an HTTP API, usually through an SDK such as boto3, and does not use filesystem calls.

In the monolith, the change has three parts:

1. One storage interface. Every call site that used to open a path now calls put, get, delete, exists or url on this interface. A configuration value chooses whether a disk backend or an S3 backend sits behind it.
2. Keys in the database. Each upload record stores an object key instead of an absolute path. The database stays the index of which files exist and who owns them.
3. Direct client access. The app still makes every authorization decision. Once it has decided, it hands the client a short-lived presigned URL, and the client downloads from or uploads to the bucket directly.

The move happens without downtime because no single deploy switches everything over. You ship the interface with the disk backend still active. Then you dual-write new uploads to both stores, read from S3 with a fallback to disk, backfill the existing files with a resumable job, and verify sizes and checksums. The local copies are removed only after verification passes. Each phase changes one behavior and can be rolled back with a configuration switch until the final one. The sections below cover each step in order.

When to use it

Use it for monoliths that have outgrown one host, or that already work around that limit with sticky sessions, rsync jobs or a shared mount. It also makes sense when you are moving to autoscaling, containers or immutable deploys, where instances are created and destroyed routinely and nothing should live on their disks. A third case is when restoring uploads after a host failure would be a manual, error-prone job.

Delay it if you run a single host with a small, stable dataset and good backups, because the cost of migrating can exceed the benefit. Be careful if your code depends on filesystem behaviour that object storage generally does not provide: atomic renames, appending to a file, cheap directory listing, or file locks. That code has to be redesigned before it can move.

Alternatives

Object storage is not the only way to stop app servers depending on their local disks. The main alternatives are:

• A shared network filesystem. NFS or a managed network file service is mounted on every instance. Existing path-based code keeps working, which makes this the lowest-effort way to scale past one host. The cost is a new shared dependency, plus network latency on every file operation and locking behaviour that can surprise you under concurrency.
• Local disk with replication or backups. You keep files where they are and copy them between hosts or to backup storage. This changes the least, but the servers still hold state. You still need sticky routing or sync jobs, and restores remain a manual process.
• A managed upload service. A hosted service handles uploads, storage and often transformations. It is quick to adopt, but you take on vendor coupling and its pricing model, and you have less control over keys, retention and access rules.

If the app relies heavily on POSIX semantics and you need to scale soon, a shared filesystem can serve as an intermediate step. Moving to it first and then to object storage later still requires the storage interface described below, so building that interface early does not waste effort.

Audit every place the app touches files

Before changing any code, find every call site that assumes a file is a path on the local filesystem. Most migrations that go wrong miss one of these, and it fails only in production on the first request that hits a server without the file.

Checklist and search patterns

• Search the codebase for the upload directory name (for example uploads/ or media/) and its config key.
• Find raw file I/O with grep -rnE "open\(|os\.path|File\.(read|write|exist)|unlink|rename|shutil".
• Find hardcoded paths with grep -rn "/var/" and with your upload root's absolute path.
• Review web server configs for directives that serve or protect the uploads folder.
• Find job payloads that contain path strings rather than storage keys or record IDs.
• Check crontabs and deploy scripts for find, rsync or tar commands that run against the upload directory.

Record each hit with its owner. This list is your migration scope.

Copy#!/usr/bin/env bash
set -u

UPLOAD_DIR="uploads"                        # dir name as it appears in code
UPLOAD_ROOT="/var/www/myapp/shared/uploads" # absolute path on app servers
OUT="file-access-audit.txt"
EXCL=(--exclude-dir=.git --exclude-dir=node_modules --exclude-dir=vendor)

section() { printf '\n=== %s ===\n' "$1" >> "$OUT"; }
: > "$OUT"

section "Upload dir name and config keys (models, templates)"
grep -rnI "${EXCL[@]}" -e "$UPLOAD_DIR/" -e "UPLOAD_ROOT" -e "UPLOAD_PATH" . >> "$OUT"

section "Raw file I/O (jobs, scripts, models)"
grep -rnIE "${EXCL[@]}" 'open\(|os\.path|File\.(read|write|exist)|unlink|rename|shutil' . >> "$OUT"

section "Hardcoded absolute paths"
grep -rnIF "${EXCL[@]}" -e "/var/" -e "$UPLOAD_ROOT" . >> "$OUT"

section "Sendfile-style headers set by the app"
grep -rnIE "${EXCL[@]}" 'X-Sendfile|X-Accel-Redirect' . >> "$OUT"

section "Web server config that serves or protects uploads"
grep -rnIF -e "$UPLOAD_DIR" /etc/nginx /etc/apache2 2>/dev/null >> "$OUT"

section "Job payloads that may carry file paths instead of keys or IDs"
grep -rnIE "${EXCL[@]}" '(enqueue|perform_async|perform_later|apply_async|delay)\(.*(path|file)' . >> "$OUT"

section "Cron and deploy scripts running find/rsync/tar on uploads"
{ crontab -l 2>/dev/null; cat /etc/cron.d/* 2>/dev/null; } | grep -nE '\b(find|rsync|tar)\b' | grep -F "$UPLOAD_DIR" >> "$OUT"
grep -rnIE '\b(find|rsync|tar)\b' ./deploy ./scripts 2>/dev/null | grep -F "$UPLOAD_DIR" >> "$OUT"

echo "Wrote $(grep -vc '^===\|^$' "$OUT") candidate lines to $OUT"

A bash script that runs the checklist's search patterns across the codebase, web server config and crontabs, and collects every hit into one file grouped by category for manual review.

Put all file access behind one storage interface

Every item on that audit list now gets routed through a single interface with five operations: put(key, stream, content_type), get(key), delete(key), exists(key) and url(key, expires_in). Write two implementations, one backed by the local upload directory and one by S3, and choose between them with a configuration value. Callers never learn which backend they're talking to, so the migration phases that follow can swap or combine backends without touching controllers.

Store object keys in the database, not absolute paths. A path like /var/app/uploads/2019/report.pdf ties every row to one server's filesystem. A key like tenants/42/attachments/7f3c9e1a-… resolves the same way on disk and in a bucket.

Design the key scheme before writing any data:

• Generate keys. Use a random UUID or similar, never the user's filename. Generated keys are non-guessable and avoid collisions and encoding problems.
• Keep the original filename and content type as columns or object metadata, so downloads can restore them.
• Don't depend on listing. Object stores can list keys, but at scale it's slow. The database stays the index.
• Put policy-relevant attributes in the prefix. Tenant, file category and retention class all belong there, because lifecycle rules and IAM policies match on prefixes.

Ship this refactor as its own deploy, with the disk backend still active and no change in behavior. If something breaks, you know the abstraction caused it.

For local development and tests, either run an S3-compatible emulator or keep using the disk implementation. The disk implementation is simpler, but it hides differences: missing-object errors look different, content-type handling differs, and metadata can be silently dropped. Run at least one test suite against the S3 backend.

Copyimport os, shutil, uuid
from typing import BinaryIO, Protocol
import boto3
from botocore.exceptions import ClientError

def new_key(tenant_id: int, category: str) -> str:
    # Policy-relevant attributes go in the prefix. The random suffix is non-guessable and never the user's filename.
    return f"tenants/{tenant_id}/{category}/{uuid.uuid4()}"

class Storage(Protocol):
    def put(self, key: str, stream: BinaryIO, content_type: str) -> None: ...
    def get(self, key: str) -> BinaryIO: ...
    def delete(self, key: str) -> None: ...
    def exists(self, key: str) -> bool: ...
    def url(self, key: str, expires_in: int) -> str: ...

class DiskStorage:
    def __init__(self, root: str): self.root = root
    def _path(self, key: str) -> str: return os.path.join(self.root, key)
    def put(self, key, stream, content_type):
        # The disk does not store content_type. The DB column is the source of truth.
        os.makedirs(os.path.dirname(self._path(key)), exist_ok=True)
        with open(self._path(key), "wb") as f: shutil.copyfileobj(stream, f)
    def get(self, key): return open(self._path(key), "rb")  # missing key -> FileNotFoundError
    def delete(self, key): os.remove(self._path(key))       # missing key -> error (S3 does not raise one)
    def exists(self, key): return os.path.exists(self._path(key))
    def url(self, key, expires_in): return f"/files/{key}"  # served by an authenticated app route

class S3Storage:
    def __init__(self, bucket: str): self.bucket, self.s3 = bucket, boto3.client("s3")
    def put(self, key, stream, content_type):
        self.s3.upload_fileobj(stream, self.bucket, key, ExtraArgs={"ContentType": content_type})
    def get(self, key): return self.s3.get_object(Bucket=self.bucket, Key=key)["Body"]  # missing -> ClientError
    def delete(self, key): self.s3.delete_object(Bucket=self.bucket, Key=key)
    def exists(self, key):
        try: self.s3.head_object(Bucket=self.bucket, Key=key); return True
        except ClientError as e:
            if e.response["Error"]["Code"] == "404": return False
            raise
    def url(self, key, expires_in):
        return self.s3.generate_presigned_url("get_object", Params={"Bucket": self.bucket, "Key": key}, ExpiresIn=expires_in)

def storage_from_env() -> Storage:
    # A configuration value picks the backend. Callers only ever see the Storage interface.
    if os.environ.get("STORAGE_BACKEND", "disk") == "s3": return S3Storage(os.environ["S3_BUCKET"])
    return DiskStorage(os.environ.get("UPLOAD_ROOT", "/var/app/uploads"))

A five-operation storage interface with disk and S3 (boto3) implementations chosen by an environment variable, plus a key generator that puts tenant and category in the prefix; comments mark where the two backends differ in error and metadata behavior.

Run the migration in phases: dual-write, fallback read, backfill, verify

With the interface in place, the migration becomes four phases that each change one behavior.

Dual-write. New uploads go to S3. While you still might revert, also keep a local copy. Deletes and overwrites must hit both stores during the whole backfill. Otherwise the backfill can copy a file a user already deleted, or upload a stale version over a newer one. Make the backfill skip any record whose updated timestamp is later than when the job started.

Fallback read. Reads try S3 first, then local disk. Log every fallback hit with the record ID. A fallback can hide bugs: a wrong key prefix or a broken write path still serves files from disk, so nothing looks broken until disk goes away. Treat each fallback hit as a defect to explain.

Backfill. Record per-file status in a table so the job is idempotent and can resume after a crash. Throttle it by files per second and bandwidth so it does not starve production disk I/O.

Verify. Compare each object's size and checksum against the database. Remove the fallback only after verification is clean and the fallback count has stayed at zero for a full traffic cycle.

Keep the local files until verification passes.

Copyimport hashlib, time
import boto3

s3 = boto3.client("s3")
BUCKET = "example-app-uploads"
MAX_FILES_PER_SEC = 20                 # throttle: protect production disk I/O
MAX_BYTES_PER_SEC = 10 * 1024 * 1024   # throttle: cap upload bandwidth

def sha256_of(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""):
            h.update(chunk)
    return h.hexdigest()

def backfill(db, job_started_at, last_id=0):  # db: psycopg connection
    while True:
        # Status filter makes reruns idempotent. Rows changed after start are skipped (dual-write owns them).
        rows = db.execute("""SELECT id, local_path, s3_key, size_bytes, sha256 FROM uploads
            WHERE id > %s AND backfill_status IS DISTINCT FROM 'copied'
              AND deleted_at IS NULL AND updated_at <= %s
            ORDER BY id LIMIT 500""", (last_id, job_started_at)).fetchall()
        if not rows:
            return
        for id_, path, key, size, digest in rows:
            last_id, start, status = id_, time.monotonic(), "mismatch"
            if sha256_of(path) == digest:  # never copy a file that disagrees with the DB
                with open(path, "rb") as f:
                    s3.put_object(Bucket=BUCKET, Key=key, Body=f, Metadata={"sha256": digest})
                if s3.head_object(Bucket=BUCKET, Key=key)["ContentLength"] == size:
                    status = "copied"
            # Race check: a delete or overwrite during the copy means our upload may be wrong.
            if db.execute("SELECT deleted_at IS NOT NULL OR updated_at > %s FROM uploads WHERE id = %s",
                          (job_started_at, id_)).fetchone()[0]:
                status = "recheck"  # verification phase re-compares this row against the DB
            db.execute("UPDATE uploads SET backfill_status = %s WHERE id = %s", (status, id_))
            db.commit()  # per-file progress survives a crash
            budget = max(1.0 / MAX_FILES_PER_SEC, size / MAX_BYTES_PER_SEC)
            time.sleep(max(0.0, budget - (time.monotonic() - start)))

A resumable, throttled backfill job that records per-file status, skips rows changed after the job started, flags rows that changed mid-copy, and checks checksum and size before marking a file copied.

Serve downloads and accept uploads with presigned URLs

Once S3 is authoritative, stop streaming bytes through the app. The app still decides who may access what, but it issues a short-lived signed URL and the client talks to S3 directly.

Downloads

Run the same authorization check you ran before serving from disk, then sign a GET for that single key. Set the response headers deliberately. Use Content-Disposition: attachment for anything user-supplied, and set an explicit Content-Type rather than trusting what the uploader claimed. A browser that renders an uploaded HTML or SVG file inline will run its scripts, and if the bucket shares a domain with your app, those scripts run in your origin. You can set these headers on the presigned GET with response-header overrides, or store them as object metadata at upload time. If a file needs per-request logic such as watermarking or audit logging, keep a proxy endpoint for that path only.

Uploads

Have the server generate the object key. Never let the client choose it. Sign a URL for that key with a fixed content type and a short expiry. Presigned POST supports a policy with a content-length-range condition, so S3 itself enforces the size limit. Presigned PUT has no equivalent. If you use PUT, check the object's size after the upload completes.

Trade-offs

• Revocation: A presigned URL is a bearer token. You cannot revoke one URL on its own, only the credentials that signed it. Short expiries reduce the damage from a leaked URL, but users then hit errors on slow connections or when they resume a download.
• Validation: Direct uploads move validation to after the bytes land. Treat each new object as quarantined until a job has checked its size, type and content.
• CORS: Browser uploads need a CORS rule on the bucket. Allow only your app's origins, the methods you actually use, and the headers you actually sign.

Copyimport uuid
import boto3

s3 = boto3.client("s3")
BUCKET = "example-app-uploads"
MAX_UPLOAD_BYTES = 25 * 1024 * 1024  # 25 MiB

def download_url(user, file_record):
    # Same authorization check the disk-backed endpoint used.
    if not user.can_read(file_record):
        raise PermissionError("forbidden")
    # Override response headers so S3 serves the file as a download with a
    # type we chose, not one the uploader claimed. This stops inline HTML or SVG.
    return s3.generate_presigned_url(
        "get_object",
        Params={
            "Bucket": BUCKET,
            "Key": file_record.s3_key,
            "ResponseContentType": file_record.safe_content_type,
            "ResponseContentDisposition": "attachment",
        },
        ExpiresIn=300,  # short-lived: a leaked URL works for 5 minutes at most
    )

def upload_form(user, content_type="application/pdf"):
    # The server picks the key. The client never chooses the path.
    key = f"quarantine/{user.id}/{uuid.uuid4()}"
    post = s3.generate_presigned_post(
        Bucket=BUCKET,
        Key=key,
        Fields={"Content-Type": content_type},
        Conditions=[
            {"Content-Type": content_type},  # S3 rejects a different type
            ["content-length-range", 1, MAX_UPLOAD_BYTES],  # S3 enforces size
        ],
        ExpiresIn=120,
    )
    # Return post["url"] and post["fields"] to the browser. Record the key so a
    # background job can validate the object before it leaves quarantine.
    return key, post

Using boto3, the server issues a presigned GET only after an authorization check, overrides Content-Type and Content-Disposition on that GET, and generates a presigned POST for a server-chosen quarantine key, with S3 enforcing the content type and size limit.

Close the gap between database records and objects, and make app servers stateless

Make every upload a two-step record

With direct uploads, the database and the bucket can disagree. A client can request a presigned URL and never upload, or upload a file the app never hears about. Treat each file as a state machine. Insert the row as pending before issuing the URL. When the client reports success, issue a HEAD request to confirm the object exists and that its size matches what was declared. Then run validation: a malware scan, image decoding to reject files that only claim to be images, and size limits. Only after validation passes does the row move to complete and become visible to users. Files that fail go to rejected, and their objects are deleted.

Reconcile, then expire

Even with this lifecycle, crashes leave gaps. Run a periodic reconciliation job that compares bucket listings against the table in both directions. Objects with no row are orphans, and you delete them after a grace period. Rows marked complete whose object is missing are dangling, and they need an alert, not silent cleanup. Write pending uploads under a dedicated prefix and add a bucket lifecycle rule that expires objects there after a day or two. Abandoned uploads then cost nothing to clean up.

Remove the last disk assumptions

Search image processing and background jobs for code that writes to fixed paths and expects another process to read them later. Pass object keys between jobs instead of file paths. If a library needs a local file, use an explicit per-job temp directory and delete it when the job finishes. Unmount any volumes shared between hosts. Then test it: scale to several instances, upload a file, terminate the instance that handled the upload, redeploy, and confirm the file is still served.

Lock down the bucket and weigh cost and engineering time

Hardening checklist

• Block public access at the bucket level, so one misconfigured ACL can't expose user files. Presigned URLs keep working without it.
• Least-privilege IAM: give the app role only s3:GetObject, s3:PutObject and s3:DeleteObject on the uploads/* prefix, never s3:*. A leaked credential then can't list the bucket or touch other prefixes. The example policy below shows this scoping.
• Default encryption, so every object is encrypted at rest without each code path having to set it.
• Versioning, so a buggy backfill that overwrites or deletes objects can be rolled back. Add a lifecycle rule that expires noncurrent versions, or storage grows unnoticed.

Cost

Every read becomes a network round trip instead of a local disk read. Downloads proxied through the app hold a worker for the whole transfer, so redirect to presigned URLs instead. A CDN lowers latency and egress, but it adds cache invalidation when objects are overwritten in place, which you can avoid by writing new keys. It also makes authorization for private files more complicated.

Illustrative estimate (assumed numbers, not a benchmark): GETs = page views × thumbnails per page = 2,000,000 × 30 = 60M. Request cost = 60M ÷ 1,000 × $0.0004 = $24. Egress = 60M × 20 KB ≈ 1,200 GB × $0.09 = $108.

In this example, egress costs far more than requests. Use your provider's actual rates.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to migrate file uploads to s3, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.