Node.js File Upload API Design: Multipart, Presigned URLs, Limits

How multipart/form-data uploads work and why buffering is the default mistake

A multipart/form-data request is one HTTP body split into parts. The Content-Type header declares a boundary string, and each part begins with -- plus that boundary. Each part carries its own headers: typically Content-Disposition with a field name and, for files, a filename, plus an optional Content-Type. The raw bytes come next. A final boundary with a trailing -- ends the body. The parser cannot know where a file ends until it sees the next boundary, so the body has to be scanned as it arrives.

The common mistake is to buffer the entire body before parsing, either with req.on('data') concatenation or with middleware that stores files in memory. Memory then grows with upload size multiplied by concurrent requests. Fifty clients sending 200 MB each means roughly 10 GB held in the heap (illustrative estimate), and the process runs out of memory well before the CPU becomes a constraint.

Stream instead. busboy emits a readable stream for each file part. Connect it to the destination (disk, a hashing transform, an object-store upload) with stream.pipeline. This matters for two reasons. If the destination is slow, backpressure pauses reads from the socket. If any stage fails, pipeline destroys every stream in the chain, so you don't leak half-open handles.

Proxying bytes through Node is reasonable when you transform content inline, need a single point for authorization, or are building a small internal tool. Avoid it for large files, high traffic, or slow clients. Each slow upload holds a socket and a request slot open for its full duration.

Copyconst http = require('node:http');
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');
const { pipeline } = require('node:stream/promises');
const busboy = require('busboy');

const UPLOAD_DIR = '/var/tmp/uploads';

http.createServer((req, res) => {
  if (req.method !== 'POST') return res.writeHead(405).end();
  // Parse parts as bytes arrive; cap file size and part count up front.
  const bb = busboy({ headers: req.headers, limits: { fileSize: 50 * 1024 * 1024, files: 1 } });
  const writes = [];

  bb.on('file', (field, file, info) => {
    // Never use the client-supplied info.filename as the on-disk path.
    const dest = path.join(UPLOAD_DIR, crypto.randomUUID());
    // pipeline applies backpressure to the socket and destroys both streams on failure.
    const done = pipeline(file, fs.createWriteStream(dest)).then(() => {
      if (file.truncated) throw Object.assign(new Error('file too large'), { status: 413 });
      return dest;
    });
    // Attach a handler immediately (avoids an unhandled rejection) and remove partial files.
    done.catch(() => fs.promises.rm(dest, { force: true }));
    writes.push(done);
  });

  bb.on('close', async () => {
    try {
      const saved = await Promise.all(writes);
      res.writeHead(201, { 'Content-Type': 'application/json' }).end(JSON.stringify({ saved }));
    } catch (err) {
      res.writeHead(err.status || 500).end(err.message);
    }
  });
  bb.on('error', () => { req.unpipe(bb); res.writeHead(400).end('malformed multipart body'); });
  req.pipe(bb);
}).listen(3000);

A plain Node HTTP server parses multipart/form-data with busboy as the body arrives. It pipes each file part to disk with stream.pipeline, so backpressure and errors propagate, and it enforces a size limit without ever buffering the whole upload in memory.

Where upload size limits must live: proxy, parser, timeouts, and counts

Uploads therefore need limits at several layers. Each layer catches a different failure, and no single one covers them all.

Proxy and load balancer

Set a maximum body size at the reverse proxy or load balancer. This is the cheapest place to reject an oversized request, because the bytes never reach Node. The trade-off is error quality. The proxy returns a generic 413 page or resets the connection, and clients often surface that as an opaque network failure rather than "file too large."

Parser

Inside the application, configure the multipart parser with a per-file byte limit, a total request limit, and caps on the number of files, fields, and parts. Count caps matter because a request made of thousands of tiny fields can exhaust memory and CPU without ever approaching the byte limit.

Content-Length is a hint, not a guarantee

Check Content-Length first and reject immediately if it exceeds the limit. Then keep counting bytes as they stream in anyway. The header is absent under chunked transfer encoding, and a client can send a value that doesn't match the body.

Timeouts

Configure Node's server request timeout, headers timeout, and socket idle timeout. Without them, a client trickling one byte every few seconds can hold a connection open indefinitely.

Aborting cleanly

When a limit trips mid-stream:

• Destroy the request stream and any downstream write stream or upload.
• Delete the partial file, or abort the in-progress object upload.
• Return a specific 413 that names the limit that was exceeded.

If you skip this cleanup, orphaned fragments accumulate quietly in temp directories and buckets.

Copyconst http = require('node:http');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const crypto = require('node:crypto');
const busboy = require('busboy');

const MAX_TOTAL = 50 * 1024 * 1024; // whole request; keep proxy limit slightly above this

function reject(res, limit) {
  // Name the limit so clients can show a useful error; close since we stop reading the body
  res.writeHead(413, { 'Content-Type': 'application/json', Connection: 'close' });
  res.end(JSON.stringify({ error: 'payload_too_large', limit }));
}

const server = http.createServer((req, res) => {
  // Hint only: NaN when absent (chunked), so the comparison is false and we fall through
  if (Number(req.headers['content-length']) > MAX_TOTAL) return reject(res, 'total');

  const bb = busboy({ headers: req.headers,
    limits: { fileSize: 10 * 1024 * 1024, files: 5, fields: 20, parts: 30 } });
  const partials = [];
  let seen = 0, failed = false;

  const abort = (limit) => {
    if (failed) return; failed = true;
    req.unpipe(bb); bb.destroy();             // stop parsing
    for (const p of partials) { p.ws.destroy(); fs.rm(p.file, { force: true }, () => {}); }
    reject(res, limit);
  };

  // Count real bytes regardless of what Content-Length claimed
  req.on('data', (chunk) => { seen += chunk.length; if (seen > MAX_TOTAL) abort('total'); });
  bb.on('file', (_name, stream) => {
    const file = path.join(os.tmpdir(), `upload-${crypto.randomUUID()}`);
    const ws = fs.createWriteStream(file);
    partials.push({ ws, file });
    stream.on('limit', () => abort('fileSize'));
    stream.pipe(ws);
  });
  ['files', 'fields', 'parts'].forEach((k) => bb.on(`${k}Limit`, () => abort(k)));
  bb.on('close', () => { if (!failed) res.writeHead(201).end(); }); // simplified: production code should await ws 'finish'
  req.pipe(bb);
});

server.headersTimeout = 15_000;  // slow header senders
server.requestTimeout = 120_000; // whole request must finish within 2 minutes
server.setTimeout(30_000);       // destroy sockets idle for 30s (byte tricklers)
server.listen(3000);

A Node.js upload handler using busboy that rejects early on Content-Length, still counts streamed bytes, enforces per-file and count limits, sets server timeouts, and on any mid-stream limit hit destroys streams, deletes partial temp files, and returns a 413 naming the limit.

The presigned URL pattern end to end

The cleanest fix for most of those problems is to stop receiving the bytes at all. Your API is good at deciding whether an upload should happen: who is asking, what it is for, and how big it may be. Object storage is better at receiving the bytes. The presigned URL pattern splits the work along that line.

The flow:

1. The client asks for permission and declares filename, size, content type, and purpose (for example, avatar or invoice-attachment).
2. The API authenticates the caller and checks the declared values against per-purpose rules. It records a pending upload and picks the object key itself. The client never chooses the key.
3. The API returns a presigned URL that expires within minutes and is scoped to that one key.
4. The client sends the bytes directly to S3. Your Node.js processes never hold the file in memory or on disk.
5. Finalization marks the upload complete.

Finalization can be client-driven or event-driven. With a client callback, the client calls POST /uploads/:id/complete. This is simple and synchronous, but clients crash, close tabs, and lose connectivity. Some uploads will succeed and never be reported. With S3 event notifications, finalization happens even if the client vanishes. The trade-off is that delivery is at-least-once and unordered, so your handler must be idempotent and must tolerate events that arrive late or duplicated.

Many teams run both. The callback gives fast feedback, and the event acts as a backstop. A periodic sweep expires pending records that neither path ever completed.

This pattern suits teams handling user-facing uploads at scale, where bandwidth and connection time through the API become real costs.

Copyimport express from "express";
import { randomUUID } from "node:crypto";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { requireAuth, db } from "./app"; // your auth middleware and data layer

const s3 = new S3Client({ region: "us-east-1" });
const BUCKET = "example-user-uploads";

// Per-purpose rules: the API decides what is allowed, not the client.
const RULES: Record<string, { maxBytes: number; types: string[] }> = {
  avatar: { maxBytes: 5 * 1024 * 1024, types: ["image/png", "image/jpeg"] },
  "invoice-attachment": { maxBytes: 20 * 1024 * 1024, types: ["application/pdf"] },
};

const app = express();
app.use(express.json());

app.post("/uploads", requireAuth, async (req, res) => {
  const { purpose, size, contentType } = req.body;
  const rule = RULES[purpose];
  if (!rule || !Number.isInteger(size) || size <= 0 || size > rule.maxBytes || !rule.types.includes(contentType)) {
    return res.status(400).json({ error: "upload not allowed" });
  }

  // Server-chosen key: the client's filename never becomes part of the path.
  const id = randomUUID();
  const key = `${purpose}/${req.user.id}/${id}`;
  await db.uploads.insert({ id, key, userId: req.user.id, purpose, size, status: "pending" });

  // Signed for this one key and content type; expires in 5 minutes.
  // Note: a presigned PUT does not enforce the declared size. Re-check the real
  // object size at finalization, or use a presigned POST with content-length-range.
  const command = new PutObjectCommand({ Bucket: BUCKET, Key: key, ContentType: contentType });
  const url = await getSignedUrl(s3, command, { expiresIn: 300 });

  res.status(201).json({ uploadId: id, url, method: "PUT", headers: { "Content-Type": contentType } });
});

An Express endpoint that validates the declared purpose, size, and content type, records a pending upload under a server-chosen key, and returns a five-minute S3 presigned PUT URL scoped to that key using AWS SDK for JavaScript v3.

Presigned PUT vs presigned POST vs S3 multipart upload

A presigned PUT is the simplest option: the server signs one URL for one key, and the client sends the raw bytes in a single request. However, the signature does not enforce a maximum size, so a client holding the URL can upload far more than you intended until the URL expires. Pair it with a size check after upload, or use one of the other two options when size matters.

A presigned POST attaches a signed policy document that the storage service evaluates before accepting the upload. The policy can include content-length-range and Content-Type conditions, so oversized or mistyped uploads are rejected before they are stored. The drawback is the client API. The browser or SDK must send a multipart/form-data request with the policy fields in the right order, which is clumsy outside an HTML form.

S3 multipart upload is the right choice for large files. The server initiates the upload and presigns each part on request. The client uploads parts in parallel, retries failed parts individually, and records the ETag returned for each part. The server then completes the upload with the ordered ETag list, or aborts it. Each part URL can expire within minutes, because the client asks for a new one only when it is ready to send that part. A dropped connection therefore costs one part, not the whole file, and a resumed session continues from the last confirmed part.

Configure a bucket lifecycle rule that aborts incomplete multipart uploads a few days after they start. Parts from abandoned uploads are billed as storage, but they do not appear as normal objects in listings, so these costs can grow without anyone noticing.

Copy#!/usr/bin/env bash
set -euo pipefail

BUCKET="example-uploads-bucket"  # placeholder bucket name

# Lifecycle rule: abort any multipart upload that has not completed
# 3 days after it was initiated. Aborting deletes the uploaded parts,
# which stop being billed as storage.
# An empty Prefix filter applies the rule to every key in the bucket.
cat > lifecycle.json <<'EOF'
{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart-uploads",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 3 }
    }
  ]
}
EOF

# Warning: this call REPLACES the bucket's entire lifecycle configuration.
# If the bucket already has rules, merge them into lifecycle.json first.
aws s3api put-bucket-lifecycle-configuration \
  --bucket "$BUCKET" \
  --lifecycle-configuration file://lifecycle.json

# Confirm the rule is in place.
aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET"

# Orphaned parts do not show up in a normal object listing.
# This command lists in-progress uploads, so you can see what the rule will clean up.
aws s3api list-multipart-uploads --bucket "$BUCKET"

Uses the AWS CLI to apply an S3 lifecycle rule that aborts multipart uploads still incomplete three days after they started, then lists in-progress uploads, which do not appear as normal objects.

Treat every upload as untrusted: keys, content verification, and quarantine

A presigned URL hands a write capability to a client you don't control, so everything the client sends about the file should be treated as a claim to check, not a fact.

Generate keys on the server

Never build the object key from the client's filename. Filenames can contain path separators, .. segments, Unicode that renders identically to other characters, or names that collide with another user's file. Generate the key yourself, typically from a tenant or user prefix plus a random identifier such as a UUID. Store the original filename as a database column or object metadata. Use it only for display and for the Content-Disposition header on download, and escape it there too.

Verify content from bytes, not headers

The Content-Type header and the file extension are both chosen by the client. Once the upload completes, read the first few kilobytes of the object and inspect its magic bytes. A PNG starts with a fixed eight-byte signature, and a PDF starts with %PDF. Compare the detected type against your allowlist and against what the client declared. Treat a mismatch as a rejection.

Quarantine first

Have presigned URLs write only to a quarantine prefix that no download path can read. A background job validates the object, runs malware scanning if your product requires it, and then copies the object to its final prefix or deletes it.

This creates a UX problem: the browser reports success before validation has run. Show the user an explicit "processing" state rather than "uploaded." When a file is rejected, give a specific reason the user can act on, such as "file is not a valid PNG," instead of letting the file silently disappear.

Modeling uploads as a state machine in the database

Those user-facing messages only stay accurate if the backend tracks each upload explicitly. Give every upload a row with a status column, and allow only the transitions below.

Enforce each transition as a conditional update that names the expected current state, such as ... WHERE id = ? AND status = 'pending'. Storage notifications can arrive more than once, and clients retry their "complete" call. Only the first attempt matches a row. Later attempts affect zero rows, so the handler reads the current state and returns it. It does not run verification a second time. This approach does not need locks or deduplication tables. The row is the lock.

A scheduled cleanup job closes the gaps in both directions:

• Stale pending rows. Select rows older than the URL lifetime plus a grace period. Then check storage for each object. If the object exists, the completion event was lost, so advance the row to uploaded and queue verification. If it does not exist, mark the row expired.
• Terminal non-verified rows. Delete the objects for rejected and expired rows.
• Orphaned objects. List keys under the upload prefix. Delete any key older than the grace period that has no matching row.

Operational pitfalls, alternatives, and impact on cost and engineering time

The cleanup sweep only works if uploads actually land, and most first-week incidents happen before that point.

Pitfalls that cause incidents

• Bucket CORS. Browser uploads fail on the preflight request unless the bucket allows your origin, the PUT/POST method, and the headers you sign. Multipart clients also need ETag exposed, or they cannot complete the upload.
• Clock skew. The signature embeds the signing host's timestamp. A drifting server clock makes URLs expire early or fail outright, so run NTP on every host that signs URLs.
• Expiry. Short expiries limit replay risk, and long ones tolerate slow mobile links. Size the expiry to the largest allowed file at a pessimistic bandwidth. For anything larger, use multipart with per-part URLs.
• Issuance limits. Rate-limit URL creation per user and enforce quotas when you issue the URL, not after the upload. Otherwise one client can fill your bucket.

Choosing an approach

Proxy through Node when files are small or you must inspect them inline. Use presigned uploads for everything else. Presigned uploads tie you to the provider's signing API, so wrap it behind an interface and run an S3-compatible emulator in CI.

Illustrative estimate (assumed numbers, not a benchmark): Proxied transfer cost = uploads × average size × per-GB processing cost. Assume 100,000 uploads × 0.05 GB × $0.01 = $50/month, plus the compute held by long-lived streams. Weigh that against an assumed 5 extra engineer-days to build CORS, the state machine, and emulator setup.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to node.js file upload api, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.