What is Node.js stream backpressure and how does it work?
Backpressure is how a stream tells the stage feeding it to slow down. Without it, a fast producer such as a local disk read keeps handing data to a slow consumer such as a client on a weak connection. The data that hasn't been written yet piles up in memory.
The signal is the return value of write(). Every writable stream keeps an internal buffer. When a call to write() brings the buffered amount to highWaterMark or above, write() returns false. The chunk is still accepted, so nothing is lost. The false is a request: stop writing until I catch up. When the buffer has fully drained, the writable emits 'drain', and the producer can resume.
The protocol is advisory. A writable never rejects a chunk for being over the limit. If the producer ignores false and keeps calling write(), every chunk is appended to the buffer. Memory then grows with the gap between read speed and write speed, multiplied by the file size. A multi-gigabyte upload served to a slow client can end up almost entirely in the heap.
In a chain, the signal propagates hop by hop. A Transform has two buffers, a writable side and a readable side. When the slowest stage fills up, each upstream stage fills and pauses in turn:
file ──► [ Readable ] ──► [ Transform ] ──► [ Writable ] ──► slow socket
[ read buf ] [ write buf | read buf ] [ write buf ]
▲ ▲ │ │ fills to highWaterMark
│ pause reading │ stop │ write()===false│
└──────────────────┴────────────┴─────────────────┘
│ resume │ │ 'drain' │
└──────────────────┴────────────┴─────────────────┘
The socket is what sets the pace. Once its buffer is full, the stages above it stop pulling data. The file is then read only as fast as the client can receive it.
const { Writable } = require('node:stream');
const { once } = require('node:events');
const chunk = Buffer.alloc(64 * 1024); // 64 KiB per write() call
const CHUNKS = 200; // ~12.5 MiB in total
// A deliberately slow consumer with a 16 KiB buffer limit.
function slowSink() {
return new Writable({
highWaterMark: 16 * 1024,
write(data, encoding, callback) {
setTimeout(callback, 5); // pretend each chunk takes 5 ms to flush
},
});
}
// Ignores the return value of write(), so every chunk is buffered.
function ignoreBackpressure() {
const sink = slowSink();
for (let i = 0; i < CHUNKS; i++) sink.write(chunk); // false is discarded
console.log('ignored: bytes buffered =', sink.writableLength); // ~12.5 MiB
sink.end();
}
// Pauses on false and resumes on 'drain'.
async function respectBackpressure() {
const sink = slowSink();
let peak = 0;
for (let i = 0; i < CHUNKS; i++) {
const ok = sink.write(chunk);
peak = Math.max(peak, sink.writableLength);
if (!ok) await once(sink, 'drain'); // wait until the buffer empties
}
sink.end();
console.log('respected: peak bytes buffered =', peak); // one chunk, 64 KiB
}
ignoreBackpressure();
respectBackpressure().catch(console.error);
Writing the same data to a slow writable shows that when write()'s false return value is ignored, writableLength grows to hold the whole payload, while waiting for 'drain' keeps it at about one chunk.
What does highWaterMark actually control?
highWaterMark is a threshold, not a memory cap. On a writable, write() returns false once the buffered amount reaches it, but the chunk is still accepted. A producer that ignores that return value can push the buffer far past the limit, and a single oversized chunk exceeds it on its own. On a readable, the value tells the stream when to stop calling _read() to refill its internal buffer.
The unit depends on the mode. Byte streams count bytes. objectMode streams count objects, whatever their size, so 16 small objects and 16 large parsed records both count as 16. Every stage has its own value. A Transform has separate readable and writable buffers, so a three-stage pipeline holds several buffers at once.
| Setting | Worst-case memory per stream | Overhead | Suits |
|---|---|---|---|
| Low | Small, predictable | More read/write calls, more event-loop turns and syscalls |
Many concurrent connections, memory-constrained containers |
| Default | Moderate | Balanced | Most file and network I/O |
| High | Large, multiplied by every buffer in the chain | Fewer, larger chunks and less per-chunk cost | Few long transfers where throughput matters |
Concurrency is what makes high values risky. As an illustrative estimate, assume 1 MiB buffers, four buffers per pipeline and 500 concurrent downloads. That allows up to about 2 GiB of buffered data before any leak is involved.
Tuning is secondary to honouring the signal. A well-chosen highWaterMark makes no difference if the code ignores false from write() and keeps pushing data.
Which anti-patterns cause memory spikes when processing large files?
The most common offenders bypass that signal entirely or never create it.
fs.readFileon large inputs. The whole file is held in one Buffer before your code sees a byte, so peak memory is at least the file size. Fix:fs.createReadStream(path).- Collecting chunks, then
Buffer.concat(chunks). This isreadFilerebuilt by hand. The concat briefly holds the chunk array and the joined copy at once, roughly doubling peak memory. Fix: process each chunk as it arrives. - Unchecked
write()in a loop.for (...) out.write(row)ignores thefalsereturn, so the writable buffers without limit. Fix:if (!out.write(row)) await once(out, 'drain'). 'data'listeners with slow async handlers. The emitter does not wait for your promise, so chunks keep arriving while earlier ones are still in flight. Fix:for await (const chunk of stream), which pulls the next chunk only after the loop body finishes.- Un-awaited promises in a loop.
items.forEach(i => upload(i))starts every upload at once and keeps every payload alive until it settles. Fix: await each one, or cap how many run concurrently. - Mixing
'data',.pipe()and async iteration on one stream. Each consumer drives flow control on its own terms, so you get one consumer starving another, data silently skipped, or buffering that nothing pauses.
Illustrative estimate (assumed inputs, not a measurement): a disk read at 500 MB/s feeds a consumer that handles 50 MB/s. Without backpressure, the backlog grows by 450 MB every second, reaching 4.5 GB after 10 seconds. With backpressure, the reader pauses whenever the buffers fill, so buffered data stays near the highWaterMark of each stream in the chain, whether the file is 1 GB or 100 GB.
// rows.mjs: ESM, so top-level await works
import { createReadStream, createWriteStream } from 'node:fs';
import { once } from 'node:events';
const SRC = './data/export-large.ndjson';
const sink = createWriteStream('./out/processed.ndjson');
const transform = (chunk) => chunk; // placeholder per-chunk work
const upload = async (item) => {}; // placeholder slow async call
// BROKEN: whole file sits in one Buffer before any work starts
// const all = await (await import('node:fs/promises')).readFile(SRC);
// BROKEN: false return from write() is ignored, so the writable buffers without limit
// for (const row of rows) sink.write(row);
// BROKEN: 'data' does not wait for the handler's promise, so chunks pile up in flight
// createReadStream(SRC).on('data', async (chunk) => { await upload(chunk); });
// BROKEN: every upload starts at once and every payload stays alive until it settles
// items.forEach((item) => upload(item));
// FIXED: pull one chunk at a time and honor the writable's backpressure signal
for await (const chunk of createReadStream(SRC)) {
if (!sink.write(transform(chunk))) {
// Stop reading until the writable has flushed below its highWaterMark
await once(sink, 'drain');
}
}
sink.end();
await once(sink, 'finish');
// FIXED: await sequentially (or use a bounded worker pool) instead of firing all at once
const items = ['a.json', 'b.json', 'c.json'];
for (const item of items) {
await upload(item);
}
Broken patterns (readFile, unchecked write(), async 'data' handlers, un-awaited promises) shown as comments next to a fixed version that uses for await and waits for 'drain', so buffered data stays near highWaterMark.
Why use stream.pipeline instead of .pipe()?
.pipe() handles backpressure correctly but does nothing about errors. If the source emits 'error', the destination is not closed. If the destination fails, the source is unpiped but not destroyed. Each stream needs its own error listener, and every listener has to clean up the other streams. Teams often skip this, which leaves leaked file descriptors from fs.createReadStream and half-open sockets that stay open until a timeout closes them.
stream.pipeline and its promise form from stream/promises connect the same chain. On the first error in any stream, they destroy every stream in the chain and report that one error to a single callback or a rejected promise. Most of the engineering time saved comes from this. It removes a whole class of leak and cleanup bugs that tend to appear only under load and are hard to reproduce.
| Approach | Error propagation | Cleanup on failure | Backpressure | Ergonomics |
|---|---|---|---|---|
.pipe() chain |
None; one listener per stream | Manual; easy to leak FDs and sockets | Automatic | Concise, but unsafe without extra code |
pipeline (callback) |
First error goes to one callback | Destroys all streams | Automatic | Good |
pipeline (stream/promises) |
Rejects with first error | Destroys all streams | Automatic | Best; works with await and try/catch |
Manual write()/'drain' |
Fully manual | Fully manual | Only if you check write()'s return value |
Verbose and error-prone |
Destroy-on-error is the wrong behavior in some cases. You may want to keep a socket open so you can send an error response. You may want to retry one stage without reopening the source. For cases like these, keep the long-lived stream out of the destroy path and handle its lifecycle explicitly.
const http = require('node:http');
const { createReadStream, createWriteStream } = require('node:fs');
const { stat } = require('node:fs/promises');
const { createGzip } = require('node:zlib');
const { pipeline } = require('node:stream/promises');
// Batch job: one await covers the whole chain. Backpressure is automatic.
// On the first error, all three streams are destroyed, so no FDs leak.
async function compressFile(src, dest) {
try {
await pipeline(createReadStream(src), createGzip(), createWriteStream(dest));
} catch (err) {
// One place to handle failure. The streams are already cleaned up.
console.error(`compress failed for ${src}:`, err.code ?? err.message);
throw err;
}
}
// HTTP case: pipeline destroys `res` on error, so check for expected
// failures (like a missing file) before handing `res` to pipeline.
http.createServer(async (req, res) => {
const path = '/var/data/export.csv';
try {
await stat(path);
} catch {
res.writeHead(404).end('not found'); // socket is still ours to answer on
return;
}
res.writeHead(200, { 'Content-Type': 'text/csv' });
try {
await pipeline(createReadStream(path), res);
} catch (err) {
// Too late for an error response: pipeline has already destroyed res.
console.error('stream failed mid-response:', err.code ?? err.message);
}
}).listen(8080);
The promise form of pipeline from 'node:stream/promises' gives one try/catch per chain and destroys every stream on failure. The HTTP handler checks for a missing file before streaming, so it can still send a 404 before pipeline takes ownership of the response.
How do you stream a large file through gzip to an HTTP response with pipeline?
Set the response headers first: Content-Type for the underlying file and Content-Encoding: gzip. Do not set Content-Length, because the compressed size is unknown until gzip finishes. Node will use chunked transfer encoding instead. Then await pipeline(fs.createReadStream(path), zlib.createGzip(), res) using the promise-based pipeline from stream/promises, and wrap it in try/catch.
The error path depends on how far the response got. If res.headersSent is false, the failure happened before any bytes left, such as a missing file, and you can still send a 404 or 500. If headers are already out, the status line is committed. Log the error and let pipeline tear the socket down so the client sees a truncated transfer rather than a corrupt "success."
Client disconnects are handled by the same mechanism. When the socket closes early, pipeline sees the response finish prematurely, destroys the gzip and file streams, and rejects with ERR_STREAM_PREMATURE_CLOSE. The file descriptor is released and no further disk reads happen. Treat that error code as expected noise in your logs, not as an incident.
For object-storage uploads, the shape depends on the client library. If it exposes a writable upload stream, make it the last stage of the pipeline. If it takes a readable body, pass the gzip stream as that body. In that case, await both the source-to-gzip pipeline and the upload call, because a failure on either side must cancel the other.
Backpressure runs end to end. A client on a slow mobile link sets the pace for disk reads and compression, so each request holds its file handle and buffers for as long as that client takes to download. This is correct behaviour, but teams can be caught out when long-lived requests pile up, so plan timeouts and connection limits around it.
const http = require('node:http');
const fs = require('node:fs/promises');
const zlib = require('node:zlib');
const { pipeline } = require('node:stream/promises');
const FILE = '/var/data/exports/report.csv'; // placeholder path
http.createServer(async (req, res) => {
// Open the file before pipeline() owns res. If any stage fails, pipeline destroys res,
// so a 404 must be sent before the pipeline starts.
let fh;
try {
fh = await fs.open(FILE, 'r');
} catch (err) {
res.statusCode = err.code === 'ENOENT' ? 404 : 500;
return res.end();
}
// No Content-Length because the compressed size is unknown. Node falls back to chunked encoding.
res.writeHead(200, { 'Content-Type': 'text/csv', 'Content-Encoding': 'gzip' });
try {
// Each stage pulls only as fast as the next one drains, so a slow client paces disk reads.
// The read stream closes the file handle when it ends or is destroyed (autoClose).
await pipeline(fh.createReadStream(), zlib.createGzip(), res);
} catch (err) {
// Client aborted. pipeline already destroyed the gzip and file streams and released the fd.
if (err.code === 'ERR_STREAM_PREMATURE_CLOSE') return;
// Headers are already committed. res was destroyed, so the client sees a truncated transfer.
console.error('gzip stream failed after headers were sent:', err);
}
}).listen(8080);
Streams a file through gzip to an HTTP response with the promise-based pipeline. The file is opened first so a missing file can still return 404, and client aborts surface as ERR_STREAM_PREMATURE_CLOSE after the upstream streams are torn down.
How do you write custom Transforms and async-iteration consumers that respect backpressure?
When you write your own stages, backpressure is only as good as the contract each stage honours. In a Transform, the rule is to call the callback only after the chunk is fully handled. The stream machinery won't hand you the next chunk until that callback fires, so if you invoke it early and keep working in the background, you have quietly removed the brake. Emit output with push() (zero, one or many times per input chunk), or pass a single result as the callback's second argument. Keep only a small carry-over in instance state, such as a trailing partial line, and emit it in _flush. A transform that appends every chunk to an array and processes it at the end is a buffered read with extra steps.
Consuming with for await...of gives you backpressure for free. The iterator requests the next chunk only when the loop body finishes, so an await on a database write or HTTP call pauses reading from the source. Replace that await with a fire-and-forget call, or collect promises into an array for a later Promise.all, and the loop drains the file as fast as the disk allows. Every pending promise then pins its chunk in memory. If you need concurrency, bound it explicitly with a fixed-size pool and await a free slot before you continue.
Async generators passed directly to pipeline give you transform behaviour without subclassing, with the same pull-based semantics and error propagation. As a rule of thumb, use async iteration and generators in application code, where the logic is local and readability matters. Use Transform classes for reusable library components that need configurable options, object mode and a conventional stream interface.
How do you measure memory usage and know when streaming is the wrong tool?
Measure under realistic load before trusting any pipeline. Sample process.memoryUsage() on an interval while a large file moves through, and record all four fields. rss is what the operating system and your container limit see. heapUsed covers JavaScript objects only. external and arrayBuffers track memory backing Buffer instances, which is allocated outside the V8 heap. A pipeline that buffers too much can show a flat heapUsed while rss climbs, so a heap-only dashboard misses the problem.
Also log the return value of write() and count drain events. If write() never returns false on a slow destination, something upstream is ignoring the signal. Third-party streams and older libraries sometimes discard that return value or buffer internally, so wrapping them in pipeline does not guarantee bounded memory. Measure each one instead of assuming it behaves.
Streaming is the wrong tool for small, bounded payloads. A config file or a short request body is simpler with readFile, and that code is cheaper to maintain. It is also the wrong tool for operations that need the whole input: sorting, parsing a single JSON document, and some image transforms. Streaming only moves that memory requirement to one stage. It does not remove it. In those cases, spill to temp files, switch to chunked or line-delimited formats, or offload the work to a separate job or service.
Backpressure is per pipeline. A thousand concurrent downloads each stay bounded, but together they can still exhaust memory, so you need concurrency limits or admission control at the service level. Streaming code costs more to write and test. Unbounded buffering costs more when it causes an incident.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nodejs stream backpressure, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.