Node.js Stream CSV Export: Backpressure Without Memory Spikes

Why does a naive Node.js CSV export run out of memory?

The naive version of an export endpoint is usually three lines long. It runs one query that returns every matching row, maps each row to a CSV line and joins them into a single string, then hands that string to res.send(). It works in development and in early production, while the largest customer has a few thousand rows.

The problem is that the whole dataset sits in memory several times at its peak:

1. Row objects. The database driver materializes every row as a JavaScript object. Each object carries per-property and per-value overhead that is often larger than the data it holds.
2. Intermediate strings. Each row becomes its own CSV line string. join() then allocates one large string containing all of them. Until garbage collection runs, the per-line strings and the combined result coexist.
3. The response buffer. To write the string to the socket, Node.js encodes it into bytes, which is one more full-size copy.

Illustrative estimate. Assume 2 million rows at about 200 bytes of CSV each. That is roughly 400 MB of CSV text before any object overhead. Add the row objects and the encoded buffer, and one request can need well over a gigabyte. A few concurrent exports can exhaust the heap or trigger the container's memory limit.

A streaming CSV export fixes this structurally. It reads rows in batches or through a cursor, converts each row to CSV as it arrives and writes the bytes to the response as the client consumes them. Memory then depends on the batch size, not the size of the result. Backend and platform engineers need this pattern when export size grows with customer data and they cannot cap it ahead of time.

Copy// Naive export: correct output, but memory grows with the size of the result.
const express = require('express');
const { Pool } = require('pg');

const app = express();
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

// Quote a value per RFC 4180: wrap in quotes, double any embedded quotes.
function csvField(value) {
  if (value === null || value === undefined) return '';
  return '"' + String(value).replace(/"/g, '""') + '"';
}

app.get('/exports/orders.csv', async (req, res) => {
  // Copy 1: the driver materializes EVERY matching row as a JS object.
  const { rows } = await pool.query(
    'SELECT id, customer_id, total, created_at FROM orders WHERE account_id = $1',
    [req.query.accountId]
  );

  // Copy 2: one string per row, then join() allocates one huge combined string.
  // The per-line strings and the joined result coexist until GC runs.
  const header = 'id,customer_id,total,created_at';
  const lines = rows.map((r) =>
    [r.id, r.customer_id, r.total, r.created_at.toISOString()].map(csvField).join(',')
  );
  const csv = header + '\n' + lines.join('\n') + '\n';

  // Copy 3: res.send() encodes the full string into a Buffer before writing.
  res.set('Content-Type', 'text/csv; charset=utf-8');
  res.set('Content-Disposition', 'attachment; filename="orders.csv"');
  res.send(csv);
});

app.listen(3000);

The load-all-rows, build-one-string, res.send pattern, annotated with the three points where the full dataset is held in memory at once.

Who should use a streaming CSV export?

Use this pattern if you own an endpoint whose output size you do not control. The typical cases are:

• Backend engineers building export or reporting endpoints in multi-tenant applications, where the largest tenant's data keeps growing and one large export runs in the same process as everyone else's requests.
• Platform and SRE engineers running Node.js services under container memory limits, where a single buffered response can push a pod over its limit and get it killed, taking in-flight requests with it.
• Teams that already see the symptoms: out-of-memory restarts that line up with export traffic, heap growth during downloads, or support tickets about exports that time out for large accounts.

You probably do not need it if the export has a hard, enforced row cap and the capped result fits comfortably in memory. In that case the naive version is simpler and fine. At the other end, if exports are very large or take a long time to generate, streaming is still the right way to produce the file, but a background job that writes to storage is often a better way to deliver it. The section on when to avoid streaming directly to the client covers that trade-off.

How does a streaming CSV export work?

A streaming export is a chain of stages connected by Node.js streams, and the client's reading speed drives it:

1. The client reads from the socket. As bytes leave, the HTTP response's internal buffer shrinks.
2. The response signals it can take more. When its buffer drops below highWaterMark, the upstream stage is allowed to write again.
3. Each stage pulls from the one before it. Optional gzip compresses the next chunk, and the CSV transform converts the next row to a line of text.
4. The source fetches only when asked. A cursor or async generator wrapped in Readable.from() requests the next row, and when the current batch runs out, it queries the database for the next one.

When the client slows down or stops reading, this chain runs in reverse. The response buffer fills, write() returns false, each stage stops pulling and the source stops querying the database. Nothing accumulates without bound, so the amount of data in memory at any moment is roughly one database batch plus whatever each stage's buffer holds. That amount does not grow with the size of the export.

Illustrative estimate. Assume a batch size of 1,000 rows at about 200 bytes of CSV each. One batch is roughly 200 KB of CSV text, plus the overhead of 1,000 row objects and the small per-stream buffers. That is the working set whether the export has ten thousand rows or ten million. Only the duration changes.

stream.pipeline connects the stages, propagates this flow control and handles errors and cleanup for the whole chain. The following sections cover each part in turn: the backpressure signal, why pipeline is preferred over .pipe(), a complete implementation and how to handle failures.

How does backpressure work in Node.js streams?

Backpressure is a signal, not a mechanism that stops you. Every writable stream has an internal buffer, and writable.write() returns false once that buffer exceeds highWaterMark. The data is still accepted. Node only tells you to stop. When the buffer empties, the stream emits 'drain', and that is your cue to resume.

A hand-written loop that reads a row from the cursor and calls res.write() without checking the return value ignores that signal. Suppose the database produces rows faster than a slow client on a mobile link reads them. Every row that cannot go out over the socket is held in the response's buffer. The loop never pauses, so the buffer grows without limit, and the process's memory follows the speed gap between the database and the client.

In object mode, highWaterMark counts objects, not bytes, with a default of 16. Sixteen narrow rows take up very little memory. Sixteen rows with large JSON or text columns can take up a lot. Byte-mode streams, such as gzip and the HTTP response, measure in bytes.

You can tune the threshold with the highWaterMark option on each stream you construct, and with the cursor's fetch batch size. The defaults are usually fine, because throughput depends on keeping data moving, not on large buffers.

CopyDB cursor ──► CSV transform ──► gzip (optional) ──► HTTP response ──► client
    ◄─────────────── pressure flows back when the client slows ───────────

Copyconst { once } = require('node:events');
const { Readable, Transform } = require('node:stream');
const { pipeline } = require('node:stream/promises');
const zlib = require('node:zlib');

// Naive CSV escaping, enough for the example.
const toCsvLine = (row) =>
  Object.values(row).map((v) => `"${String(v).replace(/"/g, '""')}"`).join(',') + '\n';

// BAD: write() returning false is ignored. A slow client means unsent bytes
// accumulate in res's buffer while the cursor keeps producing rows.
async function exportNaive(cursor, res) {
  for await (const row of cursor) res.write(toCsvLine(row));
  res.end();
}

// GOOD (manual): stop pulling rows when write() returns false and wait for
// 'drain'. While for-await is paused, the cursor is not asked for more rows.
// Note: once() rejects on 'error', but a client that disconnects mid-wait
// needs extra handling (e.g. listen for 'close'); pipeline() covers that.
async function exportManual(cursor, res) {
  for await (const row of cursor) {
    if (!res.write(toCsvLine(row))) await once(res, 'drain');
  }
  res.end();
}

// GOOD (idiomatic): pipeline() wires backpressure through every stage and
// destroys all streams if any stage fails or the client goes away.
async function exportPipeline(cursor, res) {
  const toCsv = new Transform({
    writableObjectMode: true, // input side counts rows (default highWaterMark 16)
    transform(row, _enc, cb) { cb(null, toCsvLine(row)); }, // output side counts bytes
  });
  res.setHeader('Content-Type', 'text/csv');
  res.setHeader('Content-Encoding', 'gzip');
  await pipeline(Readable.from(cursor), toCsv, zlib.createGzip(), res);
}

module.exports = { exportNaive, exportManual, exportPipeline };

Contrasts a write loop that ignores write()'s return value with one that awaits 'drain', and with stream.pipeline(), which carries backpressure from the HTTP response through gzip and the CSV transform back to the database cursor.

Why use stream.pipeline instead of .pipe()?

Chaining .pipe() moves data and respects backpressure, but it does not manage failure. Each .pipe() call links only two streams. When a stage errors, the error is not forwarded to the other stages, and nothing is destroyed for you. If the destination fails, for example because the HTTP response errors when the client drops the connection, .pipe() unpipes the source and stops there. The source is still open.

For a CSV export, the source is a database cursor. A cursor that is paused but not destroyed keeps its connection checked out of the pool. A few hundred abandoned exports can exhaust the pool, and later requests then hang while they wait for a connection. Neither the logs nor the cursor will report an error.

stream.pipeline treats the whole chain as one unit. An error from any stage destroys every stream in the chain, which closes the cursor and releases its connection. The callback then fires once with that error. The promise form from stream/promises gives you something you can await inside a try/catch. It also accepts an AbortSignal, so you can tie cancellation to a request timeout or a disconnect handler.

Use pipeline by default. Leaked connections and hung requests rarely appear in development. They show up later under real traffic, when they are much harder to trace back to their cause.

Copyimport { createServer } from 'node:http';
import { Transform } from 'node:stream';
import { pipeline } from 'node:stream/promises';
// Placeholder: returns an object-mode Readable backed by a pooled DB cursor.
import { openOrderCursor } from './db.js';

const csvCell = (v) => `"${String(v ?? '').replaceAll('"', '""')}"`;

createServer(async (req, res) => {
  // One signal cancels the whole chain: on timeout or client disconnect.
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), 60_000);
  res.on('close', () => { if (!res.writableFinished) ac.abort(); });

  res.writeHead(200, { 'Content-Type': 'text/csv' });
  res.write('id,customer,total\n');

  const toCsv = new Transform({
    writableObjectMode: true, // rows in, CSV text out
    transform(row, _enc, cb) {
      cb(null, [row.id, row.customer, row.total].map(csvCell).join(',') + '\n');
    },
  });

  try {
    // Backpressure flows end to end; completion is a single awaitable promise.
    await pipeline(
      openOrderCursor('SELECT id, customer, total FROM orders'),
      toCsv,
      res,
      { signal: ac.signal },
    );
  } catch (err) {
    // By the time we get here, every stream (including the cursor) is destroyed,
    // so its connection has been returned to the pool.
    if (err.name !== 'AbortError') console.error('CSV export failed:', err);
  } finally {
    clearTimeout(timer);
  }
}).listen(3000);

A CSV export endpoint that uses the promise form of stream.pipeline with an AbortSignal, so a cursor error, a serialization error, a client disconnect or a timeout tears down every stream and releases the database connection.

How do you implement a streaming CSV export end to end?

1. Source

Read rows incrementally, never as one array. Use a database cursor or your driver's streaming query API. If neither is available, write an async generator that fetches keyset-paginated batches and pass it to Readable.from(). The generator only fetches the next batch when the consumer pulls, so backpressure reaches the database.

Avoid OFFSET pagination. The database still reads and discards every skipped row, so each page costs more than the last. Concurrent inserts or deletes also shift rows between pages, which causes duplicates or gaps. Instead, filter with WHERE (created_at, id) > (last_created_at, last_id) on an indexed sort key. The unique id acts as a tiebreaker, which makes the order stable.

2. Transform

Use an object-mode Transform or an async generator stage that turns each row into one CSV line. The escaping rules are:

• Quote any field that contains a comma, a double quote, CR or LF.
• Double any embedded quotes.

Spreadsheets treat fields starting with =, +, - or @ as formulas. Prefix those fields with a single quote to neutralize them.

A well-tested CSV library handles edge cases that hand-rolled code misses. Still check its defaults: when it quotes, which line terminator it uses, and whether it guards against formula injection at all.

3. HTTP

Set Content-Type: text/csv and Content-Disposition: attachment; filename="..." before the first write, because headers are sent with the first chunk. Omit Content-Length so Node uses chunked transfer encoding. Make res the final pipeline destination, so its backpressure and close events govern the whole chain.

4. Optional gzip

Insert zlib.createGzip() before res and set Content-Encoding: gzip. Backpressure still propagates, but zlib adds its own internal buffering. It also costs CPU on every chunk, which shares the event loop with other requests.

Copyconst http = require('node:http');
const zlib = require('node:zlib');
const { Readable } = require('node:stream');
const { pipeline } = require('node:stream/promises');
const { Pool } = require('pg');
const pool = new Pool(); // connection settings come from PG* env vars

// Keyset pagination: the next batch is fetched only when the consumer pulls.
async function* rows(batchSize = 1000) {
  let lastTs = '-infinity', lastId = 0;
  for (;;) {
    const { rows } = await pool.query(
      `SELECT id, email, note, created_at, created_at::text AS ts_key FROM orders
       WHERE (created_at, id) > ($1::timestamptz, $2)
       ORDER BY created_at, id LIMIT $3`, [lastTs, lastId, batchSize]);
    if (rows.length === 0) return;
    yield* rows;
    // Text key keeps full timestamp precision (a JS Date would truncate to ms).
    ({ ts_key: lastTs, id: lastId } = rows[rows.length - 1]);
  }
}

function cell(v) {
  let s = v == null ? '' : v instanceof Date ? v.toISOString() : String(v);
  if (/^[=+\-@]/.test(s)) s = "'" + s; // neutralize spreadsheet formulas
  return /[",\r\n]/.test(s) ? '"' + s.replace(/"/g, '""') + '"' : s;
}

async function* toCsv(source) {
  yield 'id,email,note,created_at\r\n';
  for await (const r of source)
    yield [r.id, r.email, r.note, r.created_at].map(cell).join(',') + '\r\n';
}

http.createServer(async (req, res) => {
  const gzip = /\bgzip\b/.test(req.headers['accept-encoding'] || '');
  // Headers before the first write; no Content-Length, so Node uses chunked encoding.
  res.writeHead(200, {
    'Content-Type': 'text/csv; charset=utf-8',
    'Content-Disposition': 'attachment; filename="orders.csv"',
    ...(gzip && { 'Content-Encoding': 'gzip', Vary: 'Accept-Encoding' }),
  });
  try {
    // res is the final destination, so its backpressure governs the whole chain.
    await pipeline(Readable.from(rows()), toCsv, ...(gzip ? [zlib.createGzip()] : []), res);
  } catch (err) { res.destroy(err); } // client abort or DB error: tear down cleanly
}).listen(3000);

An end-to-end streaming CSV export in Node.js: a keyset-paginated async generator source, an escaping and formula-guarding CSV stage, optional gzip, and the HTTP response as the final pipeline destination.

How do you handle client disconnects, errors and cancellation?

When the client disconnects, res closes before it finishes. pipeline treats that as a premature close and destroys every stream in the chain. The source still has to release whatever it holds. If the source is an async generator wrapped in Readable.from(), destruction calls the generator's return(), so a try/finally around the cursor loop runs and can close the cursor and return the connection to the pool. If the source is a custom Readable, do the same cleanup in _destroy(). Without this, every abandoned download keeps a pooled connection until the database times it out.

The promise-based pipeline from node:stream/promises accepts an options object with a signal. Tie an AbortController to req's close event or to a server-side deadline, and one abort tears down the whole chain.

The committed-status problem

Once the first chunk is flushed, the status code and headers are on the wire. If the query fails at row 400,000, you cannot send a 500. Destroying the response cuts the connection. With chunked encoding, the client can detect the missing terminator, but many download tools still save the partial file as if it were complete. Ways to reduce the risk:

• Trailing marker: write a final line that consumers check for, such as #END rows=N, or a matching error line on failure.
• Row-count check: send the expected count up front, for example from a COUNT query, so the consumer can compare it with what arrived.
• Async job model: generate the file first, then serve it. The next section, on when to avoid streaming directly to the client, covers this.

Also set a statement or cursor timeout on the database side. A stalled export should fail within a bounded time instead of holding a snapshot open indefinitely.

When should you avoid streaming CSV exports directly to the client?

Direct streaming is the right default for moderate exports, but each of its costs grows with export size.

Database pressure. A long-running cursor holds a pooled connection for the whole export, and often an open transaction or snapshot. On MVCC databases that snapshot can block vacuuming of dead rows or delay replication cleanup. Use a dedicated, small pool for exports so they cannot starve request traffic, and set cursor and statement timeouts.

Keyset batches (WHERE id > $last ORDER BY id LIMIT n) release the connection between pages. Without a single snapshot, rows inserted, updated or deleted mid-export can be missed or duplicated.

Client coupling. Backpressure ties export duration to the slowest reader. Proxy and load balancer idle timeouts can then cut off a slow download partway through.

CPU. Object-mode streams pay a per-row cost. Batching rows into larger string chunks uses slightly more memory per chunk and gives higher throughput.

Intermediaries. Gzip hides progress from the client. Proxies that buffer or re-compress responses undo the benefit of compressing at the source. A framework or proxy that buffers whole bodies removes the memory benefit entirely, and nothing fails visibly when it does.

How can you verify that memory stays flat during an export?

A repeatable memory test

Opinions about streaming are cheap. A test that runs against the real deployment path costs little and catches the problems that matter.

1. Seed a table large enough to hurt. Pick a row count where buffering the full result would clearly exceed your container's memory limit. If a buffered export would only use a few hundred megabytes, the test cannot tell streaming from buffering.

2. Sample memory during the export. Log process.memoryUsage().rss and heapUsed on a fixed interval, such as once per second, while the download runs. Streaming produces a sawtooth pattern within a narrow band. Buffering produces a line that keeps climbing with rows sent. Watch rss as well as heapUsed, because Buffers allocated outside the V8 heap show up only in rss.

3. Throttle the client. Run curl --limit-rate with a low rate, such as a few hundred kilobytes per second. A fast local client drains data as quickly as the server produces it, which hides missing backpressure. With a slow reader, any layer that ignores backpressure fills memory.

4. Kill the client mid-download. Interrupt curl partway through. The database cursor should close, the query should stop, and your pool's idle connection count should return to its baseline. If the count stays down, a connection has leaked. Repeat the kill a dozen times to see whether the pool runs dry.

5. Run it through the real path. Send the test through the load balancer, reverse proxy and middleware stack that production uses, not just localhost. Compression middleware, response-logging hooks and proxy buffering settings can each collect the entire body without any error.

A failure here takes minutes to diagnose. The same failure in production, such as an out-of-memory restart during a customer's month-end export, costs an incident and a long investigation.

How can a streaming CSV export save engineering time?

Most of the time saved comes from work that no longer needs to happen.

Fewer incidents to investigate. Out-of-memory crashes caused by buffered exports are slow to diagnose. The process dies, the request that triggered it is often gone from the logs, and the restart affects unrelated requests that shared the process. Because a streaming export's memory does not depend on result size, this whole class of incident goes away instead of returning each time a customer's data grows.

No per-customer workarounds. Teams with buffered exports tend to collect workarounds over time: raising container memory limits, capping row counts, asking large customers to split exports by date range, or writing a separate code path for the biggest accounts. Each workaround needs maintenance and support. A streaming export serves the smallest and the largest account through the same code path.

Less hand-written error handling. stream.pipeline replaces per-stream 'error' listeners, manual destroy() calls and custom completion tracking with one awaitable call. Connection leaks from abandoned downloads are among the hardest bugs to find, and pipeline, combined with cleanup in the source's finally or _destroy(), prevents them by construction.

A reusable pattern. The source, transform and response stages are independent. After you build one export this way, you can turn the keyset-paginated source and the escaping CSV stage into a shared helper. New export endpoints then need only a query and a row-to-columns mapping, and they inherit backpressure, formula-injection guarding and cancellation handling.

Regressions caught before production. The memory test in the previous section can be scripted and run against a staging environment. When a middleware change or proxy setting starts buffering whole response bodies, the test shows climbing memory or a leaked connection during review instead of during a customer's export.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to node.js stream csv export, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.