Streams — reading and writing data without loading it all into memory

Reading a whole file into memory before doing anything with it works fine until the file is bigger than the memory available — streams process data in small chunks as it arrives, which is the real, structural fix, not just a performance nicety.

Beginner

3 min read

The problem: `readFile` loads the ENTIRE file into memory at once

const fs = require("node:fs/promises");
 
const data = await fs.readFile("./huge-video.mp4"); // ALL of it, in memory, before anything else can happen
res.end(data);

readFile waits until the entire file is read into a single in-memory buffer before returning anything at all — completely fine for a small config file, but a genuine problem for a multi-gigabyte video file: the server needs enough free memory to hold the whole thing at once, and the client waits for the entire read to finish before receiving a single byte, even though streaming the response as it's read would let the download start immediately.

Streams: process data in small chunks, as they arrive

const fs = require("node:fs");
 
const readStream = fs.createReadStream("./huge-video.mp4"); // reads in small CHUNKS, not all at once
readStream.pipe(res); // each chunk is written to the response AS IT ARRIVES — no full-file buffer needed

A stream reads (or writes) data incrementally, in small chunks, rather than requiring the entire dataset to exist in memory at any single point in time — createReadStream reads a manageable chunk, emits it as a data event, and moves on to the next chunk, all while earlier chunks can already be flowing onward (to the response, to another file, wherever .pipe() sends them). This is the real, structural answer to the "can't fit it all in memory" problem, not a performance micro-optimization: a server streaming a large file uses roughly constant memory regardless of the file's actual size, since it never needs more than one chunk's worth in memory at a time.

`.pipe()`: connecting a readable stream directly to a writable one

const fs = require("node:fs");
const zlib = require("node:zlib");
 
fs.createReadStream("./large-log.txt")
  .pipe(zlib.createGzip())              // each chunk gets compressed as it flows through
  .pipe(fs.createWriteStream("./large-log.txt.gz")); // then written to the output file, chunk by chunk

.pipe() connects a readable stream's output directly to a writable stream's input, automatically handling the chunk-by-chunk flow between them — and streams can be chained through intermediate transform streams (like zlib.createGzip(), which compresses each chunk as it passes through) without ever needing the full file, compressed or uncompressed, to exist in memory simultaneously. This three-line example compresses an arbitrarily large log file using a small, constant amount of memory, entirely because every stage operates on chunks rather than the whole file at once.

Backpressure: `.pipe()` automatically slows a fast source down for a slow destination

// Without streams, manually writing chunks as fast as possible can overwhelm a slower
// destination (a slow network connection, a slow disk) — the writable side's internal
// buffer fills up faster than it can drain, consuming unbounded memory
 
readStream.pipe(writeStream); // .pipe() handles this AUTOMATICALLY — it's the whole point

If a readable stream produces data faster than a writable stream can consume it, something has to give — either unbounded memory grows in an internal buffer, or the fast side needs to be told to slow down. .pipe() handles this automatically: it's aware of the writable stream's internal buffer filling up and pauses the readable stream's flow until the writable side has caught up, called backpressure. This is a genuine, real mechanism (not just a .pipe() implementation detail worth ignoring) — handling backpressure manually, without .pipe() or an equivalent library, is one of the more subtle and easy-to-get-wrong parts of working with streams directly.

Streams — reading and writing data without loading it all into memory

The problem: readFile loads the ENTIRE file into memory at once

Streams: process data in small chunks, as they arrive

.pipe(): connecting a readable stream directly to a writable one

Backpressure: .pipe() automatically slows a fast source down for a slow destination

Further reading

The problem: `readFile` loads the ENTIRE file into memory at once

`.pipe()`: connecting a readable stream directly to a writable one

Backpressure: `.pipe()` automatically slows a fast source down for a slow destination