Buffers & Streams

Computers store and send everything as bytes: numbers from 0 to 255. Text, photos, PDFs and network packets are all bytes underneath. JavaScript strings are fine for text, but Node.js also needs a way to handle raw bytes. That is the job of the Buffer.

Files and network data are also often too big to load in one go. A 3 GB video cannot fit comfortably in memory, but you can process it a small piece at a time. That is what streams do. This lesson covers both, because streams carry their data as buffers.

Buffers: raw bytes

A Buffer is a fixed-size block of bytes. It is a global class, so you do not need to import it. Turn text into bytes and back like this:

const text = "Chai ₹20";
const buf = Buffer.from(text, "utf8");

console.log(buf);
console.log(`Characters: ${text.length}, bytes: ${buf.length}`);
console.log(buf.toString("utf8"));
console.log(buf.toString("hex"));
console.log(buf.toString("base64"));
console.log(Buffer.from("Q2hhaSDigrkyMA==", "base64").toString());
console.log(buf[0], String.fromCharCode(buf[0]));
console.log(buf instanceof Uint8Array);

<Buffer 43 68 61 69 20 e2 82 b9 32 30>
Characters: 8, bytes: 10
Chai ₹20
4368616920e282b93230
Q2hhaSDigrkyMA==
Chai ₹20
67 C
true

Each pair in <Buffer 43 68 ...> is one byte written in hexadecimal (base 16). 43 is the letter C, which is 67 in normal numbers. The string has 8 characters but the buffer has 10 bytes, because UTF-8, the standard text encoding, stores the rupee sign ₹ in three bytes (e2 82 b9). English letters take one byte each, while Hindi and other Indian scripts take three bytes per character.

An encoding is the rule that turns characters into bytes. You will mostly use "utf8" for text, "base64" for sending binary data inside text (for example in JSON or email), and "hex" for showing bytes and hashes.

const empty = Buffer.alloc(4);           // 4 bytes, all zero
console.log(empty);

const part1 = Buffer.from("Namaste ");
const part2 = Buffer.from("नमस्ते");
const whole = Buffer.concat([part1, part2]);
console.log(whole.toString());
console.log(`"नमस्ते" is ${"नमस्ते".length} characters but ${part2.length} bytes`);
console.log(whole.subarray(0, 7).toString());

<Buffer 00 00 00 00>
Namaste नमस्ते
"नमस्ते" is 6 characters but 18 bytes
Namaste

Buffer.concat() joins buffers, and subarray() gives a view of part of one. When you limit the size of uploaded text or a request body, count bytes with Buffer.byteLength(str), not characters with str.length.

Why streams?

Suppose you must count the orders in a 3.4 MB CSV file. readFile() loads it all into memory, which is fine at that size. But for a 4 GB export it would need 4 GB of memory, and a server handling ten such downloads at once would crash.

A stream delivers data in small pieces called chunks, one after another. You process each chunk and let it go, so memory use stays small no matter how big the file is. There are four kinds:

Reading a stream

A readable stream is an event emitter. It emits "data" for each chunk, "end" when there is no more, and "error" if something fails:

import { createReadStream } from "node:fs";

const stream = createReadStream("orders.csv", { encoding: "utf8" });
let chunks = 0;
let characters = 0;

stream.on("data", (chunk) => {
  chunks++;
  characters += chunk.length;
});
stream.on("end", () => console.log(`Read ${characters} characters in ${chunks} chunks`));
stream.on("error", (err) => console.error(`Read failed: ${err.message}`));

Read 3518916 characters in 54 chunks

File streams read 64 KiB at a time by default, so the 3.4 MB file arrived in 54 chunks. Readable streams also work with for await, which is usually clearer:

import { createReadStream } from "node:fs";

let bytes = 0;
for await (const chunk of createReadStream("orders.csv")) {
  bytes += chunk.length; // chunk is a Buffer
}
console.log(`File size: ${(bytes / 1024 / 1024).toFixed(1)} MB`);

File size: 3.4 MB

Connecting streams with pipeline()

The real power of streams is connecting them: read from a file, compress, write to another file, all chunk by chunk. Use pipeline() from node:stream/promises. It connects the streams, passes errors from any stage to you, and cleans up every stream if one fails:

import { createReadStream, createWriteStream } from "node:fs";
import { stat } from "node:fs/promises";
import { pipeline } from "node:stream/promises";
import { createGzip } from "node:zlib";

await pipeline(
  createReadStream("orders.csv"),
  createGzip(),
  createWriteStream("orders.csv.gz"),
);

const before = (await stat("orders.csv")).size;
const after = (await stat("orders.csv.gz")).size;
console.log(`Compressed ${before} bytes to ${after} bytes`);

Compressed 3518916 bytes to 957562 bytes

pipeline() also handles backpressure. If the writing side is slower than the reading side (a slow disk or a slow mobile connection), the reader pauses until the writer catches up, so chunks do not pile up in memory. If any stage fails, for example because the input file is missing, the await throws an error you can catch with try...catch and check with err.code.

Transform streams

A Transform stream changes data as it flows through. Here is one that turns text to uppercase, connected between the keyboard input and the screen:

import { Transform } from "node:stream";
import { pipeline } from "node:stream/promises";

const toUpper = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());
  },
});

await pipeline(process.stdin, toUpper, process.stdout);

echo "jai hind from rohtak" | node upper.js

JAI HIND FROM ROHTAK

callback(null, data) passes the changed chunk on; callback(err) reports an error.

Advanced: filtering lines with an async generator

pipeline() also accepts an async generator function as a middle stage. This is a modern, readable way to write a transform. A chunk can end in the middle of a line, so keep the unfinished piece for the next chunk:

import { createReadStream, createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";

let kept = 0;

await pipeline(
  createReadStream("orders.csv", { encoding: "utf8" }),
  async function* keepKochi(source) {
    let leftover = "";
    for await (const chunk of source) {
      const lines = (leftover + chunk).split("\n");
      leftover = lines.pop(); // last piece may be half a line
      for (const line of lines) {
        if (line.includes(",Kochi,")) {
          kept++;
          yield line + "\n";
        }
      }
    }
  },
  createWriteStream("kochi.csv"),
);

console.log(`Saved ${kept} Kochi orders to kochi.csv`);

Saved 40000 Kochi orders to kochi.csv

The program never holds more than one chunk and one partial line in memory, so it works the same way on a 4 GB file.

Common mistakes

• Counting str.length as bytes. Non-English text uses more bytes than characters. Use Buffer.byteLength().
• Using new Buffer(). It is deprecated; use Buffer.from() or Buffer.alloc().
• No "error" handling on streams. A missing file then crashes the program. pipeline() with try...catch handles every stage.
• Using .pipe() chains in new code. Errors are not forwarded and streams may stay open. Use pipeline().
• Assuming a chunk is one line or one record. Chunks are cut by size, not by meaning; keep leftovers between chunks.

Key points

• A Buffer holds raw bytes; create one with Buffer.from() or Buffer.alloc() and convert with toString("utf8" | "hex" | "base64").
• UTF-8 uses 1 to 4 bytes per character; ₹ and Hindi letters take 3 bytes each.
• Streams process data chunk by chunk, so memory use stays low for any file size.
• Readable, Writable, Duplex and Transform are the four kinds of stream; HTTP requests and responses are streams too.
• Connect streams with pipeline() from node:stream/promises; it handles errors, cleanup and backpressure.

// count-cities.js
import { createReadStream } from "node:fs";

const counts = {};
let leftover = "";
let header = true;

for await (const chunk of createReadStream("orders.csv", { encoding: "utf8" })) {
  const lines = (leftover + chunk).split("\n");
  leftover = lines.pop();
  for (const line of lines) {
    if (header) { header = false; continue; }
    const city = line.split(",")[1];
    counts[city] = (counts[city] ?? 0) + 1;
  }
}

console.log(counts);

{
  Jaipur: 40000,
  Pune: 40000,
  Kochi: 40000,
  Lucknow: 40000,
  Rohtak: 40000
}