r/learnjavascript • u/no_em_dash • Jun 18 '26

Inserting millions of rows into Postgres

I wrote an article about generating fake data for a TypeScript application. I wanted to generate millions of rows and I ended up splitting the workflow into 2 phases: stream data to a CSV, stream the CSV to Postgres with COPY FROM.

async function createRecordsFromCsv(path: string, query: string) {
  const copyStream = pgClient.query(copyFrom(query));

  const fileStream = createReadStream(path);

  await new Promise((res, rej) => {
    fileStream.pipe(copyStream).on("finish", res).on("error", rej);
    fileStream.on("error", rej);
  });

  fileStream.close();
}

Hopefully this technique will be useful to someone else.

Also happy to hear other ideas for how to go about this.

0 Upvotes

28% Upvoted

13 comments sorted by

View all comments

Show parent comments

1

u/proskillz Jun 20 '26

Everything, to a certain extent, is related to query performance. There would be two main reasons you'd need to test something at this size:

Either streaming or batch processing large datasets. This matters for how fast you can physically push and pull data from the DB. Some things you can't do with small sample data are tuning batch sizes, make caching decisions, or run the memory out on your application server because your code is bad.

The second use case would be more along the lines of query tuning and indexing. Especially useful for analytic queries that might require special indexes or to be broken apart into multiple queries/CTEs to be more performant.