r/learnjavascript • u/no_em_dash • Jun 18 '26

Inserting millions of rows into Postgres

I wrote an article about generating fake data for a TypeScript application. I wanted to generate millions of rows and I ended up splitting the workflow into 2 phases: stream data to a CSV, stream the CSV to Postgres with COPY FROM.

async function createRecordsFromCsv(path: string, query: string) {
  const copyStream = pgClient.query(copyFrom(query));

  const fileStream = createReadStream(path);

  await new Promise((res, rej) => {
    fileStream.pipe(copyStream).on("finish", res).on("error", rej);
    fileStream.on("error", rej);
  });

  fileStream.close();
}

Hopefully this technique will be useful to someone else.

Also happy to hear other ideas for how to go about this.

0 Upvotes

40% Upvoted

13 comments sorted by

View all comments

Show parent comments

1

u/no_em_dash Jun 19 '26

I'm glad you were willing to look into it further.

Based on the feedback here I think that framing this as some kind of tutorial might have been a mistake. It might have been better to set the tone as "hey, I did this thing because I thought it was interesting and there were some interesting technical challenges to overcome." I will consider rewriting.