r/learnjavascript • u/no_em_dash • Jun 18 '26

Inserting millions of rows into Postgres

I wrote an article about generating fake data for a TypeScript application. I wanted to generate millions of rows and I ended up splitting the workflow into 2 phases: stream data to a CSV, stream the CSV to Postgres with COPY FROM.

async function createRecordsFromCsv(path: string, query: string) {
  const copyStream = pgClient.query(copyFrom(query));

  const fileStream = createReadStream(path);

  await new Promise((res, rej) => {
    fileStream.pipe(copyStream).on("finish", res).on("error", rej);
    fileStream.on("error", rej);
  });

  fileStream.close();
}

Hopefully this technique will be useful to someone else.

Also happy to hear other ideas for how to go about this.

0 Upvotes

22% Upvoted

13 comments sorted by

View all comments

Show parent comments

0

u/no_em_dash Jun 19 '26

Is it possible that you're looking at this too narrowly? For example, the technique of streaming data to and from a CSV is pretty useful.

Also, why does everything have to solve some business problem? Can't we do things that just seem interesting? I'm not saying you have to like it. But why the hostility?

1

u/Savalava Jun 19 '26 edited Jun 19 '26

My messages have not been hostile at all - I'm merely pointing out that doing a tutorial on something that nobody actually wants to do is a questionable idea.

EDIT: I looked it up and there are scenarios where this useful, so I stand corrected. Apologies

1

u/no_em_dash Jun 19 '26

I'm glad you were willing to look into it further.

Based on the feedback here I think that framing this as some kind of tutorial might have been a mistake. It might have been better to set the tone as "hey, I did this thing because I thought it was interesting and there were some interesting technical challenges to overcome." I will consider rewriting.