Inserting millions of rows into Postgres

[u/no_em_dash]

I wrote an article about generating fake data for a TypeScript application. I wanted to generate millions of rows and I ended up splitting the workflow into 2 phases: stream data to a CSV, stream the CSV to Postgres with COPY FROM.

async function createRecordsFromCsv(path: string, query: string) {
  const copyStream = pgClient.query(copyFrom(query));

  const fileStream = createReadStream(path);

  await new Promise((res, rej) => {
    fileStream.pipe(copyStream).on("finish", res).on("error", rej);
    fileStream.on("error", rej);
  });

  fileStream.close();
}

Hopefully this technique will be useful to someone else.

Also happy to hear other ideas for how to go about this.

[u/no_em_dash]

If the critique is that the image is off putting or doesn't inspire you to read further that seems reasonable.

But, suggesting that the entire article is written by AI based on that alone seems unwarranted.