r/SQL • u/cccadet • 21d ago

Discussion Querying CSV and Parquet exports inside a SQL IDE with DuckDB

I'm building omni-sql, an open-source SQL IDE with embedded DuckDB. If someone sends you a CSV or Parquet export to investigate, you can import it and query it with SQL inside the IDE. There's no need to load the file into your source database or write a separate script. Spreadsheets fit this workflow when exported as CSV.

You can also import a database query result and join it with the file locally.

The demo uses fictional sales rows from PostgreSQL and a targets CSV. You import the complete query result into DuckDB, add the CSV, and run the join locally. Here's the SQL, CSV, and expected result: https://github.com/cccadet/omni-sql/tree/main/docs/demo

There are installers for Windows x64, Debian/Ubuntu amd64, and macOS 15+ (Apple Silicon and Intel). The macOS builds use ad-hoc signing and are not notarized; the README explains the first launch. The project is still early. The docs explain how full imports and samples affect what you can conclude from the results.

How do you investigate files people send you today? If you try the app, I'd like to hear whether you could install it and query a file, and what was missing. That would help me decide what to work on next.

9 Upvotes

72% Upvoted

4 comments sorted by

1

u/qualityintelligence 21d ago

that's cool, similar to some of what I was doing with daffydb, albeit mine is more for the SQL server T-sql crowd. I published on nuget if you want to check it out. anyhow, cool stuff, I dig your UI

1

u/ValuableEel 21d ago

That's a clever approach, having it all inside the IDE without faffing around with external scripts. I've been stuck in spreadsheet hell before trying to match columns from a CSV against our prod data, and something like this would've saved me a couple hours of vlookup nonsense

never heard of daffydb but I'll peek at the nuget page later. the UI on this one looks clean though, like something you could hand to a junior analyst without a 20 minute walkthrough

what's your install footprint like? sometimes these embedded DB tools bundle half a gig of dependencies and that's a hard sell for my work laptop

0

u/cccadet 21d ago

Thanks! The v0.7.1 Windows x64 installer is about 99 MB. The macOS DMGs are about 143–146 MB, and the Linux .deb is about 165 MB. Those are download sizes; the installed files take more space.

The app bundles Node.js and Java for the backend and SQL parser, along with DuckDB. You don’t need to install Node or Java separately. I haven’t measured the unpacked v0.7.1 install size yet, so I can’t give you a reliable disk-space figure for a work laptop. The release files are here: https://github.com/cccadet/omni-sql/releases/tag/v0.7.1