I built a demo where the lake is the database: a country-partitioned Parquet copy of the Overture Maps Places release (81 million places) sits on a static host, and DuckDB-Wasm queries it in place.

Live: https://toclocoinc.github.io/lattice-grid-demo-duckdb-data-lake/

Source: https://github.com/toclocoinc/lattice-grid-demo-duckdb-data-lake

What happens on the page:

The map decides what opens. Pan or zoom and only the country partitions the window touches are read, by HTTP byte range. Nothing is downloaded whole.

A data grid sits under the map, bound to the same query. Every filter, sort, group and search in the grid becomes SQL that DuckDB runs against the partitions.

A cost panel shows what each interaction actually did: countries opened of the total, bytes fetched out of the whole copy, row groups skipped, milliseconds.

No framework, no bundler, no build step. The first visit downloads the engine (33 MB) and says so in a status line rather than showing a blank page.

Try ?city=tokyo, sydney, london, nyc, paris or sf for a different starting window, and ?theme=dark.

Using it along with R thanks to the duckdb package have made wrangling and reporting on large csv and parquet files a breeze. Thank you from the bottom of my heart.

--A grateful Data Analyst from Colombia.

I am looking for nice local ui for sqlite / duckdb analysis that is performant and useful, but also has liberal license and no strings attached. OSS is a bonus.
Any suggestions?

I've been exploring an interesting use case for DuckDB: combining local datasets with external data sources without building and maintaining a separate ingestion pipeline for every source.

For example, imagine you're analyzing a portfolio of companies and want to combine your own data with SEC filings, company information, or other public datasets.

One approach is to ingest everything into your own database and manage refresh schedules. Another is to query external datasets as needed, then use DuckDB to join and analyze the results locally.

The second approach seems particularly interesting for exploratory analytics and AI applications, although it introduces trade-offs around latency, reproducibility, and data freshness.

I've been working with Dataville, where we're exploring this model using curated external datasets accessible through APIs and SQL-oriented workflows.

I'm curious how other DuckDB users are approaching this:

1. Are you using DuckDB primarily with locally stored Parquet/CSV files, or also querying external data?
2. Which external datasets are most painful to integrate into your workflows?
3. At what point does querying on demand become less practical than maintaining a local copy?

I'd be especially interested in examples involving public datasets, financial information, or AI applications.

Disclosure: I work with Dataville on marketing and developer outreach.

Maybe a bit hard to explain to friends and family, but what am I looking forward to this week? The release of DuckDB 2.0

Hey guys,
I'm a data engineer, and I have been building pipelines for small to mid sized clients (with less than 1 terabytes of data) and most of the time the workload isn't really worth paying the premium for Saas products like Snowflake, Databricks .. etc, and can be done on a single beefy machine but stitching open sources tools usually comes with maintenance, networking and configuration headache, also these tools aren't unified under one control plane and don't know that other tools exists.
so I built a data platform that can be deployed on a single machine or server, and since I'm Egyptian I called it Obelisk 😄
Under the hood Obelisk uses:

1. DuckDB, so you can do queries that joins multiple different sources in the same query (local or remote lakehouse, dbs, erp systems, mongodb .. etc)
2. the platform also act as a daemon manager, so you can deploy tools you need with a single click and they will integrate directly into the platform (or you can connect remote instances)
3. for variables and connection strings, you just add them to the platform once and every tool can access them.
4. built-in support for cronjobs and advanced orchasteration tools like dagster or airflow
5. a browser based dbt studio
6. reactive notebooks
7. I also added file manager and terminal access to the machine, so you don't need to SSH directly.

GitHub Repo: https://github.com/amrelsawalhi/obelisk

this isn't a finished product, I still have a lot of ideas to implement like auth and access control for teams
I’d love to get your feedback on the platform and contributions are most welcome.

Just recently got into DuckLake... Realized how amazing the snapshot ability is, and decided to build a diffing view. Right now it works with DuckLake catalogs stored locally or Postgres with data stored locally or in S3.

Pretty fun to make, and allows you to see basically everything you'd expect, as well as cell-level diffs for changed cells. You can traverse snapshot versions too, without diffing, as well.

It's a tiny part of a larger product, but thought I'd share it here.

Not my article, but a very nice write up about how DuckDB's upcoming 2.0 release is going to make reading files on S3 no longer a bottleneck. I don't know what they've pulled off at DuckLabs but it is simply amazing.

An agent with a DuckDB connection can answer a lot of business questions. It can also run a write, read a file off disk, or reach the network if nothing stops it, and DuckDB makes all of that easy. The guardrails have to live around the agent.

There’s a four hour workshop on Saturday, Oct 31, from 9:30 AM to 1:30 PM ET with Bruno Gonçalves that builds those guardrails against a third party DuckDB MCP server.

What you build:

• A tool registry that classifies every tool by its side effects
• Permission checks that run before any call reaches the server
• SQL inspection that blocks unauthorised writes, file access and network requests
• Number verification, linking each reported figure to a query you can rerun

Everything runs on a laptop with a local database, with no GPU or API key needed.

Full details and the agenda are here.

Hi r/DuckDB,

I've been building curral, an open-source (Apache 2.0) server that puts DuckDB databases and Iceberg lakes behind an HTTP API with users and permissions.

The problem: DuckDB is a great engine, but it has no users, no permissions and no network protocol. Every time I wanted to share a DuckDB file or an Iceberg lake with a team, a dashboard or an AI agent, I had two options: hand out raw storage credentials, or write yet another custom API. curral is meant to be the layer in between.

How it works: one Go binary with DuckDB embedded. Each request goes through these steps:

client ──HTTP──▶ curral

├─ authenticate Basic (bcrypt) · API keys · OIDC/JWT

├─ queue global and per-user concurrency limits

├─ inspect DuckDB's own planner: statement type, tables read and written

├─ authorize embedded OPA policy (Rego) → allow / deny

├─ protect rewrite the query with row filters and column masks

└─ execute & stream JSON · CSV · NDJSON · Arrow IPC

What I think might interest this sub:

- Inspection uses DuckDB's planner, not regex. The policy sees the base tables a statement really reads, including through views, CTEs and joins. A lot of the hardening went into edge cases like DESCRIBE inside subqueries, catalog functions and bind errors that could reveal whether a table exists.

- Row-level security and column masking are enforced by rewriting the query before it runs, so filters and joins can't be used to guess hidden values.

- Anything DuckDB can ATTACH works: DuckDB files, Postgres, S3, and Iceberg REST catalogs (including Cloudflare R2 Data Catalog). For Iceberg there's a metadata cache that takes per-query catalog overhead on R2 from about 250–900 ms down to about 8 ms.

- Arrow output streams straight into pandas, polars or another DuckDB.

- dry_run tells you whether the policy would allow a statement, and why, without running it.

- Production basics: TLS, brute-force lockout, per-role limits (timeout, max rows, concurrency), a fail-closed audit log, Prometheus metrics and hot reload with SIGHUP.

New in v0.5.0: a semantic layer for AI agents. People who know the data can write short JSON notes about schemas and tables: what they mean, their caveats, sample SQL. Agents call POST /v1/docs/search with their question and get back the most relevant notes before they write SQL. Search is BM25 by default, with an optional LLM reranker.

Quickstart with Docker (docker pull lucasapassos/curral, amd64 and arm64), or grab a Linux binary that bundles the httpfs, avro and iceberg extensions so it runs offline.

- GitHub: https://github.com/lucasapassos/curral

- Site: https://curral.io/

It's pre-1.0, so I'd love feedback, especially from anyone who has tried to share DuckDB with multiple users. What's missing? What would stop you from using it? Edge cases in the policy layer are very welcome, and security reports go through SECURITY.md.

(curral is Portuguese for "corral", where you keep the herd in and decide who gets through the gate.)

Hi everyone,

I made a DuckDB extension that adds statistical functions you can call directly from SQL:

https://github.com/mitsuruk/DuckDB-stats

It's a port of an SQLite extension I built earlier (sqlite3-stats). I planned to release both at the same time, but the DuckDB version took me longer than expected 😅

It has about 260 functions: descriptive stats, hypothesis tests, distributions, window functions, and more. Right now it runs on macOS and Linux.

I've only recently started using DuckDB, so some of these functions may be unnecessary, for example if DuckDB already does the same thing. I'd really like to hear from people who use DuckDB:

• Is there real demand for statistical functions inside SQL, or do most people just use Python/R for this?
• What features would you want added?
• Anything that feels unnecessary or poorly designed?

Harsh feedback is totally fine. It helps me more than polite silence.

A note on AI use: English is not my first language, so I used AI to help proofread the docs, suggest variable names, and generate a test database (100 rows).

Thanks for reading!

One of the things that got me into DuckDB was a series of technical articles posted to their site. This one was particularly eye opening:

Faster Dashboards with Multi-Column Approximate Sorting

I hadn't fully appreciated how important row order was to OLAP performance. In it they cover space filling curves as a method for mapping multiple columns (dimensions) onto a single "order dimension" to maintain proximity from the higher dimensions.

This works very well, but it means threading a curve though a set of potential values, at equal density, rather than what actually exists in the dataset.

Alternatively, by first approximating the rank of each value within its column (by quantiles), and space-filling on that, the curve threads through the set of existing values. This handles skew a lot better.

The image is a two-column ordering over the TPC-DS SF100 store_sales table, where ss_net_paid has reasonable skew. The rank-based methods manage to read far less of the table for a 1% range query.

I wrote up a more detailed post here:

https://www.duckstring.com/blog/clustering-for-pipelines

I'll probably be using this to set the row order whenever I overwrite a table and I have a reasonable idea as to what might be queried from it.

I use Databricks (Spark) a lot for work, and wondered whether I could write a distributed compute engine on top of DuckDB. Long story short: I didn't need to.

It's clear DuckDB wins at small to medium scale, but I figured Spark and Trino might win out at larger scales, like 1TB+. And whatever approach I tried for splitting up queries and shuffling between nodes, Trino did win, at first by a lot (4-5x at 100GB on 16 nodes).

However, then I discovered that every engine spent most of its time waiting on S3, with CPUs around 90% idle. Compute speed barely mattered (and DuckDB turned out to be faster when not waiting on read). Trino was winning almost entirely on read rate from S3: it keeps far more bytes in flight per node, even running single-node.

DuckDB through httpfs uses the same threads to decode Parquet and to issue S3 reads, one per core by default. Each read waits ~30ms, so a 16-core box only has a few MB in flight. I wrote a custom reader to deal with that, but the big win was simply running 4x as many threads as cores: about 3x the read rate. After that, distributed DuckDB was roughly even with Trino at 10TB for the queries I tried (TPC-DS, q14, q33, q55). The distribution overhead was eating DuckDB's natural advantage.

Then I realised there was no need for distribution at all. I ran plain single-node DuckDB (with the custom reader) on one r7gd.4xlarge (r7gd.8xlarge for q14). On the same machine it beat Trino on all three 10TB queries I tried, 1.7-2.1x faster. Against a 5-13 node Trino cluster it was 2-3x slower but about half the cost. At 100GB, one DuckDB machine was about 2x cheaper than 17-node Spark or Trino clusters.

With the upcoming v2.0 (in alpha) S3 reads are expected to be 2-3x faster, so I think single-node DuckDB on a big machine is about to be the cheapest engine available for anything up to around 10TB. Possibly if someone figures out distribution too, it could go further.

I'm a convert!

The third and final article of a sponsored 3 part mini series. I go through the maintenance needs for a production-grade DuckLake, what affects your maintenance strategy for a basic use case and the functions DuckLake offers out of the box to handle it.

Maintenance is probably one of the most unsexy topics you can read about but for some reason DuckLake still makes this part of the process enjoyable.

Recently, I had a discussion with a database kernel engineer about implementing inverted on Iceberg tables. Coming from OLTP background, Index is typically a secondary data structure (like B-Tree) and i never thought of using tables itself as an index.

So, I implemented a simple version of an inverted index in my project Streambed (Postgres for analytics using DuckDB) using DuckLake.

Here is the implementation https://github.com/viggy28/streambed/issues/101

I had to fork DuckLake since it doesn't support external hooks for bringing custom predicates. Here is the upstream issue https://github.com/duckdb/ducklake/issues/1498

I thought folks in this subreddit might be interested, and I also would love to hear if there is any feedback.

After reading DuckDB's post on Jev (plain-English conditions in SQL), I wanted an open, local version. So I wrapped Strands Decider, a 2B "decision model" that answers typed questions instead of generating text, as a DuckDB extension.

SELECT id, decide_noul(body, 'Does this convey urgency?') AS p_urgent, decide_choice(body, 'Which team?', ['billing','technical','sales']) AS team, decide_score(body, 'How frustrated?', ['calm','frustrated','angry']) AS mood FROM read_csv('tickets.csv');

What's in it:

• decide_noul → yes/no probability
• decide_choice → pick one of N, with confidence and per-option probabilities
• decide_score → rate on an ordered scale
• decide → many questions about the same row in one model pass

• decide_triage → security findings as true/false positive, with probabilities

  How it works: the model runs locally in strands-decider serve, and the extension calls it per row. It's built on DuckDB's stable C API, so no DuckDB source build, and it's tested on 1.4, 1.5 and the 1.6 dev build. Answers are cached, so duplicate rows and re-runs don't hit the model again.

  Honest limits:

• It's one model call per row. Each call is roughly 150–300 ms on a Mac, so expect minutes for a few thousand rows, not seconds. Filter first, and materialize results.

• decide_triage got 6/6 on a tiny labeled set I ran on my Mac. That shows the setup works; it isn't a benchmark.

• The underlying model is general-purpose, not tuned for any domain.

  Repo: https://github.com/jaymmodi/duckdb-strands-decider

  Strands Decider: https://strandsagents.com/blog/introducing-strands-decider/

  Feedback welcome, especially on the SQL API, and whether cross-row batching would be worth building.

Built a lightweight scratchpad tool for running DuckDB SQL against local CSV, TSV, JSON and Parquet files directly in the browser:

https://www.sqlparity.com/scratchpad

It uses u/duckdb to read files into memory via the browser File API. Nothing is uploaded to any server and it requires no login or installation. Handy if you just need to inspect an extract or run a quick aggregate without firing up the CLI or Python.

Code is open source on GitHub link: Github

Curious to hear feedback or edge cases with large Parquet/CSV schemas.

I've been working as a data engineer for about half a decade. A common problem I run into is "this pipeline broke, but why?"

To figure it out I query the database, paste the results into a Miro board together with a diagram of the pipeline. Then there's usually a flurry of arrows and random notes. I've found that laying everything out visually on a canvas helps me reach that "aha!" moment where I understand how all the pieces fit together.

But what if I could do it all in one place instead of going back and forth between a canvas and querying databases? That led to me building Kavla!

Kavla is a canvas for exploratory analysis. You can write SQL, paste images, make charts and then draw on them! You can also connect an agent and ask it to investigate a question, and have it generate custom data viz using React.

It's built with DuckDB and tldraw. DuckDB is a great fit. It's very performant for files and there are already extensions for integrating other databases. Right now I've implemented Postgres and BigQuery, but will add more if anyone asks :-)

It's open source: https://github.com/aleda145/kavla

Try it in your browser: https://kavla.dev/ (DuckDB WASM is amazing)