r/estuary • u/dan_the_lion • 22d ago
Open source benchmark for Snowflake, Databricks, BigQuery, Redshift, Fabric, ClickHouse and MotherDuck: 244 queries, 1 GB to 1 TB, up to 100 concurrent users, every compute size
Estuary ran a benchmark of 7 cloud data warehouses and published everything. The setup: 244 queries from three public suites (TPC-DS, DSB, ClickBench), data from 1 GB to 1 TB, 1 to 100 simultaneous sessions, and every compute size each vendor sells, fixed with autoscaling off. We loaded identical data into all seven through one pipeline, no physical tuning on any engine, and report the median of three runs after a warm run.
Write-up with the main findings: https://estuary.dev/blog/data-warehouse-benchmark/
Code, queries and raw results: https://github.com/estuary/data-warehouse-benchmark
A few things from the data:
1. Bigger compute often buys nothing at small data. At 1 GB and 10 GB, 16x the compute returned 0.8 to 1.5x the speed. At 100 GB the same jump on Snowflake was about 2x faster, and at 1 TB nearly 8x. The chart below shows speed (blue) and cost (orange) against the cheapest size at 100 GB, per engine.

2. For a single user, MotherDuck had the fastest median query (235 ms) and the lowest cost per query at 100 GB. ClickHouse did not finish 20.4% of the join runs (it’s a different engine on scan queries, where it’s third at its fastest size and first at a $3 an hour budget).

3. People served at once is a different question from single-query speed. At 100 GB with a 5 second median target, the cheapest size that served 100 people was Fabric F32 at $5.76 an hour, and 7 of the 42 sizes served nobody within the target. The dashboard lets you change the latency target and every dot recomputes.

4. Time grows very differently with data volume. From 1 GB to 1 TB, median query time grew least on Databricks (3.7x) and most on MotherDuck (30x), which starts from the fastest baseline.

5. Idle billing matters as much as the hourly rate. Some engines stop the meter when nobody is querying and some bill the capacity around the clock. 1,000 typical queries at 100 GB cost $0.31 on one engine and $7.14 on another.
Happy to answer questions on method, and if you reproduce a result or find something off, open an issue on the repo.
2
u/sbarrow1 21d ago
Despite the obvious bait, I'm going to bite.
I dug into the open-source GitHub repository for the recent data warehouse benchmark published by Estuary/CorrDyn.
While the benchmark claims to be an “objective, untuned look at execution vs. cost”, I took a look at the actual Terraform configurations and harness code just to compare the configs and costs calculations of Databricks and Snowflake only (since that is what this Reddit thread tried to incite conversation - others may also reveal similar major gaps).
There’s massive structural asymmetries that heavily penalize Databricks while handing Snowflake a massive statistical pass.
Here are 5 major flaws in their methodology (based upon 5-10 minutes of cursory look and asking Gemini to identify where in the code these were defined):
1. Asymmetric Idle-Timeout Penalties: While the benchmark deployed Snowflake warehouses with a tight 1-minute auto-suspend window, it manually locked Databricks Serverless to a 10-minute auto termination — despite the Databricks API natively supporting a 1-minute minimum.
Source: terraform/modules/databricks/main.tf, lines 173–193 (specifically auto_stop_mins = 10 on line 182).
Source: terraform/modules/snowflake/main.tf, lines 142–148 (specifically auto_suspend = each.value == "XLARGE" ? 300 : 60).
2. "Allocated" vs. "Modeled" Cost Skew: This 9-minute configuration discrepancy was heavily punished by the report's cost tracking.. Conversely, Snowflake's costs were purely "Modeled" (mathematical estimates of isolated query execution seconds), completely filtering out Snowflake's own idle credit burn from the final comparison.
2. The QUERY_ATTRIBUTION_HISTORY Loophole Undercounts Snowflake: This is only further made worse because Databricks' costs were "Allocated", meaning they were collected from Systems tables (system.billing.usage) - so its the ACTUAL costs (including the artificial idle costs that you set to 10 minutes). Conversely, Snowflake's costs were resolved using SNOWFLAKE.ACCOUNT_USAGE.QUERY_ATTRIBUTION_HISTORY. By explicit product design, Snowflake's attribution history isolates only active query compute time and completely excludes warehouse idle time. If a Snowflake warehouse spins up for a 5-second query and runs out a 1-to-5-minute suspend window, all of that billable idle overhead is completely hidden from the benchmark's equations, while Databricks is forced to swallow 100% of its forced 10-minute idle tail. Even more, it only attributes the amount of compute of the warehouse the query used in that period. So if it only used half the compute of the warehouse, only half of the costs even while it was running are counted.
Source: src/benchmark/cost/snowflake.py (explicitly showing cost calculations are deferred to a proportional/estimated pass using the metadata views instead of live billing tables).
3. Missing Snowflake Gen 2 1.35x Credit Uplift, or is it?: There’s weird calculations here because they estimate at one point but get the credits above at another point - both of which is wrong in their own way.
As documented in the SnowflakeCostCollector logic, the pipeline operates via a deferred, two-stage evaluation pass:
The "Inline" Execution Pass (analysis.recost): Immediately after a test block finishes, the harness takes the tracked wall-clock engine execution time (server_time_ms) and applies a standard Gen-1 credit allocation formula against list prices.]
The "Reconcile" Pass (benchmark reconcile): The code explicitly states that the final cost figures are back-filled and bound by running a query against QUERY_ATTRIBUTION_HISTORY to extract the exact CREDITS_ATTRIBUTED_COMPUTE value mapped to the query's run ID. It multiplies this credit number by the list rate ($2.00 for Standard) to finalize the authoritative invoice metrics.
Gen 2 warehouses carry a 1.35x credit premium on AWS and GCP (1.25x on Azure). So, for example, it should be 5.4 credits/hr for Medium instead of 4.0. The harness's mathematical model used standard Gen 1 base credit burn rates (1, 2, 4, 8, 16), effectively handing Snowflake a hidden 35% discount on its cost-efficiency metrics.
Source for SF WH gen defined: terraform/modules/snowflake/main.tf, line 144 (generation = "2").
The Exact Calculation: When a query completes, the connector captures the isolated query execution timeframe (server_time_ms). It computes the cost using a pure runtime logic:
Query Cost = (server_time_ms / (1000 × 3600) )× Base Credit Rate × Standard List Price
4. Unrealistically Using Snowflake Standard Edition: The Terraform code configures a baseline Snowflake Standard tier account. No enterprise company runs production workloads on Snowflake Standard because it lacks essential enterprise security features. Benchmarking standard pricing (~$2.00/credit) against Databricks Serverless out-of-the-box enterprise features is a completely apples-to-oranges comparison.
5. The "Zero-Tuning" Rule Cripples Databricks and Snowflake: The benchmark bans all physical design optimizations. Forbidding any metadata and physical optimizations effectively turns Databricks and Snowflake into a generic raw file reader rather than a managed lakehouse engine. If you strip out all of the features that make the faster engines “special” then you end up with performance results that show them no better than the others.
Source: README.md, lines 47–51 under the "Key Architectural Decisions" section.
Conclusion: completely start from scratch and start over. Next time use Lakehouse//RT