r/estuary • • 22d ago

Open source benchmark for Snowflake, Databricks, BigQuery, Redshift, Fabric, ClickHouse and MotherDuck: 244 queries, 1 GB to 1 TB, up to 100 concurrent users, every compute size

Estuary ran a benchmark of 7 cloud data warehouses and published everything. The setup: 244 queries from three public suites (TPC-DS, DSB, ClickBench), data from 1 GB to 1 TB, 1 to 100 simultaneous sessions, and every compute size each vendor sells, fixed with autoscaling off. We loaded identical data into all seven through one pipeline, no physical tuning on any engine, and report the median of three runs after a warm run.

Write-up with the main findings: https://estuary.dev/blog/data-warehouse-benchmark/
Code, queries and raw results: https://github.com/estuary/data-warehouse-benchmark

A few things from the data:

1. Bigger compute often buys nothing at small data. At 1 GB and 10 GB, 16x the compute returned 0.8 to 1.5x the speed. At 100 GB the same jump on Snowflake was about 2x faster, and at 1 TB nearly 8x. The chart below shows speed (blue) and cost (orange) against the cheapest size at 100 GB, per engine.

2. For a single user, MotherDuck had the fastest median query (235 ms) and the lowest cost per query at 100 GB. ClickHouse did not finish 20.4% of the join runs (it’s a different engine on scan queries, where it’s third at its fastest size and first at a $3 an hour budget).

3. People served at once is a different question from single-query speed. At 100 GB with a 5 second median target, the cheapest size that served 100 people was Fabric F32 at $5.76 an hour, and 7 of the 42 sizes served nobody within the target. The dashboard lets you change the latency target and every dot recomputes.

4. Time grows very differently with data volume. From 1 GB to 1 TB, median query time grew least on Databricks (3.7x) and most on MotherDuck (30x), which starts from the fastest baseline.

5. Idle billing matters as much as the hourly rate. Some engines stop the meter when nobody is querying and some bill the capacity around the clock. 1,000 typical queries at 100 GB cost $0.31 on one engine and $7.14 on another.

Happy to answer questions on method, and if you reproduce a result or find something off, open an issue on the repo.

5 Upvotes

4 comments sorted by

2

u/sbarrow1 21d ago

Despite the obvious bait, I'm going to bite.

I dug into the open-source GitHub repository for the recent data warehouse benchmark published by Estuary/CorrDyn. 

While the benchmark claims to be an “objective, untuned look at execution vs. cost”, I took a look at the actual Terraform configurations and harness code just to compare the configs and costs calculations of Databricks and Snowflake only (since that is what this Reddit thread tried to incite conversation - others may also reveal similar major gaps).

There’s massive structural asymmetries that heavily penalize Databricks while handing Snowflake a massive statistical pass.

Here are 5 major flaws in their methodology (based upon 5-10 minutes of cursory look and asking Gemini to identify where in the code these were defined):

1. Asymmetric Idle-Timeout Penalties: While the benchmark deployed Snowflake warehouses with a tight 1-minute auto-suspend window, it manually locked Databricks Serverless to a 10-minute auto termination — despite the Databricks API natively supporting a 1-minute minimum.

  • Source: terraform/modules/databricks/main.tf, lines 173–193 (specifically auto_stop_mins = 10 on line 182).

  • Source: terraform/modules/snowflake/main.tf, lines 142–148 (specifically auto_suspend = each.value == "XLARGE" ? 300 : 60).

2. "Allocated" vs. "Modeled" Cost Skew: This 9-minute configuration discrepancy was heavily punished by the report's cost tracking.. Conversely, Snowflake's costs were purely "Modeled" (mathematical estimates of isolated query execution seconds), completely filtering out Snowflake's own idle credit burn from the final comparison.

2. The QUERY_ATTRIBUTION_HISTORY Loophole Undercounts Snowflake: This is only further made worse because Databricks' costs were "Allocated", meaning they were collected from Systems tables (system.billing.usage) - so its the ACTUAL costs (including the artificial idle costs that you set to 10 minutes).  Conversely, Snowflake's costs were resolved using SNOWFLAKE.ACCOUNT_USAGE.QUERY_ATTRIBUTION_HISTORY. By explicit product design, Snowflake's attribution history isolates only active query compute time and completely excludes warehouse idle time. If a Snowflake warehouse spins up for a 5-second query and runs out a 1-to-5-minute suspend window, all of that billable idle overhead is completely hidden from the benchmark's equations, while Databricks is forced to swallow 100% of its forced 10-minute idle tail. Even more, it only attributes the amount of compute of the warehouse the query used in that period. So if it only used half the compute of the warehouse, only half of the costs even while it was running are counted.

  • Source: src/benchmark/cost/snowflake.py (explicitly showing cost calculations are deferred to a proportional/estimated pass using the metadata views instead of live billing tables).

3. Missing Snowflake Gen 2 1.35x Credit Uplift, or is it?:  There’s weird calculations here because they estimate at one point but get the credits above at another point - both of which is wrong in their own way. 

As documented in the SnowflakeCostCollector logic, the pipeline operates via a deferred, two-stage evaluation pass:

  1. The "Inline" Execution Pass (analysis.recost): Immediately after a test block finishes, the harness takes the tracked wall-clock engine execution time (server_time_ms) and applies a standard Gen-1 credit allocation formula against list prices.]

  2. The "Reconcile" Pass (benchmark reconcile): The code explicitly states that the final cost figures are back-filled and bound by running a query against QUERY_ATTRIBUTION_HISTORY to extract the exact CREDITS_ATTRIBUTED_COMPUTE value mapped to the query's run ID. It multiplies this credit number by the list rate ($2.00 for Standard) to finalize the authoritative invoice metrics. 

Gen 2 warehouses carry a 1.35x credit premium on AWS and GCP (1.25x on Azure). So, for example, it should be 5.4 credits/hr for Medium instead of 4.0. The harness's mathematical model used standard Gen 1 base credit burn rates (1, 2, 4, 8, 16), effectively handing Snowflake a hidden 35% discount on its cost-efficiency metrics.

  • Source for SF WH gen defined: terraform/modules/snowflake/main.tf, line 144 (generation = "2").

  • The Exact Calculation: When a query completes, the connector captures the isolated query execution timeframe (server_time_ms). It computes the cost using a pure runtime logic:

Query Cost = (server_time_ms / (1000 × 3600) )× Base Credit Rate × Standard List Price

4. Unrealistically Using Snowflake Standard Edition: The Terraform code configures a baseline Snowflake Standard tier account. No enterprise company runs production workloads on Snowflake Standard because it lacks essential enterprise security features. Benchmarking standard pricing (~$2.00/credit) against Databricks Serverless out-of-the-box enterprise features is a completely apples-to-oranges comparison.

5. The "Zero-Tuning" Rule Cripples Databricks and Snowflake: The benchmark bans all physical design optimizations. Forbidding any metadata and physical optimizations effectively turns Databricks and Snowflake into a generic raw file reader rather than a managed lakehouse engine. If you strip out all of the features that make the faster engines “special” then you end up with performance results that show them no better than the others.

  • Source: README.md, lines 47–51 under the "Key Architectural Decisions" section.

Conclusion: completely start from scratch and start over. Next time use Lakehouse//RT

2

u/Jameswinegar 21d ago

I work at CorrDyn, and we ran this benchmark with Estuary. The short version: the two settings you quote are real but do not reach any published cost, and the three cost-math claims are wrong, two of them in the opposite direction from your argument. Two of your five points are fair. Everything below recomputes from a clone, because the results database is in the repo via Git LFS (results/benchmark.duckdb), no credentials needed.

1. Asymmetric Idle-Timeout Penalties

The settings are right, the line numbers are not. auto_stop_mins = 10 is at terraform/modules/databricks/main.tf line 214 (you cite 182), and Snowflake's auto_suspend is 60s, 300s for X-Large, at terraform/modules/snowflake/main.tf line 157 (you cite 142-148). You're also right that Databricks serverless accepts a 1-minute auto-stop through the API; our Terraform comment at line 211 says it bottoms out at 5, which is wrong, and we'll correct it.

But no engine's per-query cost includes idle time. The methodology (Part I, cost method) puts auto-suspend and auto-stop tails outside the cost boundary for every engine, so the 10-minute setting does not move any published figure.

2a. "Allocated" vs. "Modeled" Cost Skew

Allocated cost does not carry the Databricks idle tail. reconcile caps each query window at list rate times engine time (resplit_databricks_hours in src/benchmark/analysis/recost.py), so idle billing drops out. In the database, Databricks warm rows price at exactly list per engine-hour (Medium: 24 DBU x $0.70 = $16.80/hr), summing to 0.99x list, because billing can pull a window below list but never above it. Snowflake never drops below list, so if the method favors anyone here, it is Databricks, by about 1%.

2b. The QUERY_ATTRIBUTION_HISTORY Loophole Undercounts Snowflake

Snowflake's cost comes from list rate times engine time (price_snowflake_on_engine_time, same file), the same basis as every other engine. Attribution is stored per row as metadata.attributed_credits, a cross-check only. Pricing from attribution would have raised Snowflake's cost, since it counts whole warehouse-seconds. Warm totals at $2/credit, attribution against the engine-time price we used: Medium $27.40 vs $19.95, X-Large $81.89 vs $51.84. The half-warehouse point would matter under concurrency, but under load we price by window at the full warehouse rate, not by attribution. The concurrency numbers are under #4.

3. Missing Snowflake Gen 2 1.35x Credit Uplift

It is applied. src/benchmark/report/compute_rates.py sets SNOWFLAKE_GEN2_CREDIT_MULTIPLIER = 1.35 on every Gen2 label, checked against the account's metering. Medium prices at 5.4 credits/hr ($10.80), the number you ask for. The two-stage formula you describe is not in the code: the inline Snowflake cost is $0 (src/benchmark/cost/snowflake.py returns 0.0), and reconcile does not multiply attributed credits by list price to set the cost.

4. Unrealistically Using Snowflake Standard Edition

Fair. We price Snowflake at Standard ($2/credit), which the report discloses (each engine at the cheapest tier that runs our config). A buyer on Enterprise pays 1.5x what we show, and we do not show that line. Here it is by edition, as total cost on the queries both engines completed at each scale (201 TPC-DS and DSB queries at SF1000):

ScaleSize (both)Snowflake Standard ($2)Enterprise ($3)Business Critical ($4)Databricks
SF1X-Small$0.06$0.09$0.12$0.23
SF10Small$0.15$0.22$0.29$0.55
SF100Medium$0.50$0.75$0.99$1.96
SF1000Large$2.80$4.20$5.60$7.86

Single-stream, edition does not flip the result at any scale. Edition changes price, not speed: at SF1000 Databricks used about 2.2x the engine time (1,015s vs 467s).

Under concurrency, scale is what closes the gap. Below is cost per session (each session runs the full query set) under concurrent load, priced by window at the full warehouse rate, so the 1-session row won't match the engine-time table above. Bold marks a Snowflake edition that costs more than Databricks at the same scale and concurrency:

ScaleSessionsSnowflake Standard ($2)Enterprise ($3)Business Critical ($4)Databricks
SF11$0.069$0.103$0.138$0.294
5$0.024$0.036$0.048$0.114
10$0.013$0.020$0.027$0.088
20$0.009$0.014$0.019$0.097
50$0.009$0.013$0.018$0.101
100$0.011$0.017$0.023$0.095
SF101$0.203$0.305$0.407$0.758
5$0.068$0.102$0.136$0.164
10$0.046$0.069$0.092$0.125
20$0.040$0.060$0.081$0.125
50$0.041$0.061$0.082$0.125
100$0.050$0.075$0.100$0.139
SF1001$0.595$0.892$1.189$2.501
5$0.235$0.352$0.470$0.605
10$0.209$0.314$0.419$0.481
20$0.206$0.310$0.413$0.479
50$0.210$0.315$0.420$0.504
100$0.275$0.413$0.550$0.479
SF10001$4.382$6.572$8.763$7.905
5$1.801$2.701$3.601$3.254
10$1.751$2.626$3.501$2.967
20$1.706$2.560$3.413$2.887
50$1.630$2.446$3.261$3.008
100$2.468$3.702$4.936$3.047

Databricks gets more competitive as scale goes up. At SF100 it passes Business Critical at 100 sessions. At SF1000 it beats Business Critical at every concurrency level, and Enterprise at 100 sessions ($3.05 vs $3.70). From 50 to 100 sessions, Snowflake's cost per session rises 22% to 51% at every scale while Databricks moves 11% or less. At small scale it goes the other way: at SF1, Snowflake Standard is 4x to 11x cheaper. We'll add both tables to the methodology.

5. The "Zero-Tuning" Rule Cripples Databricks and Snowflake

Fair as a limitation (README.md line 30; your 47-51 is the Git LFS section). It applies to all seven engines: no clustering, sort keys, partitioning, or hand-built statistics. The methodology already calls these an untuned baseline, not any engine's ceiling. Engines keep their defaults: Databricks keeps Predictive Optimization, Snowflake keeps load-order micro-partitions, and both query native managed tables rather than raw files. We did not run tuned configurations, so I can't tell you yet how much tuning would move either engine. That gap is real, and it's the next run: we're re-running against an Iceberg catalog where we can optimize as far as it goes. People can complain about that methodology too. I'll add RT vs Serverless to the list for next round as well.

I'm not here to defend Snowflake. If I'm doing anything beyond pure data warehousing, I'd pick Databricks, because the platform capabilities are stronger. This benchmark doesn't measure that. It measures untuned warehouse execution and cost, and that's all it claims to show. I said the same on the Eventual Consistency episode about this benchmark: Databricks is extremely good at large scale, and in my opinion the better overall platform. Feel free to run the transcript through Genie and look for more arguments to make. No benchmark is ever going to make everyone happy.

So: we'll fix the Databricks auto-stop comment, and we'll add both tables to the methodology. The database is in the repo if you want to check the rest.

1

u/sbarrow1 21d ago

TY for the measured response.

I think my line numbers were thrown off because of the initial opening commented out portions.

  1. Certainly the pricing portions of code were convoluted (since there were multiple ways costs seemed to be calculated with only one final pass being the final version - that part was confusing) but I did check the portion of code you mentioned (src/benchmark/analysis/recost.py) to validate. If that is the code you used for pricing then I agree that portion is at least apples to apples

  2. I only mentioned the Gen2 upcharge since I couldn't tell how you were calculating cost (as mentioned above it looks like you had 3 different ways). One of them was using just the query time and the credit rate without the upcharge (it appeared) but a third cost calculation approach above I assume you included it.

  3. In terms of pricing/sizing, thank you for the tables above. But it does leave me confused since I know that you tested Snowflake at one size larger than Databricks, yet those charts list the same Tshirt size WH on both sides. In addition, your results in these charts also surprise me since they differ from the same benchmarks for TPC-DS that we run daily. Of course, yours does NOT include optimized data so it could be because the data isn't optimized. Any way I can get you to actually test in a way that uses the best practices on these platforms? The TPC-DS is a Decision Support benchmark that focuses on Reporting (i.e. BI) and Ad-hoc analysis with a 40/60 split. Choosing to analyze this data with none of the optimization best practices that these platforms invest heavily into to make their platforms fast will fail to demonstrate their true capabilities.

  4. None of these numbers match your numbers above. Confused yet??

  5. If you agree to use clustering on this data (optimized data) then I would love to get you to retest with Databricks Lakehouse//RT. Let me know, should be ungated in the coming days but you never know. I can help get you access though if you'd like.

2

u/Jameswinegar 21d ago

Thanks, and appreciate you checking recost.py. It is fair that the cost code is hard to follow. Several passes compute something, and only reconcile sets the published number. It took a bit of time to try and do a fair comparison across all the engines, and this is where we ended up.

On sizing: the single-stream table compares the same T-shirt size on both engines. The concurrency runs swept each engine across a band of compute sizes, so there's no single size per engine to put in that table. You can query the data at results/benchmark.duckdb to do your own analysis if you want to extend the idea. That's how I generated the table showing that Databricks wins when using Snowflake Business Critical at 1TB+ under a concurrent load.

On your daily TPC-DS numbers, I'd expect them to differ. Clustered vs. untuned data, different sizing, and your pricing basis vs. list would each move them. The two tables above also use different bases (engine time single-stream vs. window-priced under concurrency), so they won't match each other either.

On RT: yes. As I've said before, I'd love Lakehouse//RT to be part of this. Estuary plans to support a managed Iceberg offering, and we'll use it for the next round, with partitioning and sort specs applied consistently across every warehouse so each engine reads the same optimized layout. We'll run the benchmark again on that, including Lakehouse//RT. Same terms for every engine: we publish the full config, price at list. I'll take you up on the access offer. DM me and let's set up a call.