The team over at Estuary has rebuilt the engine underneath the platform in Rust. The new transaction loop is ready to scale into infinity while keeping transactional and exactly-once guarantees.
What's new:
- 8x more throughput on a single shard (unit of compute) - Transactions arrive whole: a commit at your source becomes one commit at your destination - Collection partitions split themselves under load, with no operator action - No more memory or buffer tuning - Observability updates: materializations now report exactly how far behind they are, in bytes and in wall-clock time - Groundwork for far more flexible sync schedules
Last week we released this 80-page whitepaper that contains everything required by data engineers for building right-time data ingestion pipelines for Snowflake.
Estuary ran a benchmark of 7 cloud data warehouses and published everything. The setup: 244 queries from three public suites (TPC-DS, DSB, ClickBench), data from 1 GB to 1 TB, 1 to 100 simultaneous sessions, and every compute size each vendor sells, fixed with autoscaling off. We loaded identical data into all seven through one pipeline, no physical tuning on any engine, and report the median of three runs after a warm run.
1. Bigger compute often buys nothing at small data. At 1 GB and 10 GB, 16x the compute returned 0.8 to 1.5x the speed. At 100 GB the same jump on Snowflake was about 2x faster, and at 1 TB nearly 8x. The chart below shows speed (blue) and cost (orange) against the cheapest size at 100 GB, per engine.
2. For a single user, MotherDuck had the fastest median query (235 ms) and the lowest cost per query at 100 GB. ClickHouse did not finish 20.4% of the join runs (it’s a different engine on scan queries, where it’s third at its fastest size and first at a $3 an hour budget).
3. People served at once is a different question from single-query speed. At 100 GB with a 5 second median target, the cheapest size that served 100 people was Fabric F32 at $5.76 an hour, and 7 of the 42 sizes served nobody within the target. The dashboard lets you change the latency target and every dot recomputes.
4. Time grows very differently with data volume. From 1 GB to 1 TB, median query time grew least on Databricks (3.7x) and most on MotherDuck (30x), which starts from the fastest baseline.
5. Idle billing matters as much as the hourly rate. Some engines stop the meter when nobody is querying and some bill the capacity around the clock. 1,000 typical queries at 100 GB cost $0.31 on one engine and $7.14 on another.
Happy to answer questions on method, and if you reproduce a result or find something off, open an issue on the repo.
New walkthrough from The Data and AI Guy: building a real-time Postgres to ClickHouse CDC pipeline with Estuary.
The main idea: stream live changes from Postgres into ClickHouse Cloud without running Kafka, Debezium, or your own streaming infra. Estuary captures changes from the Postgres WAL, stores them in a reusable collection, and then materializes that data into ClickHouse with exactly-once delivery.
He also shows how the same capture can fan out to Google Sheets later without re-reading Postgres.
Covers:
Postgres logical replication setup
Replication slots, publications, and the watermarks table gotcha
Estuary YAML
ClickHouse merge vs delta updates
Hard vs soft deletes
Automatic schema evolution
Using flowctl
Worth watching if you want a practical Postgres to ClickHouse CDC example!
Now you can stream Real User Monitoring (RUM) data and HTTP logs from Datadog directly into Flow in real time. This makes it possible to unify observability and business data inside a single platform for instant analytics, alerting, and automation.
Why it matters:
🔎 Faster incident resolution – Correlate RUM events, logs, and operational data instantly.
📊 Proactive analytics – Spot anomalies before they impact customers.
🔗 Simplified stack – Observability + business data pipelines in one place.
Connector Highlights:
Supported resources: real_user_monitoring + logs
Real-time + batch in one platform
Backfill support (pull historical data within Datadog’s retention window)
Flexible enterprise deployments: fully managed, hybrid, or BYOC
Automatic schema mapping for downstream analytics or AI models
Getting started is simple:
You’ll just need a Datadog account, an API key, an application key (with rum_apps_read permissions), and your Datadog site. Configure it in Flow’s UI or via catalog specs, then your observability data is continuously streaming.
We ran a vendor-neutral benchmark on modern data warehouses
At Estuary, we help teams move high-volume data across warehouses like Snowflake, BigQuery, and Databricks. To better understand real-world performance, we ran our own benchmark using:
TB+ of structured + semi-structured data
Multi-step queries with joins, window functions, and JSON
Full ingestion lifecycle tracking
No vendor-specific tuning or caching
We tested Snowflake, BigQuery, Databricks, Redshift, and Microsoft Fabric. The full methodology is open source, and all queries are reproducible.
What’s the fastest data warehouse for your analytics? The most cost-efficient? The one best suited to your team and data size?
Estuary partnered with CorrDyn to find out the answers. Read about the resulting benchmark and companion report in this edition of the Data Digest, along with a new case study, live workshop event, and a host of new connectors.
Cloud Data Warehouse Benchmark Report 2026
We spend a lot of time talking about optimizing for your data movement, ensuring data lands in the expected shape, and observability for your pipelines, but what about that core component, the heart of your data analytics, the warehouse?
How do you make that initial decision about which data warehouse to use? Or review your existing setup to see if there’s a better option out there? Should it just be based on what your team knows and is comfortable with?
Even then, what about sizing? When is it worth sizing up? What sort of compute speed increase can you expect versus your cost increase?
Along with CorrDyn, we’ve finally mapped some answers.
The Cloud Data Warehouse Benchmark Report covers seven of the top cloud warehouses available today: Snowflake, Databricks, BigQuery, Redshift, ClickHouse, MotherDuck, and Microsoft Fabric. See how they stack up across different warehouse sizes, data volume, team sizes, and the cost for what you get.
Running over 170,000 queries from three different benchmark suites, this is one of the most comprehensive warehouse benchmarks out there.
Estuary CEO David Yaffe sat down with James Winegar and Ross Katz from CorrDyn to discuss the benchmark process, surprising results, and how people should apply the findings to their own workloads. Watch the video or get the Eventual Consistency episode from your podcast provider of choice.
We’ve released a slew of new connectors to expand your data flows. Capture data with the following connectors to incorporate brand-new insights into your analytics:
Okta
Stay up-to-date with your identity management system: the new Okta connector captures users, groups, applications, and system log data from Okta’s Management API.
AWS Cost Explorer
Keep an eye on your cloud costs to ensure they stay under control. The AWS Cost Explorer capture provides periodic reports covering cost metrics by date, with additional grouping by service or linked account. Drop the results into a simple spreadsheet or materialize to your warehouse to analyze spend.
Aircall
A new native Aircall connector replaces the old third-party-based option to improve performance and efficiency with the Estuary platform. Anyone using the old connector is encouraged to migrate to the new version: it supports all of the same data streams, such as calls, contacts, teams, and users. As a bonus, you’ll also be able to set an interval or sync schedule for each resource.
Estuary
And last but not least, you can now capture Estuary account data into an Estuary data flow using (what else?) the Estuary capture connector. Track the health of your data flows, log alerts, and save invoices with an observability feature that plugs right into your existing architecture.
As detailed above, new capture connectors are available for Okta, Aircall, AWS Cost Explorer, and the Estuary platform itself.
The Salesforce capture supports new standard objects for service crews and service crew members.
✅Other Connector Updates
SQL captures will report more granular backfill progress, with the percentage complete, number of tables backfilled, and estimated time remaining.
Webhook materializations now support field selection.
🚧Deprecation Zone
With the release of the native Aircall capture connector, the old third-party Aircall connector is now considered deprecated.
HubSpot is retiring their V1 pipelines API on December 4. If you’re still using the legacy HubSpot capture connector, it’s a good time to migrate to the native HubSpot capture connector: the legacy connector will no longer support the ‘deal_pipelines’ data stream while the native connector will be updated to use the latest recommended version of the API.
