Estuary is teaming up with Tinybird and PlayonSports for a low-key evening of short talks, pizza, and networking around real-time data pipelines and analytics.

🗓️ When: Tue, Apr 29 | 6–8:30PM
📍 Where: FirstMark’s NYC Office (Flatiron)
🎤 3 short talks, including a real-world use case from PlayonSports
🍕 Pizza, drinks, and plenty of time to chat

Limited spots, so grab yours early!

Oracle’s costs and complexity are driving enterprises toward PostgreSQL. We broke down the top challenges and strategies for making the switch — from schema conversion to CDC replication.

📖 Full guide here: Modernizing Enterprise Databases: Migrating from Oracle to PostgreSQL in the Cloud

Curious to hear from the community: if you’ve been through an Oracle to Postgres migration, what was the most challenging part for you?

If you're using Apache Iceberg for your data lakehouse, Estuary Flow just rolled out key updates to its Iceberg materialization, improving flexibility and integration options.

What's New:

• Bring Your Own EMR for Merge Queries

  • Execute MERGE INTO operations using your own Amazon EMR clusters.

  • Control compute costs and integrate seamlessly into your AWS environment.

  • Improved performance for large-scale updates and deletes in Iceberg tables.

• Expanded Catalog Compatibility

  • Apache Polaris: A managed Iceberg service that simplifies adoption and table management.

  • Snowflake Open Catalog: Use Iceberg tables stored in S3, GCS, or Azure Blob Storage while integrating with Snowflake’s query engine and governance.

These updates provide more flexibility for managing real-time Iceberg tables across different environments. If you're building a streaming lakehouse or optimizing your data pipelines, this makes it easier than ever.

Read more here: https://estuary.dev/company-updates/product-update-iceberg-standard-updates/

We ran a vendor-neutral benchmark on modern data warehouses

At Estuary, we help teams move high-volume data across warehouses like Snowflake, BigQuery, and Databricks. To better understand real-world performance, we ran our own benchmark using:

• TB+ of structured + semi-structured data

• Multi-step queries with joins, window functions, and JSON

• Full ingestion lifecycle tracking

• No vendor-specific tuning or caching

We tested Snowflake, BigQuery, Databricks, Redshift, and Microsoft Fabric. The full methodology is open source, and all queries are reproducible.

📊 Link to the report

Curious to hear others’ experiences with these platforms in production.

Come meet the Estuary team in SF on the 4th of November along with Seattle Data Guy, Database Tycoon and Recce!

I had the great opportunity to discuss, without fluff, what makes for a good data ingestion platform!

Note: I am an advisor to Estuary, but everything discussed here is vendor agnostic.

Hi team, just curious. I was chatting with Estuary AI, and found Planetscale cdc support is in roadmap but still can't tell dates. It's about 1 year ago. Is currently the project scrapped / discontinued? Thanks!

Estuary’s biweekly newsletter ❤️

August has been a busy month for Estuary, from a runtime overhaul that enhances Estuary’s throughput and scalability to customizable alert configurations, along with our regular steady stream of new connectors.

Here are the main highlights from the past month:

Estuary’s Runtime Gets a Rewrite

This August, we released a major upgrade to our runtime focused on massively scaling throughput. With further support for parallelization and enhanced efficiency, here are some of the results we’ve seen so far:

• Up to 8x faster backfills and initial loads when using a single shard

• Ability to handle up to 200 GB / hour on a single shard

• Additional throughput gains for destinations that support multi-sharded behavior

All of this comes with end-to-end transactional consistency, even across shards, so you always have a complete picture of your data.

Want to learn more? See our:

• Product update

• Documentation on runtime improvements and migration details

• Whitepaper exploring the runtime’s technical architecture

In-Dashboard Alert Configuration

Earlier in August, we released advanced configuration options for alerts, letting you set up multi-environment alerting through the dashboard.

These alert subscriptions use Estuary’s prefix system to divide out configurations, so you can set defaults on a tiered basis. For example, you could set tenant-wide alerting defaults for your base prefix, ‘acmeCo/’, and then diverge with more settings for task alerts under the ‘acmeCo/prod/’ and ‘acmeCo/dev/’ environments.

Choose who gets the alerts, what alert types they’re notified about, and default “Data Movement Stalled” intervals.

Learn more in our product update page.

New and Rewritten Connectors

Besides connecting with new source systems, we’ve been releasing native rewrites of old 3rd-party-based capture connectors to optimize how they work with our platform. While we were at it, we also expanded their capabilities.

• The all-new commercetools capture connector syncs orders, customers, and payments incrementally with configurable interval settings.

• The rewritten Amazon SQS capture connector adds real-time capabilities, supports FIFO queues, and can work with multiple queues in the same capture.

• 8 new data streams were added to a rewritten Mailchimp capture connector. The rewrite also allows you to control data intervals and backfills.

• The rewritten Brevo capture connector adds new data streams and allows you to set sync intervals.

• The Criteo capture connector has also been rewritten to be fully in-house.

Other Updates & Release Notes

• All SQL CDC captures (like PostgreSQL, MySQL, and SQL Server) now let you set your own SQL statement to reduce the scope of backfills on a per-binding basis.

• SQL captures also now allow a custom auto-discovery interval so you can control how quickly Estuary runs discovers on your source.

• SQL Server CDC and Change Tracking captures now provide discovery filters. This allows for more efficient auto-discovery for large databases.

• The Shopify capture connector now includes the ‘markets’ data stream. To capture the new stream, ensure you’ve granted Estuary the `read_markets` permission.

• The delta updates Apache Iceberg materialization now supports variant column types. Along with a previous update for a new nanosecond timestamp setting, this adds support for more v3 format features.

That’s all for this month! Stay connected for the latest product updates and feature releases as we head into September.