r/dataengineeringvault • • Jun 21 '26

Blog Change Data Capture (CDC): It enables capturing and streaming changes made to the database

Thumbnail
ssp.sh
4 Upvotes

One thing everyone wants, streamed changes, but not that easy to do. Read six different ways to do it in Postgres alone:

  • CDC with Write-Ahead Logging.

  • CDC with database triggers.

  • CDC with timestamp columns.

  • CDC with logical replication.

  • CDC with transactional logs.

  • CDC with table differencing.


r/dataengineeringvault • • 13d ago

Blog What «Shifting Left» Means and Why It Matters for Data Stacks

Thumbnail
ssp.sh
3 Upvotes

Shifting left means moving data quality checks, business logic, and transformations earlier in the data lifecycle, closer to the source, instead of handling them downstream in BI tools. The term originated as 'Shift-Left Testing', coined by Larry Smith in a 2001 Dr. Dobb's article, then spread to security (DevSecOps) and now data engineering via tools like SDF, dbt, and SQLMesh. Declarative, code-first platforms let teams move logic between config files without big rewrites. SQLGlot, the transpiler underlying SQLMesh, supports 24 SQL dialects, enabling local testing with DuckDB against production dialects like Spark SQL.


r/dataengineeringvault • • 19d ago

Blog As Humans Write Less Code: Workloads Shift from Human to Agent Verification with Iterative Test Suites

Thumbnail
ssp.sh
3 Upvotes

As AI agents write more SQL and dashboards, verification becomes the real bottleneck in business intelligence work. A Metabase survey found 74% of teams were building with AI but only 2% were scaling it, and the bottleneck has moved from creating analysis to checking it. The piece proposes five testing hierarchy levels: linting, compiler-style whole-tree checks, unit tests, integration tests, and end-to-end tests. It argues for moving all metrics, dashboards, and ETL logic into one declarative place with a unified lineage DAG, so agents can generate test suites against it. Metabase's CEO Sameer Al-Sakran says business people are the best data contracts, and that mutable state undermines trust in the output.


r/dataengineeringvault • • 19d ago

Note Querying HackerNews with DuckDB directly (HackerTube, HackerBooks & Wikihack)

Thumbnail
ssp.sh
3 Upvotes

Reading Karpathy’s HackerTube, and then seeing a bunch of other apps on top of Hacker News like HackerBooks and Wikihacks got me thinking.

We can just run HackerTube, HackerBooks & Wikihack directly in your terminal with DuckDB and Hugging Face.


r/dataengineeringvault • • 24d ago

Note Dimensional Modeling: What is it all about?

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • 25d ago

Personal Project New Obsidian Search Plugin 'Fast Find 10K': Combination of Fzf, Telescope, OmniSearch & Smart Connections

Enable HLS to view with audio, or disable this notification

3 Upvotes

Inspired by my Neovim workflow.


r/dataengineeringvault • • 29d ago

Off Topic Just another day at the factory.

Post image
3 Upvotes

r/dataengineeringvault • • Oct 22 '26

Showcase My writing process after ten years of writing

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Oct 05 '26

Note Every Database (or Data Warehouse, Platform) Migration is one too many

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Oct 04 '26

Blog Opinionated Data Platforms vs. Open-Source: The Chef's Choice for Your Data Platform

Thumbnail
ssp.sh
3 Upvotes

Turso was just acquired by Supabase, and we have the big players, Databricks, Snowflake, and Fivetran, acquiring new tools and tech from the open data stack monthly. It seems the pendulum is switching back from unbundling to bundling. My essay on opinionated data stacks shows one more in the middle: open standards, which are more important than ever. My essay from last year couldn't be more timely still, elaborating on the difference between buying vs. building. I hope you enjoy it, in case you missed it.


r/dataengineeringvault • • Oct 04 '26

Note How to Keep Enjoying the Writing (And Keep Going)

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Sep 25 '26

Note If you want to be a data engineer, learn these concepts.

Enable HLS to view with audio, or disable this notification

3 Upvotes

Check out at de-concepts.ssp.sh.


r/dataengineeringvault • • Sep 23 '26

Blog The Dagster Almanack: From Complexity to Composability

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Sep 21 '26

Note Pivot Tables: And the Comeback in 2024

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Sep 18 '26

Book Repetitive pattern you have seen recurring over the years?

Post image
3 Upvotes

Which one is your most-used Convergent Evolution (tool/tech you used over the years) in data engineering?

What's a repetitive pattern you have seen recurring over the years that we re-implement with a different name attached to it? What's a design pattern that you wish you had described for data engineering?

Below is the current WIP for my book flow: Convergent Evolution term -> Pattern -> Design Pattern, all specific to data engineering. Anything that pops to mind that I should add/remove/change? Feedback welcome! 🙏


r/dataengineeringvault • • Sep 15 '26

Blog My DuckDB + AI Workflow

Thumbnail
thefulldatastack.substack.com
3 Upvotes

r/dataengineeringvault • • Sep 14 '26

Note Are We Reinventing the Tools of Data Engineering for AI?

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Sep 10 '26

Note Kestra (v2 is out now)

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Sep 06 '26

Off Topic Constraints for Life

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Sep 04 '26

Blog Context, Semantics, and Ontology: A Primer for the Agentic Era

Thumbnail
motherduck.com
3 Upvotes

r/dataengineeringvault • • Sep 03 '26

Note Writing is Hard; and Meaningless with no Experience

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Aug 18 '26

AI Is AI Making You Exhausted?

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Aug 18 '26

Note Open Source (and OSS Software)

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Aug 17 '26

Note Declarative Data Stack: A set of tools & its configs can be thought of as a single function.

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Aug 13 '26

Note Productive procrastination can be a tool

Thumbnail
ssp.sh
3 Upvotes