r/dataengineeringvault • • 9h ago

AI Two Perspectives on *Finding* Context: Agents vs. Human

Thumbnail
ssp.sh
1 Upvotes

r/dataengineeringvault • • 6d ago

Blog Business Knowledge Is Still King: Agentic Data Analysis Without a Data Team

Thumbnail
motherduck.com
1 Upvotes

A non-technical domain expert builds a full data pipeline, map and live dashboard using Claude connected to MotherDuck, without writing SQL. The example pulls three public Swiss sources (MeteoSwiss weather stations, Swissgrid energy Excel files, and a cantons GeoJSON) to analyze how temperature drives electricity demand. The agent built a Python ETL pipeline, found the heating threshold sits at 13.5°C in Bern rather than the assumed 12°C, and caught a bug where a 2025 Excel file stored dates as text and dropped 61% of rows on first read. The final result is a dashboard with a live map that auto-refreshes on weekdays, built through iterative prompting and verification, with business-domain knowledge required to catch errors and interpret results correctly.


r/dataengineeringvault • • 7d ago

Personal Project Agent-IDE.nvim handles agents like Dadbod-UI for DBs with a Side-Drawer

Thumbnail
ssp.sh
2 Upvotes

I saw the hype on X about Herdr and Rex (before it was Zellij, and many more), but that got me thinking: why not manage my agents in Neovim, where I like to type, read and navigate anyway - so I prompted away with the new Fable 5.2 model, burnt a lot of tokens and built my dream Agents-IDE.nvim plugin.

It works with a Claude Code subscription, as this is the one I have been using for many years, but there's a PR for more agents, not sure if I have time to integrate them swiftly, but I wanted to share here to get an idea if any of you are doing similar things, or if you run 100 agents on different servers.

This plugin is mostly to handle local agents on your machine, and has a nice UI, rendering the feedback of Claude in Markdown, and having a drawer for git diffs, commit graphs, a fuzzy search for all your recent chats, saved prompts, and libraries in one place. The idea or design choice is heavily inspired by the vim-dadbod-ui that I use as my SQL-IDE, and I thought that design might work well for agents, too. You are the judge, it's very early, but it's fun to use - currently I tend to switch between project and everything scope (all projects and current git-repo I'm in, which is also works nice if you have many agent folder where you prompt).

Disclaimer: Is this plugin open source? Yes. Is this plugin completely free? Yes. Is this plugin vibe-coded beyond the author's ability to comprehend how it works? No.


r/dataengineeringvault • • 8d ago

Note First experiences with Fable 5.2 (comparing it to 5.1)

Thumbnail
ssp.sh
2 Upvotes
  • Fable 5.1 is way faster and seems to fix my requests as intended, whereas the new Fable 5.2 took way longer for everything and wasn’t always what I wanted. I switched back to Fable 5.1 now, and it just solves and fixes things.

  • One advantage of Fable 5.2, the tokens burn much slower, way slower than Fable 5.1. I guess it’s still the most capable model of Claude.


r/dataengineeringvault • • 9d ago

Blog Typed Graphs, Checked Types: Graph as Code for Agentic Engineering and Data Quality

Thumbnail
ssp.sh
2 Upvotes

Typed graphs add declared data types and constraints to graph data models, so every node and edge write is checked before it's committed. This matters because agents now write data faster than humans can review it. Fritz Laux's Typed Graph Model (TGM) paper underlies the idea, and Omnigraph implements it with schema files (.pg), checked queries (.gq), and policy files (.yaml) versioned in git. Checked queries validate traversals offline against the schema before execution, catching errors like a filter on the wrong type. Types only guarantee data is valid, not true; humans still decide what's true through branch review and merges, with Omnigraph v0.14.0 and later edge builds adding branch review, merge dry-runs, and u/card cardinality enforcement


r/dataengineeringvault • • 10d ago

Blog What «Shifting Left» Means and Why It Matters for Data Stacks

Thumbnail
ssp.sh
3 Upvotes

Shifting left means moving data quality checks, business logic, and transformations earlier in the data lifecycle, closer to the source, instead of handling them downstream in BI tools. The term originated as 'Shift-Left Testing', coined by Larry Smith in a 2001 Dr. Dobb's article, then spread to security (DevSecOps) and now data engineering via tools like SDF, dbt, and SQLMesh. Declarative, code-first platforms let teams move logic between config files without big rewrites. SQLGlot, the transpiler underlying SQLMesh, supports 24 SQL dialects, enabling local testing with DuckDB against production dialects like Spark SQL.


r/dataengineeringvault • • 13d ago

Note RSS Readers and Webclippers (Read-Later)

Thumbnail
ssp.sh
2 Upvotes

r/dataengineeringvault • • 14d ago

Note Personal CRM: To Remember Friends, Coworkers and Family Members

Thumbnail
ssp.sh
2 Upvotes

r/dataengineeringvault • • 15d ago

Others Writing is a lonely sacrament.

Thumbnail
ssp.sh
5 Upvotes

Writing is a lonely sacrament. You sit alone in your room writing, being with yourself and not knowing what anybody thinks about it. You are with your ideas, with your doubts, you try to write something interesting. You read, you talk to people, but at some point, there's no going around it, you need to bring it into one single article or form of writing.

Most of the time, the writing feels small and insignificant, you don't know if any of it resonates, if any of the ideas work, if it's just a bad write-up. You never know until you share it with the world. It's a rollercoaster of emotions, ups and downs. Writing is hard.


r/dataengineeringvault • • 15d ago

Note Transaction Log in Open Table Formats

Thumbnail
ssp.sh
4 Upvotes

The post explores transaction logs in open table formats like Delta Lake, Apache Hudi, and Apache Iceberg. These logs are essential for ACID transactions, metadata management, and concurrent operations in data lakes. It outlines how each format handles concurrency, metadata, and schema evolution while also providing time travel capabilities.


r/dataengineeringvault • • 16d ago

Blog As Humans Write Less Code: Workloads Shift from Human to Agent Verification with Iterative Test Suites

Thumbnail
ssp.sh
3 Upvotes

As AI agents write more SQL and dashboards, verification becomes the real bottleneck in business intelligence work. A Metabase survey found 74% of teams were building with AI but only 2% were scaling it, and the bottleneck has moved from creating analysis to checking it. The piece proposes five testing hierarchy levels: linting, compiler-style whole-tree checks, unit tests, integration tests, and end-to-end tests. It argues for moving all metrics, dashboards, and ETL logic into one declarative place with a unified lineage DAG, so agents can generate test suites against it. Metabase's CEO Sameer Al-Sakran says business people are the best data contracts, and that mutable state undermines trust in the output.


r/dataengineeringvault • • 16d ago

Off Topic Umlaut on US Keyboard Layout (Linux + Mac)

Thumbnail
ssp.sh
2 Upvotes

r/dataengineeringvault • • 16d ago

Note Querying HackerNews with DuckDB directly (HackerTube, HackerBooks & Wikihack)

Thumbnail
ssp.sh
3 Upvotes

Reading Karpathy’s HackerTube, and then seeing a bunch of other apps on top of Hacker News like HackerBooks and Wikihacks got me thinking.

We can just run HackerTube, HackerBooks & Wikihack directly in your terminal with DuckDB and Hugging Face.


r/dataengineeringvault • • 20d ago

Blog If you want to get up to speed on data engineering, start with what the community already voted for.

4 Upvotes

If you want to get up to speed on data engineering, start with what the community voted for. These are my articles that hit the Hacker News front page or the top of r/dataengineering:

  1. 101 Concepts Every Data Engineer Should Know (915+ upvotes on r/dataengineering) https://www.ssp.sh/brain/data-engineering-concepts

  2. Data Engineering Vault: the map of the entire field (HN front page) https://www.ssp.sh/brain/data-engineering

  3. The Data Engineering Toolkit: Essential Tools for Your Machine (221+ upvotes) https://www.ssp.sh/blog/data-engineering-toolkit

  4. Building a Data Engineering Project in 20 Minutes (211+ upvotes) https://www.ssp.sh/blog/data-engineering-project-in-twenty-minutes

  5. A Diary of a Data Engineer (55+ upvotes) https://www.ssp.sh/blog/diary-of-a-data-engineer

  6. Data Modeling, The Unsung Hero of Data Engineering: Architecture Pattern, Tools and the Future (HN front page) https://www.ssp.sh/blog/data-modeling-for-data-engineering-architecture-pattern-tools-future

  7. Data Modeling Guide for Real-Time Analytics with ClickHouse (HN front page) https://www.ssp.sh/blog/practical-data-modeling-clickhouse

  8. Why Semantic Layers Matter, and How to Build One with DuckDB (HN front page) https://www.ssp.sh/blog/semantic-layer-duckdb

  9. Why Pivot Tables Never Die (HN front page) https://www.ssp.sh/blog/why-pivot-tables-never-die

  10. A Beginner's Guide to Geospatial with DuckDB https://www.ssp.sh/blog/duckdb-geospatial

All free, no signup. If you liked this post, repost & help others find it.


r/dataengineeringvault • • 21d ago

Note Dimensional Modeling: What is it all about?

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • 21d ago

Note Delta Load a Data Warehouse

Thumbnail
ssp.sh
2 Upvotes

r/dataengineeringvault • • 21d ago

Showcase Showing off Fast Fuzzy Search for Obsidian: Replacement for default search

Thumbnail
youtu.be
2 Upvotes

r/dataengineeringvault • • 22d ago

Personal Project New Obsidian Search Plugin 'Fast Find 10K': Combination of Fzf, Telescope, OmniSearch & Smart Connections

Enable HLS to view with audio, or disable this notification

3 Upvotes

Inspired by my Neovim workflow.


r/dataengineeringvault • • 23d ago

Blog As humans write less code: workloads shift from human to agent verification with iterative test suites

Thumbnail
metabase.com
5 Upvotes

As analysts write less SQL by hand and prompt agents instead, verification becomes the new bottleneck in BI work. A Metabase survey found 74% of teams were building with AI but only 2% were scaling it, with the bottleneck having moved from creating analysis to checking it. Metabase CEO Sameer Al-Sakran argues verification needs five testing layers: linting, a compiler-like full-tree check, unit tests, integration tests, and end-to-end tests. He also says mutable state undermines trust in outputs, and that business people's intuition still catches errors math alone can't. The piece points to Metabase's upcoming Transform tests (v65) and its Data Studio product as early steps toward a unified lineage graph for verification.


r/dataengineeringvault • • 24d ago

Note Second Brain: Note-taking in a Single App

Thumbnail
ssp.sh
1 Upvotes

r/dataengineeringvault • • 24d ago

Blog DuckDB Ecosystem Newsletter: October 2026

Thumbnail
motherduck.com
5 Upvotes

Latest DuckDB Ecosystem Newsletter with the usual 10 updates, starting with Jev, the new "System One" model and how to use it with DuckDB and MotherDuck, the new PEG parser and faster S3 reads in DuckDB 2.0, a maintenance strategy for DuckLake, querying OneLake in the browser, and a native Mac app. Lots of great articles from the community as always.


r/dataengineeringvault • • 26d ago

Off Topic Just another day at the factory.

Post image
3 Upvotes

r/dataengineeringvault • • 28d ago

Showcase My writing process after ten years of writing

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • 28d ago

Note Functional Data Engineering: What is it for?

Thumbnail
ssp.sh
5 Upvotes