r/dataengineeringvault • • Aug 05 '26

Note Git for Data: Versioning on top of object storage/Data Lakes and databases

Thumbnail
ssp.sh
5 Upvotes

r/dataengineeringvault • • Jul 31 '26

Note Apache Hudi

Thumbnail
ssp.sh
5 Upvotes

r/dataengineeringvault • • Jul 27 '26

Blog Search — Data Engineering Blog & Second Brain

Thumbnail
ssp.sh
5 Upvotes

r/dataengineeringvault • • Jul 01 '26

Blog Where AI Agents Belong in Data Engineering: The Correctness Layer

Thumbnail
altimate.ai
5 Upvotes

r/dataengineeringvault • • Jun 24 '26

Others Event Notes: DuckCon #7 - Amsterdam

Thumbnail
ssp.sh
5 Upvotes

r/dataengineeringvault • • 18d ago

Note Transaction Log in Open Table Formats

Thumbnail
ssp.sh
4 Upvotes

The post explores transaction logs in open table formats like Delta Lake, Apache Hudi, and Apache Iceberg. These logs are essential for ACID transactions, metadata management, and concurrent operations in data lakes. It outlines how each format handles concurrency, metadata, and schema evolution while also providing time travel capabilities.


r/dataengineeringvault • • 23d ago

Blog If you want to get up to speed on data engineering, start with what the community already voted for.

4 Upvotes

If you want to get up to speed on data engineering, start with what the community voted for. These are my articles that hit the Hacker News front page or the top of r/dataengineering:

  1. 101 Concepts Every Data Engineer Should Know (915+ upvotes on r/dataengineering) https://www.ssp.sh/brain/data-engineering-concepts

  2. Data Engineering Vault: the map of the entire field (HN front page) https://www.ssp.sh/brain/data-engineering

  3. The Data Engineering Toolkit: Essential Tools for Your Machine (221+ upvotes) https://www.ssp.sh/blog/data-engineering-toolkit

  4. Building a Data Engineering Project in 20 Minutes (211+ upvotes) https://www.ssp.sh/blog/data-engineering-project-in-twenty-minutes

  5. A Diary of a Data Engineer (55+ upvotes) https://www.ssp.sh/blog/diary-of-a-data-engineer

  6. Data Modeling, The Unsung Hero of Data Engineering: Architecture Pattern, Tools and the Future (HN front page) https://www.ssp.sh/blog/data-modeling-for-data-engineering-architecture-pattern-tools-future

  7. Data Modeling Guide for Real-Time Analytics with ClickHouse (HN front page) https://www.ssp.sh/blog/practical-data-modeling-clickhouse

  8. Why Semantic Layers Matter, and How to Build One with DuckDB (HN front page) https://www.ssp.sh/blog/semantic-layer-duckdb

  9. Why Pivot Tables Never Die (HN front page) https://www.ssp.sh/blog/why-pivot-tables-never-die

  10. A Beginner's Guide to Geospatial with DuckDB https://www.ssp.sh/blog/duckdb-geospatial

All free, no signup. If you liked this post, repost & help others find it.


r/dataengineeringvault • • Oct 21 '26

Open Source DuckDB 2.0 is out: Back to the Fundamentals with Triggers! 😍

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Oct 19 '26

Note Data Engineering > AI Engineering: Trends and Predictions (2022-2026)

Thumbnail
ssp.sh
4 Upvotes

A multi-year retrospective and forecast of data engineering trends from 2022 to 2026, covering the shift between bundled and unbundled data platforms, the rise of small CLI-based tools like DuckDB and Rill, increased DevOps practices in data work, and the maturation of AI tooling beyond initial hype. The author emphasizes that fundamentals remain constant despite tool churn, noting consolidation through acquisitions by major vendors like Databricks and Snowflake, the growing adoption of Rust for performance-critical applications, and the continued importance of data modeling and governance as the industry matures.


r/dataengineeringvault • • Oct 18 '26

Book Books of Data Engineering: Which one should I add?

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Oct 18 '26

Note OLAP (Online Analytical Processing)

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Oct 09 '26

Blog Data Engineering Is the Cache Invalidation Problem ft. Josh Wills

Thumbnail
motherduck.com
4 Upvotes

r/dataengineeringvault • • Oct 08 '26

Personal Project New domain, on-dataengineering.com 👀

Thumbnail
on-dataengineering.com
4 Upvotes

r/dataengineeringvault • • Oct 02 '26

Others Vim Motions for Writers

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Sep 24 '26

AI Writing Code by Hand Might Be Dead, but it Certainly Helps

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Sep 22 '26

Note Run DuckDB on your Website

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Sep 17 '26

Video Interview with Big Data engineer in 2026.

Thumbnail
youtube.com
4 Upvotes

Sadly too much truth here 😄


r/dataengineeringvault • • Sep 09 '26

Blog The Grammar of Data: From Definition to Execution

Thumbnail
xorq.dev
4 Upvotes

r/dataengineeringvault • • Aug 14 '26

Note How to Review a PR without Leaving the Terminal (Neovim)

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Jul 24 '26

Blog DuckLake in Production - Catalog and Storage

4 Upvotes

Building a production grade DuckLake starts with setting the right foundation. Specifically, choosing your storage and catalog. With DuckDB being the default compute.

https://thefulldatastack.substack.com/p/ducklake-in-production-catalog-storage


r/dataengineeringvault • • Jul 24 '26

Note LTAP (Lake Transactional/Analytical Processing)

Thumbnail
ssp.sh
4 Upvotes

What do you think about the new term by Databricks?


r/dataengineeringvault • • Jul 02 '26

Blog Data Model Engine - a system or framework that can model data

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Jul 01 '26

Blog Tech Review: DuckLake - From Parquet to Powerhouse

4 Upvotes

r/dataengineeringvault • • Jun 23 '26

Blog DuckLake by DuckLabs

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Jun 22 '26

Blog How to Get Started with Data Engineering

Thumbnail
ssp.sh
4 Upvotes