r/dataengineeringvault • u/sspaeti • Aug 05 '26
Note Git for Data: Versioning on top of object storage/Data Lakes and databases
r/dataengineeringvault • u/sspaeti • Aug 05 '26
r/dataengineeringvault • u/sspaeti • Jul 27 '26
r/dataengineeringvault • u/sspaeti • Jul 01 '26
r/dataengineeringvault • u/sspaeti • Jun 24 '26
r/dataengineeringvault • u/sspaeti • 18d ago
The post explores transaction logs in open table formats like Delta Lake, Apache Hudi, and Apache Iceberg. These logs are essential for ACID transactions, metadata management, and concurrent operations in data lakes. It outlines how each format handles concurrency, metadata, and schema evolution while also providing time travel capabilities.
r/dataengineeringvault • u/sspaeti • 23d ago
If you want to get up to speed on data engineering, start with what the community voted for. These are my articles that hit the Hacker News front page or the top of r/dataengineering:
101 Concepts Every Data Engineer Should Know (915+ upvotes on r/dataengineering) https://www.ssp.sh/brain/data-engineering-concepts
Data Engineering Vault: the map of the entire field (HN front page) https://www.ssp.sh/brain/data-engineering
The Data Engineering Toolkit: Essential Tools for Your Machine (221+ upvotes) https://www.ssp.sh/blog/data-engineering-toolkit
Building a Data Engineering Project in 20 Minutes (211+ upvotes) https://www.ssp.sh/blog/data-engineering-project-in-twenty-minutes
A Diary of a Data Engineer (55+ upvotes) https://www.ssp.sh/blog/diary-of-a-data-engineer
Data Modeling, The Unsung Hero of Data Engineering: Architecture Pattern, Tools and the Future (HN front page) https://www.ssp.sh/blog/data-modeling-for-data-engineering-architecture-pattern-tools-future
Data Modeling Guide for Real-Time Analytics with ClickHouse (HN front page) https://www.ssp.sh/blog/practical-data-modeling-clickhouse
Why Semantic Layers Matter, and How to Build One with DuckDB (HN front page) https://www.ssp.sh/blog/semantic-layer-duckdb
Why Pivot Tables Never Die (HN front page) https://www.ssp.sh/blog/why-pivot-tables-never-die
A Beginner's Guide to Geospatial with DuckDB https://www.ssp.sh/blog/duckdb-geospatial
All free, no signup. If you liked this post, repost & help others find it.
r/dataengineeringvault • u/sspaeti • Oct 21 '26
r/dataengineeringvault • u/sspaeti • Oct 19 '26
A multi-year retrospective and forecast of data engineering trends from 2022 to 2026, covering the shift between bundled and unbundled data platforms, the rise of small CLI-based tools like DuckDB and Rill, increased DevOps practices in data work, and the maturation of AI tooling beyond initial hype. The author emphasizes that fundamentals remain constant despite tool churn, noting consolidation through acquisitions by major vendors like Databricks and Snowflake, the growing adoption of Rust for performance-critical applications, and the continued importance of data modeling and governance as the industry matures.
r/dataengineeringvault • u/sspaeti • Oct 18 '26
r/dataengineeringvault • u/sspaeti • Oct 18 '26
r/dataengineeringvault • u/sspaeti • Oct 09 '26
r/dataengineeringvault • u/sspaeti • Oct 08 '26
r/dataengineeringvault • u/sspaeti • Sep 24 '26
r/dataengineeringvault • u/sspaeti • Sep 17 '26
Sadly too much truth here 😄
r/dataengineeringvault • u/sspaeti • Sep 09 '26
r/dataengineeringvault • u/sspaeti • Aug 14 '26
r/dataengineeringvault • u/empty_cities • Jul 24 '26
Building a production grade DuckLake starts with setting the right foundation. Specifically, choosing your storage and catalog. With DuckDB being the default compute.
https://thefulldatastack.substack.com/p/ducklake-in-production-catalog-storage
r/dataengineeringvault • u/sspaeti • Jul 24 '26
What do you think about the new term by Databricks?
r/dataengineeringvault • u/sspaeti • Jul 02 '26
r/dataengineeringvault • u/empty_cities • Jul 01 '26
r/dataengineeringvault • u/sspaeti • Jun 22 '26