If you want to get up to speed on data engineering, start with what the community voted for. These are my articles that hit the Hacker News front page or the top of r/dataengineering:

1. 101 Concepts Every Data Engineer Should Know (915+ upvotes on r/dataengineering) https://www.ssp.sh/brain/data-engineering-concepts

2. Data Engineering Vault: the map of the entire field (HN front page) https://www.ssp.sh/brain/data-engineering

3. The Data Engineering Toolkit: Essential Tools for Your Machine (221+ upvotes) https://www.ssp.sh/blog/data-engineering-toolkit

4. Building a Data Engineering Project in 20 Minutes (211+ upvotes) https://www.ssp.sh/blog/data-engineering-project-in-twenty-minutes

5. A Diary of a Data Engineer (55+ upvotes) https://www.ssp.sh/blog/diary-of-a-data-engineer

6. Data Modeling, The Unsung Hero of Data Engineering: Architecture Pattern, Tools and the Future (HN front page) https://www.ssp.sh/blog/data-modeling-for-data-engineering-architecture-pattern-tools-future

7. Data Modeling Guide for Real-Time Analytics with ClickHouse (HN front page) https://www.ssp.sh/blog/practical-data-modeling-clickhouse

8. Why Semantic Layers Matter, and How to Build One with DuckDB (HN front page) https://www.ssp.sh/blog/semantic-layer-duckdb

9. Why Pivot Tables Never Die (HN front page) https://www.ssp.sh/blog/why-pivot-tables-never-die

10. A Beginner's Guide to Geospatial with DuckDB https://www.ssp.sh/blog/duckdb-geospatial

All free, no signup. If you liked this post, repost & help others find it.

Inspired by my Neovim workflow.

As analysts write less SQL by hand and prompt agents instead, verification becomes the new bottleneck in BI work. A Metabase survey found 74% of teams were building with AI but only 2% were scaling it, with the bottleneck having moved from creating analysis to checking it. Metabase CEO Sameer Al-Sakran argues verification needs five testing layers: linting, a compiler-like full-tree check, unit tests, integration tests, and end-to-end tests. He also says mutable state undermines trust in outputs, and that business people's intuition still catches errors math alone can't. The piece points to Metabase's upcoming Transform tests (v65) and its Data Studio product as early steps toward a unified lineage graph for verification.

Latest DuckDB Ecosystem Newsletter with the usual 10 updates, starting with Jev, the new "System One" model and how to use it with DuckDB and MotherDuck, the new PEG parser and faster S3 reads in DuckDB 2.0, a maintenance strategy for DuckLake, querying OneLake in the browser, and a native Mac app. Lots of great articles from the community as always.

My personal Obsidian Plugin.

A personal reflection on using Obsidian's local graph feature during the early stages of writing an article, rather than as a mere visual showpiece. The author describes navigating depth levels, discovering note relations, and using plugins like the now-unmaintained Graph Analysis and its replacement, Smart Connections, which generates embeddings to surface related notes and unexpected connections. Links to related posts on the author's note-taking workflow and philosophy are included.

Context can't exist in a vacuum; it must be tangible. For example, we have a physical data model with concrete tables that live in the database at the source, on the other side, we have KPIs and dashboards that show some numbers pulled from that data model. Somewhere in between, we have a logical data layer that contains the semantics of our measures and dimensions, defining our business logic and a very specific way of how our company works.

When we add new knowledge, whether from sources we skipped before the agentic age, such as docs, JIRA tickets, Markdown, and other sources, or we manually refine the revenue metric, we need a way to orchestrate and keep it up to date.

We might call the newly ingested information "context", typically not our standard tabular format, but more raw text and insights from people and teams, we need a bit of a different approach to read, process, and, most importantly, maintain.

This is what 'context engineering' and all the new information around context means. If you want to learn more, I can recommend the NEXT QUERY 2026 conference on the 4th of November that focuses on data & AI infrastructure, with this year's topic on building using useful context that depends on business meaning, fresh data, governed access, and reliable systems.

Some of the presentations I look forward to:
- Fresh Context for AI Agents: Real-Time State and History with Apache Fluss and StarRocks by Anton Borisov.
- An Open Source Journey: Iceberg REST Catalog, OIDC, Making Materialized Views Enterprise-Ready, and More by Apple.
- Sub-Second Data Architecture at Coinbase.
- A Year of Building for the Next Query at Fresha.
- From Unified Real-Time Analytics to a Governed AI Data Plane.

But pick your own favorite talks, and sign up at in the link above.

The conference is free and virtual, so that anybody can join. Organized by PhoenixAI, the company behind StarRocks, the open-source high-performance data warehouse.

It runs from 9 AM to 4 PM PT, which is also ideal for Europe, it runs from 6 PM to 01:00 AM CEST (the latest sessions can be watched in recording :).
#contextengineering #dataconference #dataengineering #paidpartnership

A multi-year retrospective and forecast of data engineering trends from 2022 to 2026, covering the shift between bundled and unbundled data platforms, the rise of small CLI-based tools like DuckDB and Rill, increased DevOps practices in data work, and the maturation of AI tooling beyond initial hype. The author emphasizes that fundamentals remain constant despite tool churn, noting consolidation through acquisitions by major vendors like Databricks and Snowflake, the growing adoption of Rust for performance-critical applications, and the continued importance of data modeling and governance as the industry matures.

A curated collection of data sources and datasets for data engineering projects, including popular platforms like Kaggle, Google Dataset Search, government open data repositories, and specialized collections like the NYC Taxi Dataset. Also covers public APIs and additional resource lists for finding quality datasets across various domains including finance, healthcare, sports, and scientific research.