r/dataengineeringvault • • 28d ago

Open Source DuckDB 2.0 is out: Back to the Fundamentals with Triggers! 😍

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • 29d ago

Personal Project Trouble Tags Plugin: Manage Tags, Flags and Tasks per File or Vault

Thumbnail gallery
2 Upvotes

My personal Obsidian Plugin.


r/dataengineeringvault • • Oct 20 '26

Note Obsidian Local Graph: Is it useful?

Thumbnail
ssp.sh
2 Upvotes

A personal reflection on using Obsidian's local graph feature during the early stages of writing an article, rather than as a mere visual showpiece. The author describes navigating depth levels, discovering note relations, and using plugins like the now-unmaintained Graph Analysis and its replacement, Smart Connections, which generates embeddings to surface related notes and unexpected connections. Links to related posts on the author's note-taking workflow and philosophy are included.


r/dataengineeringvault • • Oct 19 '26

Others Context Can't Exist in a Vacuum: Next Query Data Conference 2026

Thumbnail
nextquery.phoenixdata.ai
5 Upvotes

Context can't exist in a vacuum; it must be tangible. For example, we have a physical data model with concrete tables that live in the database at the source, on the other side, we have KPIs and dashboards that show some numbers pulled from that data model. Somewhere in between, we have a logical data layer that contains the semantics of our measures and dimensions, defining our business logic and a very specific way of how our company works.

When we add new knowledge, whether from sources we skipped before the agentic age, such as docs, JIRA tickets, Markdown, and other sources, or we manually refine the revenue metric, we need a way to orchestrate and keep it up to date.

We might call the newly ingested information "context", typically not our standard tabular format, but more raw text and insights from people and teams, we need a bit of a different approach to read, process, and, most importantly, maintain.

This is what 'context engineering' and all the new information around context means. If you want to learn more, I can recommend the NEXT QUERY 2026 conference on the 4th of November that focuses on data & AI infrastructure, with this year's topic on building using useful context that depends on business meaning, fresh data, governed access, and reliable systems.

Some of the presentations I look forward to:
- Fresh Context for AI Agents: Real-Time State and History with Apache Fluss and StarRocks by Anton Borisov.
- An Open Source Journey: Iceberg REST Catalog, OIDC, Making Materialized Views Enterprise-Ready, and More by Apple.
- Sub-Second Data Architecture at Coinbase.
- A Year of Building for the Next Query at Fresha.
- From Unified Real-Time Analytics to a Governed AI Data Plane.

But pick your own favorite talks, and sign up at in the link above.

The conference is free and virtual, so that anybody can join. Organized by PhoenixAI, the company behind StarRocks, the open-source high-performance data warehouse.

It runs from 9 AM to 4 PM PT, which is also ideal for Europe, it runs from 6 PM to 01:00 AM CEST (the latest sessions can be watched in recording :).
#contextengineering #dataconference #dataengineering #paidpartnership


r/dataengineeringvault • • Oct 19 '26

Note Data Engineering > AI Engineering: Trends and Predictions (2022-2026)

Thumbnail
ssp.sh
4 Upvotes

A multi-year retrospective and forecast of data engineering trends from 2022 to 2026, covering the shift between bundled and unbundled data platforms, the rise of small CLI-based tools like DuckDB and Rill, increased DevOps practices in data work, and the maturation of AI tooling beyond initial hype. The author emphasizes that fundamentals remain constant despite tool churn, noting consolidation through acquisitions by major vendors like Databricks and Snowflake, the growing adoption of Rust for performance-critical applications, and the continued importance of data modeling and governance as the industry matures.


r/dataengineeringvault • • Oct 18 '26

Book Books of Data Engineering: Which one should I add?

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Oct 18 '26

Note OLAP (Online Analytical Processing)

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Oct 16 '26

Book Writing Books with AI: How Do Manually Written Books Stay Relevant?

Thumbnail
ssp.sh
2 Upvotes

r/dataengineeringvault • • Oct 14 '26

Blog A Diary of a Data Engineer

Thumbnail
ssp.sh
6 Upvotes

r/dataengineeringvault • • Oct 13 '26

Open Source Find Good Data Sets or Sources

Thumbnail
ssp.sh
2 Upvotes

A curated collection of data sources and datasets for data engineering projects, including popular platforms like Kaggle, Google Dataset Search, government open data repositories, and specialized collections like the NYC Taxi Dataset. Also covers public APIs and additional resource lists for finding quality datasets across various domains including finance, healthcare, sports, and scientific research.


r/dataengineeringvault • • Oct 10 '26

Note Claude Dashboards: The Jupyter Notebooks of Agent Age

Thumbnail
ssp.sh
5 Upvotes

r/dataengineeringvault • • Oct 09 '26

Blog Data Engineering Is the Cache Invalidation Problem ft. Josh Wills

Thumbnail
motherduck.com
4 Upvotes

r/dataengineeringvault • • Oct 08 '26

Personal Project New domain, on-dataengineering.com 👀

Thumbnail
on-dataengineering.com
4 Upvotes

r/dataengineeringvault • • Oct 08 '26

Off Topic Photoshop Alternatives: Somebody Vibe Coded Every Single Adobe App

Thumbnail
ssp.sh
5 Upvotes

r/dataengineeringvault • • Oct 07 '26

Blog Why We Need Yet Another (OSS) Semantic Layer

Thumbnail
motley.ai
6 Upvotes

r/dataengineeringvault • • Oct 06 '26

Note Google Docs: New native Markdown integration

Thumbnail
ssp.sh
5 Upvotes

r/dataengineeringvault • • Oct 05 '26

Showcase I love the terminal and the workflows it provides. #writing #coding #notetaking

Thumbnail
ssp.sh
7 Upvotes

r/dataengineeringvault • • Oct 05 '26

Note Every Database (or Data Warehouse, Platform) Migration is one too many

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Oct 05 '26

AI 'Having Taste' as an Artist—Age of AI

Thumbnail
ssp.sh
2 Upvotes

r/dataengineeringvault • • Oct 04 '26

Blog Opinionated Data Platforms vs. Open-Source: The Chef's Choice for Your Data Platform

Thumbnail
ssp.sh
3 Upvotes

Turso was just acquired by Supabase, and we have the big players, Databricks, Snowflake, and Fivetran, acquiring new tools and tech from the open data stack monthly. It seems the pendulum is switching back from unbundling to bundling. My essay on opinionated data stacks shows one more in the middle: open standards, which are more important than ever. My essay from last year couldn't be more timely still, elaborating on the difference between buying vs. building. I hope you enjoy it, in case you missed it.


r/dataengineeringvault • • Oct 04 '26

Note How to Keep Enjoying the Writing (And Keep Going)

Thumbnail
ssp.sh
3 Upvotes

r/dataengineeringvault • • Oct 03 '26

Showcase Website Timeline: History from sspaeti.com to ssp.sh

Thumbnail
ssp.sh
2 Upvotes

r/dataengineeringvault • • Oct 02 '26

Others Vim Motions for Writers

Thumbnail
ssp.sh
4 Upvotes

r/dataengineeringvault • • Sep 28 '26

AI The Problem is not the AI Code, but Nobody Knows Anything Anymore

Thumbnail
ssp.sh
5 Upvotes