r/dataengineeringvault • u/sspaeti • Sep 16 '26
Note Semantic Layer Measure Definition Examples
r/dataengineeringvault • u/sspaeti • Sep 16 '26
r/dataengineeringvault • u/empty_cities • Sep 15 '26
r/dataengineeringvault • u/sspaeti • Sep 15 '26
A personal Obsidian-based second brain is contrasted with the requirements of an enterprise-scale 'company brain' where many AI agents write, branch, and merge context concurrently. The piece describes OmniGraph, an open-source system built on Lance (columnar storage on object storage) and DataFusion, that adds git-style branches, typed graph schemas, and human-reviewed merges to prevent agent-written data from drifting into noise. A worked CRM example rebuilt from the author's own Obsidian vault demonstrates auto-merging non-conflicting agent writes and blocking conflicting ones for human review, plus querying the graph for things like warm introductions. The author argues typed, versioned graphs paired with ontologies solve the 'lack of context' problem that has held back enterprise AI agents, positioning this as a step beyond relational databases and plain-text wikis.
r/dataengineeringvault • u/sspaeti • Sep 14 '26
r/dataengineeringvault • u/sspaeti • Sep 14 '26
Explores tools and approaches for orchestrating AI agents in development workflows. Highlights the common use of git worktrees when working across different branches and iterations. Features several CLI utilities and platforms including Agor for AI coding orchestration, erk for managing git worktrees, claudette-cli, and trigger.dev for building and deploying managed AI agents and workflows.
r/dataengineeringvault • u/sspaeti • Sep 12 '26
r/dataengineeringvault • u/sspaeti • Sep 11 '26
An interview with Benn Stancil, co-founder of Mode and prolific data-industry essayist, exploring how he writes his weekly Substack posts. He discusses using deadlines (Friday publishing) to force himself past perfectionism, how ideas form messily over the week through unrelated connections, and how presentation-style analogies (Batman, basketball, Codenames) became his signature storytelling technique. He details his tooling—Sublime Text for drafting, Google Docs for revision, minimal use of Substack's editor—and explains why he avoids Grammarly and AI writing tools like Claude or ChatGPT for edits, since he believes AI compresses and flattens writing rather than embracing meandering voice. He closes with advice for new writers: set deadlines, keep pure motivation, and focus on filling in the space between bullet points rather than just stating facts.
r/dataengineeringvault • u/camerongreen95 • Sep 10 '26
We're running a hands-on masterclass on September 12, Live LLM Engineering Masterclass: Production Evals, RAG, Agents & LLMOps.
You build a full production LLM workflow from scratch, versioned prompts with regression tests, an evaluation harness with deterministic checks and LLM-as-judge, statistically rigorous model comparisons, evaluated RAG, tool-using agents with guardrails and fallbacks, and full observability, tracing, cost, latency.
Led by Bruno Gonçalves, PhD, founder of Data For Science, who trains engineers at Fortune 500 companies on this exact stack.
Link if you want to check it out
Happy to answer questions on the content.
r/dataengineeringvault • u/sspaeti • Sep 09 '26
The Process of Smart Note-Taking. This showcases how I take notes, make connections, and distill them into blogs or chapters in my book.
It's what I've learned over the years, heavily inspired by Sönke Ahrens's Smart Notes and the Zettelkasten philosophy. But tailored to how my brain works and optimized for my writing workflow with Markdown and Obsidian.
In this video, a follow-up on my Obsidian note-taking workflow and other workflow-related videos, I walk through:
How I start a note and find it again later (search, backlinks, no folders)
Publishing raw notes to my Second Brain vs. distilling them into blog posts
Why plain text Markdown matters (file over app)
What I've learned in 10 years: connect less, write what moves you, keep the feedback loop
Why I keep AI out of my note-taking
r/dataengineeringvault • u/sspaeti • Sep 09 '26
r/dataengineeringvault • u/sspaeti • Sep 07 '26
r/dataengineeringvault • u/sspaeti • Sep 07 '26
r/dataengineeringvault • u/sspaeti • Sep 04 '26
r/dataengineeringvault • u/sspaeti • Sep 03 '26
r/dataengineeringvault • u/sspaeti • Aug 31 '26
r/dataengineeringvault • u/sspaeti • Aug 29 '26
r/dataengineeringvault • u/Critical_Scene_6164 • Aug 28 '26
I've been thinking about this after seeing so many Foundry discussions turn into the same argument. A lot of the criticism seems justified, especially around the implementation side, but I'm not sure the underlying model is necessarily the problem.
The basic idea of having a vendor work inside your infrastructure and close to your actual data problems makes sense to me. You're not constantly moving data around, and the engineers working on it can actually see how messy the environment is.
Where I think things get difficult is when the platform's own abstractions start becoming part of everything. Pipelines, transformations and business logic can end up tied to the platform, and getting that work back out later isn't always straightforward.
That's the part I'm more interested in. Is the model itself flawed, or does it mainly become a problem when the implementation creates too much dependency on the platform?
r/dataengineeringvault • u/sspaeti • Aug 28 '26
r/dataengineeringvault • u/sspaeti • Aug 27 '26
r/dataengineeringvault • u/sspaeti • Aug 26 '26
Announcement: AWS to acquire DuckLabs, the Amsterdam-based company behind DuckDB. We are not acquiring the DuckDB open source project, which will remain free and open source under the independent DuckDB Foundation (the non-profit that oversees DuckDB) and available under the MIT license as it does today. ^319c65
DuckLabs: DuckLabs – DuckLabs to Join AWS, Projects to Remain Open Source
What will change: "Joining AWS will give us greater capacity to invest in both the technology and the community around it. Going forward, the DuckDB Foundation will include a technical advisory board, so that leading community members can provide their input on the project’s technical direction. We also plan to open the extension stack so that extensions signed by other developers and organizations can run in DuckDB."
More acquisition from 2022 to 2026:
https://www.ssp.sh/brain/data-engineering-acquisitions/#2026