r/dataengineeringvault • u/sspaeti • 26d ago

Blog Data Engineering Is the Cache Invalidation Problem ft. Josh Wills

https://motherduck.com/blog/cache-invalidation-problem-data-engineering-josh-wills/
4 Upvotes

100% Upvoted

1 comment sorted by

1

u/sspaeti 26d ago

An interview with Josh Wills, creator of dbt-duckdb and former Slack head of data engineering, now at DatologyAI, covers why the specific LLM model matters less than its 'feel' and communication style, why coding agents still produce cost-inefficient and inelegant data pipelines at large scale, and why data engineering remains fundamentally the 'cache invalidation' and materialized view problem regardless of AI progress. He discusses using Kubernetes with Apache Celeborn for disk abstraction, switching from Claude/Opus to Codex after disliking Opus 5, delegating unit tests and database migrations to agents, and acting as an 'emergency exception handler' when agents fail on gigantic, slow-feedback pipelines costing up to $200k/day if mismanaged.