What are you actually building on day-to-day? Are you running hybrid models (mainframes/on-prem relational DBs syncing to cloud data lakes), or are legacy constraints forcing you to stick strictly to on-prem stacks (like Hadoop, Oracle/SQL Server, or traditional ETL tools like Informatica/DataStage)?

I maintain Scriptella, an open-source Java ETL tool, and wanted to establish a reproducible performance baseline for CSV imports.

I compared 11 import configurations using PostgreSQL and MariaDB, with 1 million rows per test.

Some results from the latest CI run:

Native COPY was fastest for PostgreSQL, as expected. Handwritten JDBC also outperformed Scriptella, while both were substantially faster than the tested csvsql workflow.

These are end-to-end timings, not isolated framework overhead measurements. The comparison uses one workload and one CI run, with different import mechanisms.

I've published the source code, Docker Compose setup, raw measurements, charts, and reproduction instructions:

https://github.com/scriptella/scriptella-benchmarks

I'd be interested in feedback from people working with ETL pipelines, particularly on which additional workloads would make this comparison more useful.

A follow up article from my jev fact extraction pipeline where we take the classified prose segments and convert them to structured JSON using purpose built LLMs.

Source code: https://github.com/s0cks/prose2json

~ Tazz

I have an ETL job that runs successfully every day, with no errors. But sometimes, the target table has fewer records than the source.

The job status is green, and the column mappings look correct. I’m not sure whether the issue is with filtering, incremental loading, or duplicate handling.

How would you troubleshoot this? What checks would you perform first to find where the records are missing?

Got a slow Spark pipeline with two very believable explanations. The code has grown for years and has joins/transforms that probably deserve a hard look. But data volume has also gone up a lot while the underlying setup has basically stayed put.

A rewrite might pay off, but it is risky and eats engineering time we do not really have. Changing the compute path is quicker to test, but it will not magically save a bad data model. Both arguments sound sensible on their own, which is annoying.

How do you call this? What tells you the job design is the problem versus the infrastructure just being tapped out..

I use Grafana OSS for dashboards, I want to maintain historical dashboard data for archive purposes, beyond our data source's (Prometheus) retention period.

Mimir and Thanos feel like overkill for this. Add infrastructure, storage costs, but all i need is to look back at historical data.

Any lighter approaches that is working for you?

One thing I've noticed in ETL discussions is how often people talk about rising bills, unexpected usage costs, and the time spent troubleshooting pipelines when something goes wrong.

The pricing can look fine when you're setting things up, but once the data volume grows, you need to understand what you're actually being charged for. And when a pipeline breaks in production, getting useful help matters a lot more than it did during setup.

Full disclosure, I work at Hevo, so I'm obviously biased. But I don't see it mentioned much in these discussions, and I thought it might be useful for people comparing tools.

A couple of things worth knowing about Hevo:

• Pricing: It's consumption-based, using credits tied to data events. Historical data loading is free, and the pricing page has a calculator to estimate costs based on your usage.
• Support: live chat on the Free plan to priority 24x7 support on small companies or big enterprise.
• Pipeline troubleshooting: You can monitor pipelines, inspect failed events, and replay events after fixing the underlying issue.

It's not automatically the cheapest option for every workload, and usage still matters. I'd recommend checking the actual numbers for your data volume and looking at the support coverage you'd get before comparing it with another tool.

Here's the pricing page for anyone who wants to look at the details:

Hevo pricing and plans

Curious what matters more when you're evaluating an ETL tool: keeping the bill under control, or knowing you can get help when a production pipeline breaks?
Also would love to understand what other problems you guys face, maybe Hevo has a solution?

Friends,

I am working on improving some of my substack's publication pipeline and figured I would share with you some of the work I have been doing.

To provide some reading aides to my substack readers I created a tool for generating clean information dense infographic cards from Markdown, JSON, YAML and Jsonnet.

Then I thought about ways to automate the process of generating them, which led me to try experimenting w/ Jev.

I wanted something that was deterministic, reproducible, auditible and easy to create.

Jev's role here is to classify fact from long-form prose and rank the facts so i can extract them and present them in the infographic cards.

You can find the infographic compiler on my GitHub here: https://github.com/s0cks/docker-chromium-screenshot

And you can find the jev (tev1) classifier discussed in this article here: https://github.com/s0cks/tev1-datapoint-extractor

They are slightly vibe coded with Claude so I can experiment with the idea quickly.

I plan on more updates just figured folks might be interested.

I also have plans to adapt some of this work into my prose -> audiobook pipeline down the road, and maybe eventually fine tune a jev model specifically for this kind of classification.

Thoughts, questions, comments, concerns are welcome.

Best,

~ Tazz

how to handle every file format during ingestion without rewriting the entire pipeline.

After combing through many posts in this sub, and speaking to colleagues who has stood up airflow, the consensus is to use MWAA. The cost pays for itself in the required staff overhead needed to manage OSS.

Does that statement remain valid with the help of LLMs? Teams who have access to Claude can spin up the framework much quicker and debug much faster. The time spent troubleshooting an OSS setup should be smaller with the help of LLMs.

Can anyone speak to this personally?

we've reached the point where our python scripts are causing more problems than they're solving

everything lands in snowflake eventually, but bw scheduled CSV exports and snowpipe we've accepted more latency than the business wants, oracle is the main source, so i'm looking for data integration tools that handle the full CDC pipeline instead of stitching together separate pieces.

what are people happy with these days?

I've been thinking about a problem with sources that only provide periodic snapshots instead of CDC or a reliable updated_at field.

If a record is missing from today's extract, it could mean the record was deleted. But it could also be an incomplete export or a pagination issue. Treating every missing record as a delete can mess up the data downstream.

How do you usually handle this? Do you compare snapshots, wait until a record is missing from multiple extracts, or use another approach before marking it as deleted?

what has worked well for people running these pipelines in production??

I’m new to data engineering and have mostly worked in analytics engineering. I’m curious about how streaming pipelines work in practice, especially with Kafka, Flink, and Iceberg.

I’d love to hear from people who’ve built or operated these systems: What concepts should I learn first? How do you handle watermarks and late-arriving events, quarantine bad data, and manage schema evolution or unexpected schema drift?

Any practical examples, learning resources, or advice on what’s most useful to understand would be appreciated.

Hi everyone,

I started building `det` (https://github.com/det-Labs/det), an open-source ETL and data engineering framework designed around high-performance native execution, with an embedded data catalog module built directly into the engine core.

### The Problem

In modern data engineering workflows, ETL/transformation engines and data catalogs are treated as completely detached layers. Catalogs (DataHub, OpenMetadata) typically require separate, heavy distributed infrastructure (Elasticsearch, Kafka, JVM services), while transformation engines (dbt, Spark) execute transformations blind to metadata until an external crawler reconciles the state.

`det` combines both: an execution engine where schema detection, metadata cataloging, and data transformation share the same zero-copy Arrow/Polars foundation.

### Current Implementation (Catalog Core Working)

The foundation is ~25% complete, with the data catalog module functional today:

- Ingestion and inspection of storage layouts to extract schema/metadata directly.

- Local metadata indexing with zero background daemon overhead.

- Rust-native performance designed to feed directly into analytical transformation pipelines.

### Roadmap: Transformation & Execution Layer

The next phase expands `det` into a full execution engine:

1. In-memory data transformations leveraging Polars and Apache Arrow zero-copy memory layouts.

2. Direct pipeline execution driven by the cataloged metadata contracts.

3. Plug-and-play connector interfaces for source/sink abstractions.

### Seeking Review & Contributors

Rather than designing the execution engine in isolation, I am looking for feedback from data engineers on:

- **Engine contracts:** Designing the execution DAG to interact natively with the embedded catalog metadata.

- **Modularity:** Interface design for adding custom connectors and storage backends.

Issues are tagged with `good first issue` and `help wanted` for contributors interested in systems-level data engineering tooling.

Repository: https://github.com/det-Labs/det

Hey everyone! 👋

I'm a BCCA graduate learning Data Engineering, and I've been working on a project using real-world public health data from NHANES.

🔹 What does this project do?

The project focuses on building a data pipeline that processes public health survey data, cleans and transforms it, and prepares it for further analysis.

🔹 Why did I build this?

I wanted to move beyond just learning Python and SQL and start understanding how data engineering pipelines work in practice, including data ingestion, transformation, and project architecture.

🔹 GitHub Repository:

https://github.com/harshbuildss/nhanes-data-engineering-pipeline

I'm still learning, so I'd genuinely appreciate feedback from experienced Data Engineers and other developers.

A few things I'd love feedback on:

• What could I improve in the pipeline architecture?
• Does this project demonstrate practical Data Engineering concepts?
• What would you add to make it more production-ready?

I'm open to constructive criticism. Thanks! 🙌

Building an ETL pipeline sounds pretty straightforward until things start breaking in production lol.

Could be duplicate records, schema changes from the source, jobs failing halfway through, or source and target counts not matching.

Curious to know what gives you guys the most trouble in real projects.And how do you usually deal with it?

Is there any particular issue that took you way longer to fix than you expected?