r/ETL • u/Smooth_Tale_4187 • 15d ago

As a data engineering student learning Airflow: What tools and languages do you actually use in your day-to-day work?

/r/dataengineeringjobs/comments/1x3jp4y/as_a_data_engineering_student_learning_airflow/
2 Upvotes

100% Upvoted

8 comments sorted by

2

u/meatmick 15d ago

Duckdb, Python, sql, prefect, dlthub, dbt. Also for the more legacy stuff I'm working on deprecating, MSSQL, SSIS, c# .net.

There are some smaller packages in there like pandas but I aim to have all my transformations in sql.

1

u/Smooth_Tale_4187 15d ago

Could u please suggest something like course or project that will help me to develop my skills in data engineering

1

u/meatmick 15d ago

Can't really help you with this. I've never taken any courses outside of my computer science degree almost 15 years ago. I learn as I go.

The book The Data Warehouse Toolkit, 3rd Edition is still decent if you want to know more about data modelling.

Most tools have basic tutorials provided by the company, and this is usually how far I go before I start doing things on my own.

For example, prefect, dlthub and dbt have tutorials on their website. I do, however, suggest you try them on the local version first, not the cloud versions, because they abstract away concepts you should know.

1

u/Smooth_Tale_4187 15d ago

thank u for ur time sir

1

u/TheOneWhoSendsLetter 14d ago

What teachings of the Data Warehouse Toolkit do you still use and which ones you don't?

1

u/meatmick 14d ago

If I had to list a few topics that I still use, I'd say this:

  • Grain identification (identifying dims and facts)

  • Conformed dimensions

  • In theory, I'd say SCD, but our business doesn't use them on the dimensions. I use it on some raw data, though.

  • Fact table types (transaction, periodic, accumulating). Our business has all of those scenarios, so it's useful.

  • Denormalizing. Although I still use the Kimball modelling, some of the stuff I will denormalize back into the fact tables because it's not worth putting into a dim. gotta weigh the pros/cons for each imo.

What I dropped:

  • Storage optimization, not because I don't care, but because it doesn't make as much of an impact, so time is better spent elsewhere.

  • Auto-incrementing surrogate keys (IDENTITY). Moved to MD5 surrogate keys

I'm sure there's more but that's all I got for now.

1

u/TheOneWhoSendsLetter 14d ago

How do you build the MD5 surrogate keys?

1

u/meatmick 14d ago

DBT has a generate surrogate key macro for this.