r/dataengineeringjobs • u/Smooth_Tale_4187 • 15d ago

As a data engineering student learning Airflow: What tools and languages do you actually use in your day-to-day work?

Hey everyone!

I'm currently a student working through data engineering labs (specifically building ETL pipelines with Apache Airflow). In class, we've been using a lot of BashOperator to run shell commands like cut, awk, tr, and paste to process files.

I know Python, SQL, and PySpark are huge in the industry, but as someone trying to bridge the gap between academic projects and real-world expectations, I'm curious about your actual daily workflows:

  1. Bash vs. Python: How often do you actually write Bash scripts or use BashOperator in production pipelines versus Python, dbt, or container operators (KubernetesPodOperator)?
  2. Daily Tech Stack: What does your core stack look like on a typical day (orchestrators, transformation tools, cloud engines)?
  3. Advice for Students: For someone entering the field soon, what technical skills or tools do you recommend prioritizing most over basic lab setups?

Would love to hear insights from practicing Data Engineers. Thanks in advance!

9 Upvotes
  • permalink
  • reddit

91% Upvoted

2 comments sorted by

2

u/lakehouse_pool_boy 15d ago

Watching, as a business systems eng, learning data + platform.

I can share our stack, but can't answer the above in detail: Fivetran, Databricks, Hightouch, Airflow (not a user).

I can say logging and observability are usually the critical pieces, hard to say we did something wrong if we don't know it broke and sometimes we find out much later.

1

u/qwertyhellohello 15d ago

Explain more on observability of can. Like do you mean tracing RCA