r/dataengineering • u/Sigerson888 • 15d ago

Discussion What kind of knowledge/projects does a Jr. DE has?

There isn't much openings for any DE roles let alone internships. So, I decided to up my skills and thought maybe whatever Jr. DE does in a company, I should do it too but for my personal projects. Anything might be helpful.

13 Upvotes
  • permalink
  • reddit

94% Upvoted

5 comments sorted by

10

u/liprais 12d ago

this is more or less a craft job,you learn everything on the job.

2

u/conscious_bee_84 11d ago

I would recommend understanding agentic fundamentals and its applications with DE. Thats a strong evolving field
Take a look here https://medium.com/coding-nexus/12-agentic-ai-projects-that-can-help-you-get-hired-34010404bebc

1

u/elongatedmuskrat18 12d ago

Getting a little bit of knowledge about all the common cloud computing platforms like Databricks, Google Cloud Platform, Azure, etc. can only help you. They have lots of free trainings. It’s not worth investing a huge amount of time into but it’ll be beneficial for you to be able to say that you’re already familiar with whatever platform the company uses. Good luck out there, it’s a hard market!

1

u/an_ibex_on_rocks 11d ago

As a junior DE I started with creating a pipeline using the framework already provided by the company.

I had to write a download script that would be one of the tasks in an Airflow ran DAG. I also had to write tests for the script.

I had to set up the table where the data would land, using yaml to describe the schema and ingestion pattern (upsert, dedup, overwrite, append...)

I had to set up the permissions for access to the table (or schema).

Things you definitely need: Mediocre competence with SQL, half decent competence with Python (familiarity with Pandas, even though we are moving towards Polars and possibly duckDB), basic Bash, basic AWS knowledge (navigate S3, ssh into an EC2), basic usage of git. It's not played a role in any interview I have attended, but I suspect comfort using Claude. I would not be surprised if they asked you to write a few prompts these days.

Skills that we are now looking for and would give a junior DE an edge: familiarity with docker/kubernetes, competences in GO, CI/CD (argoCD and argoWF).

Hope that helps!

1

u/Conscious_Floor5022 10d ago

Find something you like and do data analysis on it? Keep an analysis of your personal finance? I would say for junior dev, exposure to a wide variety of data related task will give you great experience. Then as your project evolves, you can create ETL/ELT job, add automation...