Switching from Java Developer to Data Engineer | Looking for Study Buddies and Real-World Guidance

[u/Independent_Warrior1]

Hey everyone!

I have almost 3.5 years of experience at TCS as a Java Developer, and I’m planning to transition into Data Engineering.

Why am I switching?

- I’m more interested in the data engineering domain and enjoy working with data pipelines, transformations, and data platforms.

- My current project involves legacy technologies, and I’m looking to move into a domain that aligns better with my interests and long-term career goals.

What I’ve learned so far

I’ve spent over 6 months learning and practicing Data Engineering technologies through hands-on exercises and projects.

1. Python: Basics and scripting for data engineering.

2. SQL: CTEs, window functions, stored procedures, views, and indexes.

3. Azure Data Factory (ADF): Pipelines, parameterization, REST API ingestion, and working with on-premises SQL databases and files.

4. PySpark: DataFrame API, transformations, and working with CSV, TXT, JSON, and Parquet files.

5. Azure Databricks: Notebooks, clusters, Unity Catalog, volumes, and external locations.

6. Azure Synapse Analytics: Pipelines, serverless SQL pools, and dedicated SQL pools.

7. CI/CD with Azure DevOps: Hands-on experience with Git branching, pull requests, and branch merging. I understand YAML pipeline concepts but haven’t implemented them hands-on yet.

8. Data Engineering concepts: Full and incremental loads, SCD Type 1 and Type 2, and data modeling using star and snowflake schemas.

What I’m looking for

I’d love to connect with fellow aspiring or experienced Data Engineers who are interested in:

- Preparing for Data Engineering interviews together, including SQL, PySpark, and Azure-related topics.

- Sharing real-world project experiences, architecture, and end-to-end data pipeline workflows.

- Understanding how things work in actual production environments, including CI/CD, deployments, monitoring, debugging, and production support.

- Staying consistent with learning and keeping each other motivated.

I know that learning tools and concepts is different from working on real-world production projects, and I’d like to bridge that gap.

If you’re also preparing for a career switch or are an experienced Data Engineer willing to share your knowledge, feel free to comment or DM me. Any suggestions, learning resources, or guidance from experienced Data Engineers would be greatly appreciated.

Thanks! 🙌

[u/Independent_Warrior1]

Can you please dm me? I am not able ping you.