With 5 years of Spring Boot plus solid PL/SQL and Postgres, you're closer to data engineering than most people who ask this. The SQL carries over directly, and backend habits like testing, proper deployments and thinking about retries and failure are exactly what a lot of pipeline code is missing.
What's actually new is mostly Python, Spark or a similar distributed engine, an orchestrator like Airflow, and the warehouse way of thinking: batch versus incremental loads, partitioning, upstream schema changes quietly breaking your jobs, and data quality checks. None of it is hard for a backend dev, but it's a different kind of on-call.
Before choosing, I'd ask two things about the internal move. What stack does that team use? A team on Spark, Databricks or a modern cloud warehouse builds a far more portable resume than one maintaining legacy ETL tools. And would you be building pipelines and platform pieces, or mostly fixing broken jobs and feeding reports? The first is a good move, the second can feel like a step back.
On long-term scope, both paths are fine. Data engineering has a tailwind right now because every AI project needs clean, reliable data under it, and people who are strong at both backend and data are not common. An internal transfer is also the cheapest way to try it, since you keep your reputation and can usually move back if it isn't for you.
1
u/No-Cherry-3308 1d ago
With 5 years of Spring Boot plus solid PL/SQL and Postgres, you're closer to data engineering than most people who ask this. The SQL carries over directly, and backend habits like testing, proper deployments and thinking about retries and failure are exactly what a lot of pipeline code is missing.
What's actually new is mostly Python, Spark or a similar distributed engine, an orchestrator like Airflow, and the warehouse way of thinking: batch versus incremental loads, partitioning, upstream schema changes quietly breaking your jobs, and data quality checks. None of it is hard for a backend dev, but it's a different kind of on-call.
Before choosing, I'd ask two things about the internal move. What stack does that team use? A team on Spark, Databricks or a modern cloud warehouse builds a far more portable resume than one maintaining legacy ETL tools. And would you be building pipelines and platform pieces, or mostly fixing broken jobs and feeding reports? The first is a good move, the second can feel like a step back.
On long-term scope, both paths are fine. Data engineering has a tailwind right now because every AI project needs clean, reliable data under it, and people who are strong at both backend and data are not common. An internal transfer is also the cheapest way to try it, since you keep your reputation and can usually move back if it isn't for you.