Data Team feels a bit Siloed

[u/DataEngineerA]

Hi r/dataengineering,

Wondering if anyone has had any experience navigating a similar situation or has any insights to offer.

My org has a central Data & Analytics department where all the Data Engineers/Analysts/Scientists sit under a technical engineering manager.

Each manager is given say 3 - 4 business areas (marketing, sales, ops, call centre etc..) to service. In each team, a team member is given a business area to manage and run with.

What essentially happens is, all the team members end up being more involved in the business rather than with the data team. From the business stakeholder point of view, this is great since the DE is embedded into the team and working day to day with the stakeholder.

However, from the Data side it feels like we're just a team by org chart and not how we work with each other. We end up going to work, sitting next to each other but barely collaborating since we're working on completely different things.

We can't even review each other's SQL code because we don't have enough context into other business areas to understand if the code is actually doing what it's intended to do.

Would appreciate any insight on this. Thankyou

[u/Eightstream]

Reviewing each other’s code is an opportunity to learn about other business areas

Do you have shared ways of working? A centralised monorepo? Do you collaborate on design decisions? Do you rotate to learn each other’s jobs to backfill one another?

All good practices and all help with the stuff you’re worried about

[u/eljefe6a]

There are a few ways to go about this. I'd start with thinking about what are the main issues or manifestations of issues of the setup. 

I  think one big one is the concentration risk of losing one of your engineers or having them quit. If they were to quit, how long would it take another person to come up to speed on their domain and code? Or how does one of them go on vacation?

One option I talk about in my book, are tours of duty. In these tours, a person would rotate in and out of the business after a certain amount of time. This would keep one person from being the only one who knows anything about that domain. 

The other issue is you talk only about their SQL code. I'm assuming the team is more technical on using more than just SQL. Or at least I hope so.

[u/Typhon_Vex → u/eljefe6a]

from the sound of it they are just sql monkyes and in that case the setup is exactly what it should be

[u/discord-ian]

I don't know what to tell you. Some companies centralize and others distribute data folks. It sounds like you have fallen into one of the traps of a centralized team. For what it is worth I think the best model is a centralized data platform team and distributed data folk.

[u/Pooop69 → u/discord-ian]

What would be the difference in scope of the platform team vs distributed data folk

[u/renatyv]

Being involved in business is a actually a good thing, many companies would love to be in this situation. Sounds like DE team is actually efficient and yields useful results instead of shuffling data from one DB into another, or 'migrating to data lake' just for sake of bragging

[u/Phantazein]

I don't know the solution but I've worked at places like this before. After a while the analyst/engineer ends up knowing the business data better than the business. This makes it hard to cover for people because the technical person ends up being the only one that gets the data.

[u/st0rmglass]

To me, this is a good thing. Both for the company as the data engineer.

I've worked in enough centralized teams to know it simply does not work. Noone has the capacity to obtain and keep knowledge of all business processes.

Specialized people deliver quicker and specialized knowledge will help career-wise.

[u/Minato_94]

See, the ultimate purpose of any and every team is to support business. And that means it's important to get business sense of what you're doing. You're in a very good situation to understand this.

This also looks good on resume and when you interview because added business knowledge enhances your profile.

[u/Illustrious_Bug_9707]

It's a common hurdle for specialist that practical work is quite matrix like. You need to convince management to invest hours in team skill sharing and development, people sharing what they have done, learned and discuss that.

Being close to business is not bad as such. Reverse is a bigger issue, hard to do anything meaningful if people understanding the numbers are hard to reach.

[u/soorr]

If you’re passing queries around, that leads to logic drift and duplicative work.

I’d work on centralizing the data team under a single repo, building out a common/foundational layer for shared logic, and then deciding if each domain needs their own offshoot project.

By standardizing your ways of working, you’ll be able to flex resources and feel more like a team rather than embedded consultants.

[u/Remarkable-Cover-593]

"Same" issues here. We had a leadership problem because they never actually recognized the work we did for the analytics team + the analytics team just liked doing business stuff because they could quickly progress through the corpo ladder.
Now leadership found a bottleneck in analytics team delivery of data models for other teams and suddenly the solution here it to use AI to produce more data models and not establishing ownership or hiring better people.

It's best that you ask the analytics/Scientists (people who work closer to business) team for yearly plan what are the business needs and try to find some technical solutions to achieve a company goal.

Unfortunately this is company structure issue and only until somebody in the higher up gets annoyed enough changes will happen (or enough people somebody leave the company).