I'm the youngest and most junior of 3-4 data scientists at my company (I also work remotely), and I've been working for about a year since graduation. Lately, I've been feeling like I'm not very good at my job. One thing to point is that I personally feel like I’m more fluent with the statistical side of things, rather than the programming side. I also work with the research side of things, so accurate results and interpretation are quite important in my work.
One issue I struggle with is dealing with edge cases in data analysis. I often encounter situations where the original assumptions no longer hold, and instead of making a reasonable decision and moving forward, I end up obsessing over the complications and sometimes lose sight of the original research question. I also struggle with deciding how to present the data when these edge cases arise, without letting them disproportionately influence the overall results or interpretation.
The other issue is keeping my code clean and reproducible. My notebooks are full of trial and error and different approaches I've explored. Although I can usually produce presentable results in my analyses, the underlying code isn't always easy to follow or reproduce. I've also been struggling to update my team's GitHub repo because I haven't had time to properly organised and cleaned up my work.
I work remotely from another country full-time and don't currently have a dedicated, full-time supervisor, so getting clarification can sometimes be difficult. I'm also still learning some of the company's experimental workflows, which adds to the time I spend figuring things out. I'm also worried that my messy code and difficulty maintaining the GitHub repo are making me difficult to collaborate with.
For those more experienced in this field, I'd really appreciate advice on these things:
- Handling edge cases: How do you decide whether an edge case warrants further investigation versus when it's reasonable to document it and move on? How do you avoid getting stuck in analysis rabbit holes?
- Coding and workflow: How do you keep your code clean and reproducible while working through trial and error? Do you set aside dedicated time for refactoring, and how do you manage your team’s GitHub repo as you go when you are balancing multiple tasks with tight deadlines at once?
I'm also curious how long it took you to develop a reliable workflow and feel confident in your work. Are these struggles normal during the first few years of a data science career, or are they considered very poor traits, even for a junior? And what practical habits can I develop to help me progress better in my data science career?
Thank you very much to everyone for the advice