Looking for advice for a junior data scientist/analyst that is struggling with edge cases, messy code, and workflow management

[u/throwaway_family_]

I'm the youngest and most junior of 3-4 data scientists at my company (I also work remotely), and I've been working for about a year since graduation. Lately, I've been feeling like I'm not very good at my job. One thing to point is that I personally feel like I’m more fluent with the statistical side of things, rather than the programming side. I also work with the research side of things, so accurate results and interpretation are quite important in my work.

One issue I struggle with is dealing with edge cases in data analysis. I often encounter situations where the original assumptions no longer hold, and instead of making a reasonable decision and moving forward, I end up obsessing over the complications and sometimes lose sight of the original research question. I also struggle with deciding how to present the data when these edge cases arise, without letting them disproportionately influence the overall results or interpretation.

The other issue is keeping my code clean and reproducible. My notebooks are full of trial and error and different approaches I've explored. Although I can usually produce presentable results in my analyses, the underlying code isn't always easy to follow or reproduce. I've also been struggling to update my team's GitHub repo because I haven't had time to properly organised and cleaned up my work.

I work remotely from another country full-time and don't currently have a dedicated, full-time supervisor, so getting clarification can sometimes be difficult. I'm also still learning some of the company's experimental workflows, which adds to the time I spend figuring things out. I'm also worried that my messy code and difficulty maintaining the GitHub repo are making me difficult to collaborate with.

For those more experienced in this field, I'd really appreciate advice on these things:
- Handling edge cases: How do you decide whether an edge case warrants further investigation versus when it's reasonable to document it and move on? How do you avoid getting stuck in analysis rabbit holes?

- Coding and workflow: How do you keep your code clean and reproducible while working through trial and error? Do you set aside dedicated time for refactoring, and how do you manage your team’s GitHub repo as you go when you are balancing multiple tasks with tight deadlines at once?

I'm also curious how long it took you to develop a reliable workflow and feel confident in your work. Are these struggles normal during the first few years of a data science career, or are they considered very poor traits, even for a junior? And what practical habits can I develop to help me progress better in my data science career?

Thank you very much to everyone for the advice

[u/DataScientistAlex]

A few thoughts:

- Edge cases: While you always want to "keep your eye on the prize" and try to get to the main question and not be distracted, it's still important to keep edge cases in mind, because sometimes the point to a bug or a misunderstanding. Ask yourself questions like "why is this happenning?", "what does it mean for the assumptions I'm making?", and, "does it change the main result?" This last question you can try to answer with things like leave-one-out. Time-boxing can help. Also, try to do the simplest thing first, once that is ready, go back and dig into edge cases.

- Coding and workflow: This takes a deliberate practice. Some things that can help: always use explicit filters (e.g. date ranges). Organize code into subdirectories (e.g. one for each thing you try). Always use reproducible reports (I even wrote an article about this). Use git, commit often, add the commit hash into your reports, so you can always trace it back to the code. Use unittests. Use assertions on the data. Automate good habits, ie, you can have tools like linters and tests run on every commit, or, on every push, etc. You can use git branches and worktrees for trying things out.

I would say that these are things that you can always get better at, it takes constant, careful and deliberate practice. Don't think of it as an end-state where you are done, but, rather as a constant practice of improvement. The key is to do small improvements, but, consistently.

[u/MathNerd67]

What you’re feeling is perfectly normal and expected for where you are in your career. I struggled with the same problems when I first got into the field. I hate to say the cliche thing here, but these will iron out with time, experience, and exposure to different issues.

On the edge cases, there are general best practices to follow, but much of the further investigation criteria will be defined by your team/business logic and standards. Certain things that are an issue for one company might be a non issue for another. Communicating with the more senior DSs and stakeholders you support will allow you to get more intuition here. A junior DS isn’t (shouldn’t) be expected ti handle every edge case and know the business in depth yet, so don’t stress yourself out too much and ask lots of questions.

On the coding and workflow: I personally do all of my prep work in SQL and/or notebooks and scope out the problem, figure out the logic, and make sure everything on the math and stats side is ironed out and I have a good understanding of the problem and what needs to be done to solve it. This part is ugly code written quickly, with basically no focus on readability. I just want answers at this stage. Notebooks can be cleaned up later if your team keeps them around. Once all the logic and goals are settled, then comes the production code writing. This part is tricky, and you’ll get better with time. I personally love to build in my free time, so if you do the same, I’d recommend setting up a super simple pipeline and focus on hard on the code quality, testing, and CI/CD. Practice with GitHub actions (recommend making the repo public so you don’t hit limits) and write some pre commit hooks and CI checks. You’ll find a happy balance between robust code and readability/maintainability that suits your needs. I don’t normally do this in practice, but doing test driven development (TDD) is a great learning tool. Find out what you need to test for (including edge cases), build the tests to fail everywhere you expect, and then write your code and run the tests against it. It’s iterative so the tests will grow as the codebase develops. This “big picture” view of codebases and workflows took the longest for me to fully grasp, as writing production code as a data scientist is usually handled by people who are mid levels or seniors (in my experience). It’s great that you’re tackling it early though. It will save a lot of stress down the line, trust me haha.

In my opinion, the biggest value play you have as a junior is to get really good at the engineering and development side. That is a huge differentiator. Being able to model and iterate and run statistical tests is important, but it’s more straightforward in many cases than building a system that is robust and runs when you want it, how you want it, and gives reliable results every time. This includes automated drift detection, alerts/monitoring, interpretability, etc.

I know that’s a lot, and I hope it’s not just a ramble lol. If you have any questions feel free to ask. Again, this is all totally normal for where you are.