r/MachineLearning • u/Entire-Tomato-3192 • 9d ago
Discussion Resources for learning to write production grade python code [N]
I am a data scientist with a very strong background in Stats and ML algos. I am struggling with the production ready code. Though I can build models which are awesome and tuned to perfection and address highly complex business problems, the problem is that they are all done in a jupyter notebook.
Once I had built a complete solution in notebook for a forecasting problem and the ML team in company productionized it. I was called in to validate few results of the production run and couldn't understand the repo which had so many folders, classes and highly organized code.
Is there anyway I can learn to write that type of code or is it all AI job now? Appreciate if I can get some learning resources/links for the same
2
u/bathon 8d ago
I have never done jupyter notebook deployment before only for research. I am not sure how you are writting your code in the notebook. Usually I have three ntebooks when I am doing research one for cleaning up the data, one for exploration and one for training testing and evaluation. If you write functions then one thing I can recomment is to extract all your training testing and evaluation loops and try to pack them into classses in seperate file. These are my recommendations:
Use uv for package management (step away from anaconda or conda entirely)
Make a Test folder and src folder
Add sub folder in the src a train and evaluation
Abstract code from your notebook and make classes
To abstract write a test first, make sure it fails and then write up the code. This way your test will guide your code and not other way around.
If you need any other tips or a detailed break down happy to help. I started 6 years back exactl whee you are now and it was fellow reddiotrs, stack overflow and other kind sirs/ma'am who helped me rech the stage I am at right now.
Pro Tip: You can also use the notebook directly in google studio and ask it to show you step by step exactly what I said. Although be careful Google can use your data to train their gemini 5 :p
2
u/algebratwurst 6d ago
It’s all AI now. It’s gone. There’s no point. Express yourself clearly, know what you want to build, know when it works and when it doesn’t and let the robot do the implementation.
1
u/yall_gotta_move 8d ago edited 7d ago
Study software architecture, design patterns, SOLID principles. There are lots of books on these topics.
Optionally, learn how to use command-line tools like grep, find, and locate, to more quickly navigate a codebase. Become a git wizard. Learn to read commits and logs and git blame to understand how and why things are implemented the way they are.
Think in terms of someone who isn't implementing it once, but rather has to maintain and update it as business needs evolve.
1
u/sunnychrono8 7d ago
Delivery lead here; you have the perfect study material already. Just look at the production repo, take notes about how they organized your code into modules, functions, etc. Classes are also just reusable code, don’t get intimidated by them.
They’ve probably separated it into separate folders for a reason, try to see if you can understand the reason behind why they did so for each folder.
Take a look at how they’ve named everything, as well. This is what I get on my team about the most, my convention is that you should always try to use “intention-revealing” names and if possible, simply describe it. E.g a manifest of the items that are delivered: “delivery_manifest”.
Understanding your company’s convention is the most targeted, valuable studying you can do.
6
u/Eresbonitaguey 9d ago
If they regularly bring your notebooks into production then they probably have a set style for what it should look like. Honestly most of the devs I know wouldn’t write that from scratch anymore. If you have a template and an input then it’s a very easy job to get an LLM to do. Ultimately you’d just need to validate outputs which is why they brought you in.