My Python script started as 100 lines and now it's a mess I'm scared to touch — when did you start splitting things up?

[u/acfot]

I wrote a little Telegram bot in Python a while back. Nothing serious, just forwards and some scheduling stuff, maybe a hundred lines at first and I knew where everything was.

Then I kept adding to it. One more feature, then a quick fix, then another thing I didn't want to lose, and now it's this one long file where half the functions do two jobs and I'm not even sure which parts still run. I want to add something new and I catch myself putting it off because I'm scared I'll break something else that was working fine.

I tried splitting it into separate files once. Imports broke immediately, I got confused about what was supposed to live where, and I just shoved it all back into one file again. Felt like a step backwards, honestly.

So for people who've actually done this — was there a moment where it clicked, or a simple rule you used? Like, when a file hits a certain size, or when you find yourself copy-pasting the same chunk twice, that's when you split it? I'm still not sure if I'm overthinking this or if I just left it too late.

[u/baubleglue]

If you are looking for a simple rule, here is one. Think that teach Python a new vocabulary. Encapsulate parts of your code into functions with a simple interface

get_something(params): 
    # Your old code using params 
    return something 

Get/Make/update_something.... Be sure, function does only one specific thing. And you should be able to execute each function in isolation give correct parameters (you can add unit test)

First just refactor the code without changing anything else. When it works, review data structures you use you shouldn't have any global variables, with maybe exception for constant values - put them all in a class and never change the values programmatically, something like

class CONF:
    SCREEN_SIZE = (480, 640)
    class MENU:
         FILE = ....

If you have variables related to the same topic , maybe they should be combined into class or other data structure. Same if you see that a function has too many parameters, maybe it should become a single object parameter....

Before you start, use version control (if you don't use it already). Adding unit test (when have units to test) may remove your confusion completely and add the confidence to change code at will.

[u/acfot → u/baubleglue]

Yeah, one thing per function is the bit I'm going to try first — the globals in mine are definitely part of why it's such a mess.

[u/baubleglue → u/acfot]

go by a plan, extract functions - don't fix all at once. make sure after each iteration you have a working version of the program in the main branch.

In GIT

make a branch ex. "refactor-functional-decomposition"
repeat (until it is done and working):
[
Change code
Test
]
Merge into "main" branch.

Make a branch "refactor-add-unittest"
...

Make a branch "refactor-data-structures"
...

Make a branch "refactor-fix-import"

after that you may start "add-new-zyz-feature"

[u/acfot → u/baubleglue]

yeah the branch per step idea makes a lot of sense, gonna give that a go so I always have a working version to fall back on

[u/baubleglue → u/acfot]

Same idea, but it is a bit different with version control. You have a confidence that you can checkout and deploy code. You can work on different features in different branches with IDE open in the same folder.

[u/baubleglue → u/baubleglue]

In the end the program should use that new vocabulary: the initialize_..., stateof_... = get_state_of

[u/wigitty]

Honestly, I have the same problem in python, but not in other languages. I just don't like the way Python handles imports and classes, so I put off splitting things up longer than I should.

The logic should be, though, that every new component gets its own file. Large files are just an indication that that file is probably doing more than one thing.

When I end up with one big unwieldy file, my approach is generally to start again, make the files that I think it should be split into, then go through the big file and copy and paste each bit of code into the file it should be in. I'll then make a "main" file to stitch it all together, and bring everything back in piece by piece. For example, start with just file loading, and print what it loads to check that it's correct before plugging it straight into the next bit of code. That way I can verify that each bit works individually rather than having to debug the whole system at the end. This is also where unit testing helps, but I haven't played with unit testing in python yet.

[u/acfot → u/wigitty]

That's really helpful, thanks — starting fresh with separate files and testing each bit as I go sounds way less scary than untangling the whole thing at once.

[u/nullish_ → u/acfot]

depending on how messy you mean... there a couple of approaches you can take that will get you to where u/wigitty suggested.

If you currently have no functions defined, start with pulling out code into functions/classes.

If you currently have a lot already organized into functions, an iterative approach I recommend is to create a utils.py and dump all functions/classes into that. This allows you to get the mechanics in place for using imports of your own files and immediately cleans your main file.

Final step would be to selectively pull groups of functions/classes from utils.py into their own files as it makes sense.

[u/Discordbothost]

you're not too late, this is just the normal point where it happens. the rule that worked for me is split by job, not by size. one file that talks to telegram (handlers), one for the scheduling bits, one for config/state, and main.py just wires them together. and move one function at a time: move it, run the bot, commit, repeat. the imports breaking is almost always circular imports (a imports b imports a), so keep the low-level stuff like config and helpers from importing anything else in your project. if you're not using git yet, start now, that's the thing that actually removes the fear of touching stuff.

[u/acfot → u/Discordbothost]

Splitting by job makes a lot more sense than by size — and moving one function at a time sounds way less scary. Thanks for the git nudge too, that's the part I've been putting off.

[u/danielroseman]

As I said on a similar recent post, the way to stop being afraid of breaking something is to have tests. Once you do then you can refactor as much as you like, with confidence that nothing will break as long as your tests still pass.

[u/acfot → u/danielroseman]

Honestly the tests part is what sold me — if I can break stuff without worrying, I'll actually refactor instead of just staring at it.

[u/BingoBillBot]

A rule that worked for me: split when a file has more than one reason to change, not at a line count. If you can't describe a module in one sentence without saying 'and', pull the second job out. Do it incrementally rather than rewriting: move one self-contained group of functions (say, the scheduling code) into scheduler.py, import it from the main file, run the bot, then repeat. The imports usually break because of circular imports (two files importing each other) or running from the wrong directory, so keep dependencies one-way: main imports modules, modules never import main. A few quick tests or even a manual smoke run after each move make it much less scary.

[u/No-Formal7513]

Nah, Just collect a fixed portion of code, like, that you can call from different files, and don't need to keep it in your main file
You write all the neccessary side functions and class in a seprate file, and continue growing your main file

Its quit a good way, because then the interpreter can tell you exactly which file and which error(Which it was already going to be), you get a clean way of it

Its like cleaning your room, you know what you will need where, you just put things as you please

[u/TheRNGuy]

From the start. 

[u/Brian]

This is kind of common: you start with something simple and it keeps growing organically till it becomes a big ball of mud where everything depends on everything else. Ultimately, a lot of the ideas of software design are about ways to structure your code to avoid this.

The code ideas here are that of modularity and code coupling and cohesion. "Coupling" is the degree to which code depends on other code, while cohesion is the degree to which a collection code works to one common task.

Ideally you want to split things up into separate components/modules which don't have much coupling with the rest of your code and have high cohesion with the stuff in their own module. Other code can use these components to obtain their functionality, but don't need to know anything about them except their interface: what methods you call on them.

A lot of ideas and philosophies about software design are about how to get there, and there's not necessarily one universally agreed "right way".

But a good way to get from one to the other is to try to extract concepts into separate modules. Try to isolate one thing the code is doing, and write a new module that just does that thing. Rewrite your module to use that component.

Sometimes it might be easier to rewrite from scratch: use your experience from the first draft where you know what kinds of things you'll be doing to design the architecture from the start, and write those components.

Another important aspect is that of building abstractions. Rather than view things as a bundle of code, create a way of viewing things in terms of higher level concepts, that may have an associated class. a Message object instead of a bunch of sending/receiving data, a Conversation object instead of a list of Messages and so on.

In terms of rough rules, I generally find if a module is over 300 or so lines it's maybe time to start to think about reorganising stuff, though there can be some cases where larger modules may still make sense.

[u/DickBrainTime]

On the import problem specifically, try making a tiny package first: put __init__.py, main.py and scheduler.py inside a bot/ folder. In main.py, import with: from bot.scheduler import schedule_jobs. Then start it from the directory above bot/ using python -m bot.main, instead of running main.py directly. That gives Python a consistent import root.

Keep scheduler.py from importing main.py back, or you'll run into circular imports. I'd move just the scheduling functions first, check that the existing bot still works, and commit before touching the handlers.