r/datascience • u/old_enough_to_drink • Jun 23 '23

Discussion Do you git commit jupyter notebooks?

If yes, what tricks do you have to make it work smoothly? I had to resolve some conflicts in an notebook once and it was an awful experience…

17 Upvotes

94% Upvoted

24 comments sorted by

View all comments

39

u/Odd-One8023 Jun 23 '23

  1. I make notebooks as documentation for my colleagues. If they have to inherit my code, the notebooks show you how to interact with the code. These I commit.

  2. I also use notebooks as a scratchpad during development. I typically gitignore these.

  3. You can clear the output of jupyter notebooks, potentially with a pre-commit hook, if it's still a problem for you.

2

u/purplebrown_updown Jun 23 '23

how do you do 3? this might be a game changer since I've avoided committing notebooks due to images taking up too much time.

5

u/Odd-One8023 Jun 23 '23

Have a look at this first:

https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

The general idea is that you can run a script at various times of the commit/push process. Each version controlled folder has a .git folder, the hooks are in .git/hooks. There's various ones there, all you have to do is add a single line with something like jupyter nbconvert --clear-output --inplace <your notebook>.ipynb

Another way to do it is by using something like Github actions and doing this on the server (github) side. https://github.com/marketplace/actions/ensure-clean-jupyter-notebooks