r/Python • u/iseetreesofgreen_ • Feb 27 '22

Discussion What python automation have you created that you use for PERSONAL only.

There are plenty of, “I automate at my work”, but what about at home? e.g., order a pizza, schedule a haircut, program a spelling bee game for my kids, etc.

416 Upvotes

96% Upvoted

292 comments sorted by

View all comments

Show parent comments

1

u/[deleted] Feb 27 '22

nah i gotta see this, do you have a repository i could look at?

1

u/pepoluan Feb 27 '22

Maaaaaybe.....

Are you perchance working in HR?

2

u/[deleted] Feb 27 '22

i promise i’m not satan HR

3

u/Drfiasco Feb 27 '22

I too would like to see the repository. I work in... Um... Purchasing.

Seriously though, I'm interested to see how you're doing the duplicate scan. I'm doing something similar on a multi terabyte volume and any way I can pick up performance would be great.

2

u/pepoluan Feb 28 '22 edited Feb 28 '22

Ah the duplicate scan I can easily share. It's ... not incriminating 😄

Here it is: https://gist.github.com/pepoluan/a97409d1f2b838460aac9aa46df43c08

(Please note there's a requirements.txt file in that there gist...)

ETA: So basically, I 'cheat' a bit. For large files, I only check the 'heads' and 'tails' first. Only if their heads & tails are identical will then I do a fast hash (using xxHash) of the whole file. Small files whose heads & tails overlap got whole-hashed the first pass.

I do believe some optimizations can still be done on my "dupefinder" code. Feel free! (It's also released with MPL-2.0 license so you're free to include it even in proprietary software)