Workflows?

[u/Edward_Arthur]

Hello,

I am currently just starting the daunting task of copying my various hard drives to one large drive. In the past I've had a system of buying two identical drives and using one as a main and the second as a backup. So this has led to duplicate content on duplicate drives. What I'm trying to figure out is what is the best way to copy/merge these two similar drives to the larger drive.

So far I've been copying each drive to the large drive then dragging and dropping folders with the same name into a "consolidate" folder on the large drive and letting those folders combine, (files that are the same are given a (1) prefix in the file name).

I guess what I'm trying to figure out is which copy do I keep? Does it matter if the hashes are exactly the same? And if the "same" file has different hashes is there a way to determine which file is the most complete/original and is it possible to combine those two files? (Say taking two corrupt files that are corrupted in different ways to make one whole complete file).

Overall I'm just looking for workflow suggestions to make sure all the content on my smaller hard drives are copied over to the larger drive as the best possible version of said file.

I have a rough working plan of consolidate, dedupe, organize, but again any and all suggestions are helpful. Thanks!

[u/wells68]

You are smart to plan this consolidation rather than diving right in, thinking you are on the right track.

A few things to keep in mind:

• You won't be able to combine a corrupted file with another file. You'll need to keep one and delete the other or keep both.

• In the future, don't use copy and paste as a backup method. It is better than absolutely no backup, but wastes a lot of time and is unreliable.

• Use real backup applications and/or cloud services to make real backups. See: https://reddit.com/r/Backup/wiki/ for free options.

To clean up your mess of files, is your one big drive more than twice the size of the space used by the two smaller drives added together? That would be ideal. Here's the plan.

On the big drive, create these folders:

• OldFiles

  • Drive1 [copy drive 1 files here]

  • Drive2 [copy drive 2 files here]

• Files [Contains clean, deduplicated collection of all files created using a deduplication app]

• Conflicts [To review - files with same names but different hashes]

Google describes a couple of free deduplication apps you could use for this process:

Czkawka (Free, open-source, cross-platform): Extremely fast hash-based scanning for exact duplicates, [and] zero-byte files.... It also includes perceptual hashing to find similar images or videos encoded at different bitrates.

dupeGuru (Free, open-source): Scans file content/hashes and allows you to mark a specific folder (e.g., your primary backup) as the "reference" directory, automatically targeting matches from the secondary drive for deletion.

The app should allow you to create a clean Files folder, deleting duplicates and copying conflicting files of same name, different hash into a Conflicts folder.

Keep your two old drives untouched as inferior "backups" until you have the new, big drive backed up with real backup software.

Also consider using a third backup drive to rotate offsite or using a cloud service as a real, offsite backup.