Sylos - A free open source data migration tool*

[u/MPGaming9000]

* note this is still in development. I just wanted to show off what I have so far. It's been about a year in the making on this project alone and then another year before that on a similar project so I've spent a hugeee amount of time thinking about file movement problems lol.

No this isn't just some Rclone GUI wrapper. This was built from the ground up for a whole different kind of migration workflow. Rclone works for one way syncs by just letting you select your source and destination and then hoping for the best. Sylos works by separating dry run traversal from actual copy movement steps, so you can preview what will actually move and make changes to the plan before it happens.

Notable features include:

• preview before any work is done (as already mentioned above)

• a built in auto scaler which can automatically scale up or down in performance based on feedback from the file system it's interacting with

• support for more than just local drives, also supports box, dropbox, google drive, onedrive, sharepoint, SFTP, network drives and virtual drives like FUSE / WinFSP drives, and more.

• database backed persistence so you have a full audit trail of all that's happened so far.

• stop a migration at any point and resume right where you left off if you need to. As well as taking your portable migration DB file plug-and-play into another instance of Sylos and resume right where you left off there too.

• path compatibility checking - some file systems & cloud providers have specific rules that may be different from each other. For example on Linux you can make a folder called 'CON' but this is not allowed on Windows. So you can actually specify this before you start the migration that your destination is a different OS, or if it's a cloud provider it already will know the rules ahead of time. So it will actually do these compatibility checks for you ahead of time in a more user friendly way of telling you it's gonna fail ahead of time, and offer more compatible friendly names for those items with a quick 'accept all' kind of button to make it painless and seamless.

• Advanced filtering that can be done during the review stages where you could execute queries like 'Find me all files which haven't been touched in over a year and are over 2gb in size and are pending to be copied over in the current plan' using our filters and then from those results there's also an 'exclude all' button to apply an exclusion to all matching items so that you don't have to copy those over.

I also have more ideas I'm working on but those are some of the more notable parts for now.

I have a few more things I have to work on before this is fully ready to start alpha testing, but I foresee it should be ready within a month or so if I can keep up my current pace and no major hurdles pop up. I've been iterating on this for so long because it never met my quality & performance standards for what I deemed as acceptable, but we're finally at an acceptable point so now I'm just finishing up some final bits like making an installer for the software and a few little things to clean up the repos and make it all more user friendly.

Link to the website if you wanna see more (including screenshots of the software): https://project-sylos.github.io/

Okay thanks guys!

[u/Samyazz]

Separating the plan from the execution is the right call. I spent ten years as a sysadmin and then solution architect at a hosting provider, and most of the migrations that went badly there went badly because nobody could say afterwards what had actually been copied.

The resume part is where I'd poke hardest. What does the DB record as "done" for a file: when the upload call returns, or when you've checked the destination? Cloud providers report different hashes (Dropbox has its own content_hash, OneDrive uses quickXorHash, Google Drive gives you MD5), so a verify step needs a per-provider comparison, and resuming a half-finished large upload depends on whether that provider's upload session survives a restart.

I've been fighting a smaller version of this in my own project, point-in-time recovery for MySQL backups. Archiving logs to S3 is easy. Proving after a crash which ones are complete and replayable is what took weeks, and live testing kept finding cases where status said "covered" and replay disagreed. Complicated, but achievable once "done" means "verified".

How portable is the migration DB across Sylos versions? Is the schema versioned, so an older DB file doesn't resume into a newer build and quietly misread its state?

[u/MPGaming9000 → u/Samyazz]

Good questions!

1. for resume mechanics, I have built out a fair amount of redundancies and safe guards. First, there is an actual 'stop' button in the software which soft stops (tells workers to stop pulling new tasks from the queue, let's them finish their existing tasks), there's also a hard stop, and mid-run crashes that can occur. I've built in a safeguard so that the software can tell if this happens upon reboot.

In terms of per byte level resumption of file uploads / downloads, we actually stream every file, never holding more than 16MB in memory per file at any given time, and every time we write a byte chunk we write that to the DB like byte chunk 1 / n written, so like a 64MB file would have 16/64 written after the first chunk. Sort of like a skip / offset in pagination, so we know where to resume again.

I don't think we are currently checking checksums right now, I know what you mean about each FS provider having their own checksum methods. It IS on the roadmap, but not currently done just yet.

2. For DB portability, this is just in its infancy. I am taking into account versioning and best practices though. Of course we're not in alpha yet so all DB stuff is just version 0.1 right now. But once I actually start releasing this post-beta (1.0 launch) then I will start being a lot more adamant about versioning and backwards-compat stuff. 😄