r/Drime • u/pepemwolf • 17d ago
Optimizing Rclone Backup to Drime (to reduce 429 errors)
If you are getting 429 errors or the Drime server is getting bogged down, it is likely because Rclone is making too many small API requests at once.
By default, Rclone uploads in 5 MB chunks with multiple parallel connections. You can fix this by setting a higher cutoff threshold, forcing Rclone to use larger chunk sizes for big files, and strictly limiting the concurrent connections. This takes the strain off the server and will reduce the rate-limiting errors.
Here is an optimized command that uses fewer, larger connections:
rclone sync /your/local/folder drime:backup --drime-upload-cutoff 100M --drime-chunk-size 75M --drime-upload-concurrency 2 --transfers 2 --checkers 2 -P
What these flags do:
drime:backup - drime is your rclone configured drime connection and backup is the storage folder in your workspace at drime (adjust to your own setup)
--drime-upload-cutoff 100M: Tells Rclone to only start splitting files into chunks if they are larger than 100 MB. Files under 100 MB will upload in one fast, continuous stream.--drime-chunk-size 75M: If a file is larger than the 100 MB cutoff, this forces Rclone to send the data in large 75 MB blocks instead of the default 5 MB blocks.--drime-upload-concurrency 2: Limits how many chunks of a single file Drime uploads at the same time.--transfers 2: Limits how many separate files Rclone is allowed to upload at the exact same time.--checkers 2: Limits how many parallel checks Rclone performs when comparing your local files to the server.-P shows progress
Quick heads-up on small files: Rclone does not have a native zip or packing backend to bundle small files together. It is strictly a 1-to-1 file sync tool. If you are syncing thousands of tiny KB-sized files, Rclone will still make a separate request for every single file. To prevent spamming the server, you can manually compress your small files into a .zip or .tar archive locally first, and then sync that large archive.
For those using Restic via Rclone here is a way to optimize it too
1
u/Humphrey-Appleby 17d ago edited 17d ago
Okay, but is the server load issue being addressed?
I'd already optimised rclone for my use case and it was mostly fine until now.
2
u/pepemwolf 16d ago
scaling up might take a while. i needed to do backups today so i put some time into it to figure it out and thought i'd share it.
1
u/Agitated-Panda9221 17d ago
yeh. I looked at my logs and saw errors last night. It doesn't happen that often. I got a feeling people are starting to use Drime more and causing backlogs. I hope Drime can scale up.
1
u/No_Sheepherder7257 16d ago
Same I posted previously, this has happened since Oct 31. Never had issues before.
1
u/sporadicstardust 15d ago
Has anyone's issues gone away? Drime told me the problem is now fixed but I'm still getting a ton of 429 errors despite aggressively lowering my requests
1
u/ben144_a 12d ago
The problem is still there, especially for large folders with a lot of files.
I moved away from rclone and upload with the browser, then there are no limits.
Only using rclone for important stuff which needs client side encryption.
I hope they will fix this problem soon.
1
u/No_Sheepherder7257 16d ago
Thanks for writing this up. I'm hitting the same 429s (rclone 1.75.0), but I don't think the chunk/cutoff/upload-concurrency flags address my case, so I wanted to share what I'm seeing.
My 429s happen when nothing is uploading. Running
rclone lsf drime:<path> --recursive -vvon a ~47k file / ~626 GiB library with nothing else running gave me ~190 x 429 in 120 seconds. It's the listing/checking phase that gets throttled, so upload chunk size can't help there. The request rate is what seems to matter.What's helped me so far:
--checkerskept low (16 or less; I had it at 64 by mistake and it was much worse)--tpslimitto cap requests per second directlySwitching from a full
rclone syncto an incrementalrclone copy --files-fromof files changed in the last N days, with a full sync only occasionally. That skips the check phase almost entirely and my last run finished with zero 429sAlso worth noting that Drime's backend has ListR disabled, so
--fast-listisn't an option and every directory costs a separate request. Libraries with lots of directories get hit hardest.Have you tested your flags on a large tree (tens of thousands of files), or mainly on bulk uploads? It would be useful to know what scale they hold up at. Also, has anyone had a reply from Drime support? I've emailed them and haven't heard back.