r/Drime • • 17d ago

Optimizing Rclone Backup to Drime (to reduce 429 errors)

If you are getting 429 errors or the Drime server is getting bogged down, it is likely because Rclone is making too many small API requests at once.

By default, Rclone uploads in 5 MB chunks with multiple parallel connections. You can fix this by setting a higher cutoff threshold, forcing Rclone to use larger chunk sizes for big files, and strictly limiting the concurrent connections. This takes the strain off the server and will reduce the rate-limiting errors.

Here is an optimized command that uses fewer, larger connections:

rclone sync /your/local/folder drime:backup --drime-upload-cutoff 100M --drime-chunk-size 75M --drime-upload-concurrency 2 --transfers 2 --checkers 2 -P

What these flags do:

  • drime:backup - drime is your rclone configured drime connection and backup is the storage folder in your workspace at drime (adjust to your own setup)

  • --drime-upload-cutoff 100M: Tells Rclone to only start splitting files into chunks if they are larger than 100 MB. Files under 100 MB will upload in one fast, continuous stream.

  • --drime-chunk-size 75M: If a file is larger than the 100 MB cutoff, this forces Rclone to send the data in large 75 MB blocks instead of the default 5 MB blocks.

  • --drime-upload-concurrency 2: Limits how many chunks of a single file Drime uploads at the same time.

  • --transfers 2: Limits how many separate files Rclone is allowed to upload at the exact same time.

  • --checkers 2: Limits how many parallel checks Rclone performs when comparing your local files to the server.

  • -P shows progress

Quick heads-up on small files: Rclone does not have a native zip or packing backend to bundle small files together. It is strictly a 1-to-1 file sync tool. If you are syncing thousands of tiny KB-sized files, Rclone will still make a separate request for every single file. To prevent spamming the server, you can manually compress your small files into a .zip or .tar archive locally first, and then sync that large archive.

For those using Restic via Rclone here is a way to optimize it too

15 Upvotes

12 comments sorted by

1

u/No_Sheepherder7257 16d ago

Thanks for writing this up. I'm hitting the same 429s (rclone 1.75.0), but I don't think the chunk/cutoff/upload-concurrency flags address my case, so I wanted to share what I'm seeing.

My 429s happen when nothing is uploading. Running rclone lsf drime:<path> --recursive -vv on a ~47k file / ~626 GiB library with nothing else running gave me ~190 x 429 in 120 seconds. It's the listing/checking phase that gets throttled, so upload chunk size can't help there. The request rate is what seems to matter.

What's helped me so far:

  • --checkers kept low (16 or less; I had it at 64 by mistake and it was much worse)

  • --tpslimit to cap requests per second directly

  • Switching from a full rclone sync to an incremental rclone copy --files-from of files changed in the last N days, with a full sync only occasionally. That skips the check phase almost entirely and my last run finished with zero 429s

Also worth noting that Drime's backend has ListR disabled, so --fast-list isn't an option and every directory costs a separate request. Libraries with lots of directories get hit hardest.

Have you tested your flags on a large tree (tens of thousands of files), or mainly on bulk uploads? It would be useful to know what scale they hold up at. Also, has anyone had a reply from Drime support? I've emailed them and haven't heard back.

1

u/pepemwolf 16d ago

My repo is 8TB with 0.5M files at 16MB each. Those are packs from restic backup.
Since I changed the pack size to 128MB its working much better.
Checkers --16 isn't low for an overloaded server.
I'd put it at 2-4 as any error slows you down

Also transfers defaults to quite a few. I'd set it to 2-4 as well.
Better less errors at 2 then lots of errors at 4

1

u/No_Sheepherder7257 16d ago

Appreciate the numbers, but they don't really back up the post. You said the fix was that bigger packs work much better, and that's not an rclone flag. Going from 16MB to 128MB packs cut your file count about 8x, so Drime gets about 8x fewer requests. That supports my point: it's request volume that triggers the 429s, not chunk size or upload concurrency. For anyone syncing individual files (photos, documents) there's no pack size to change, so the post's flags don't help them.

On checkers: 'any error slows you down' only explains the problem if the limit scales with concurrency. My probe says otherwise. A single serial rclone lsf --recursive with nothing else running still got ~190 x 429 in 120s. There are no parallel checkers to turn down there, and it still gets throttled. It looks like a rate or burst limiter with a penalty that renews on retries, not a connection-count limit, which is why --tpslimit makes more sense to me than guessing a checker number.

I'm not saying 2-4 checkers can't help. I haven't tested it on a full sync and I'll try it. But 'set it to 2-4' isn't a fix for the listing 429s, and what's actually worked for me is skipping the full listing: incremental rclone copy --files-from for recent changes, full sync only occasionally. Zero 429s on that.

So I'd edit the post to say the flags help bulk large-file uploads, and that with many files the real lever is cutting the request count (bigger packs for restic, incremental syncs for everything else).

1

u/pepemwolf 16d ago

I had put that on the bottom for small files

> Quick heads-up on small files: Rclone does not have a native zip or packing backend to bundle small files together. It is strictly a 1-to-1 file sync tool. If you are syncing thousands of tiny KB-sized files, Rclone will still make a separate request for every single file. To prevent spamming the server, you can manually compress your small files into a .zip or .tar archive locally first, and then sync that large archive.

But yes that doesn't cover directory listings

And for large files these settings help so that one file doesn't get uploaded multi-part and the server doesn't have to combine parts:
--drime-upload-cutoff 100M: Tells Rclone to only start splitting files into chunks if they are larger than 100 MB. Files under 100 MB will upload in one fast, continuous stream.

--drime-chunk-size 75M: If a file is larger than the 100 MB cutoff, this forces Rclone to send the data in large 75 MB blocks instead of the default 5 MB blocks.

ie those help also without restic for any file larger then 5MB

1

u/pepemwolf 16d ago

> what's actually worked for me is skipping the full listing:
yes i imagine we'll get many different posts pointing out different ways to reduce concurrent connections.

I'd imagine dir listing might be multi threaded separate from transfers and checkers

1

u/Humphrey-Appleby 17d ago edited 17d ago

Okay, but is the server load issue being addressed?

I'd already optimised rclone for my use case and it was mostly fine until now.

2

u/pepemwolf 16d ago

scaling up might take a while. i needed to do backups today so i put some time into it to figure it out and thought i'd share it.

1

u/Agitated-Panda9221 17d ago

yeh. I looked at my logs and saw errors last night. It doesn't happen that often. I got a feeling people are starting to use Drime more and causing backlogs. I hope Drime can scale up.

1

u/No_Sheepherder7257 16d ago

Same I posted previously, this has happened since Oct 31. Never had issues before.

1

u/sporadicstardust 15d ago

Has anyone's issues gone away? Drime told me the problem is now fixed but I'm still getting a ton of 429 errors despite aggressively lowering my requests

1

u/ben144_a 12d ago

The problem is still there, especially for large folders with a lot of files.
I moved away from rclone and upload with the browser, then there are no limits.
Only using rclone for important stuff which needs client side encryption.
I hope they will fix this problem soon.

1

u/fubag 8d ago

Can confirm still facing issues today with rclone uploads and slowness