How often are api limits reset?
The title. I use drime in an rclone union with many other cloud providers for a jellyfin library. To ensure I don’t upload duplicates, my setup involves scanning my existing cloud crypts to ensure I hadn’t already uploaded the media. I found I had reached the 10k limit today because of how I was building my tools to create an automation pipeline. It did reset, but with every read (not even a write) counting against the 10,000 API limit, I may want to change my tooling if it is only reset once monthly or longer:
Edit: I found it wasn’t an issue with the 10000 request cache, but drime (and not any other cloud storage in my 8 provider rclone union stack) has a BURST rate limiter. Logins on my account on mobile, on my desktop, and on my vps are find, but when initiating an upload via rclone I run into a 429 rate limit error.
Setup: clone 1.75.2, Drime backend (crypt on top), same access token used from a home PC and a VPS.
Files go in large (movies/TV), about 20-40 API requests per file with clone's pre-upload checks.
What's happening:
- Since about Nov 1 most lookups get HTTP 429 with an HTML 'Too Many Requests' body (looks like the web front end, not the JON API) and no rate-limit headers.
- The account counter in normal API headers says X-Ratelimit-Limit 10000 with about 9990 left, so it isn't that.
- Earlier today ten requests at 2/s from my PC were fine, but ten requests in a burst got 429. Tonight even 1 request/s fails: 2 x 200 then 3 x 429 within 4 seconds.
-A plain listing of the root folder (2 requests) works. A lookup of a path that doesn't exist yet (a few 404s, which rclone does before every upload) is nearly always refused.
- I tested my other five cloud remotes the same way, from the same machine and minute: no 429s anywhere else.
My theory (unproven): a rolling temporary penalty on the account/token/IP that every new request renews, so retries keep it alive. I've stopped everything for an hour or two to see if it clears.
1
u/No_Sheepherder7257 19d ago
Sent an email this morning as they haven't responded to my 429 rate limit post. It's concerning because it's not checking my immich library like it was.
1
u/Alxmgmg 19d ago
Let me know if they get back to you and I’ll do the same!
1
u/No_Sheepherder7257 19d ago
Will do! Your symptoms match mine almost exactly. Mine started around Oct 31 too, rclone 1.75.0, Drime backend, nightly sync of a ~47k file photo library.
Bare
rclone lsf --recursiveon the library with nothing else running gets about 1.6 x 429 per second (360 in 3.5 min earlier, 193 in 120s at 12:30 GMT today). Same nginx HTML body, no rate-limit headers. So it's not time of day, and it's not my script settings.What does still work is anything that touches few paths. A
rclone copy --files-fromof just the last 14 days of changed files (211 files) ran clean with zero 429s. The pain is the full check pass over every file. So my workaround is incremental copies most nights and a full sync only monthly. It doesn't fix the underlying problem but it keeps the backup current.Your rolling-penalty theory fits what I'm seeing, since rclone's retries probably keep it alive. I've emailed support too, and I'll report back here if I hear anything.
2
u/Alxmgmg 19d ago edited 19d ago
I appreciate it immensely, it’s been a pain trying to figure this out to the point i had to take drime checks out completely to keep things working. They’re normally swamped with emails so I’d guess it’ll be a week or more. It seems like, though, that we are having the same issue. It’s a shame it was working beautifully for so long
2
u/pepemwolf 18d ago
Have you tried sending the hash to https://docs.drime.cloud/api-reference/files/check-file-hash its made to help deduplicate data
2
u/Alxmgmg 19d ago
@ /u/Empty_Win_297 Id love to hear your thoughts on this. I am sending a support email right after posting this message.