fix: back off exponentially when retrying rate-limited Discogs cover fetch - #498
Open
yottanami wants to merge 1 commit into
Open
fix: back off exponentially when retrying rate-limited Discogs cover fetch#498yottanami wants to merge 1 commit into
yottanami wants to merge 1 commit into
Conversation
…fetch AttachCoverImageFromDiscogsJob retried Integrations::Service::TooManyRequests with a flat 1-minute wait and unlimited attempts. Discogs rate-limits to 60 authenticated requests/minute, so for a library with many queued jobs (a 1600-album library queues ~1540 jobs after the first batch), every rate-limited job retries again after exactly 60 seconds, keeping the request rate pinned at the limit indefinitely instead of letting it drain. Use Rails' built-in :polynomially_longer backoff instead of a flat wait, so retry spacing grows with the number of attempts and the queue actually drains. Closes blackcandy-org#414
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the flat
wait: 1.minuteonAttachCoverImageFromDiscogsJob's rate-limit retry with Rails' built-inwait: :polynomially_longerbackoff.Why
As reported in #414: Discogs rate-limits to 60 authenticated requests/minute. With a large library, hundreds/thousands of jobs get queued after the first batch, and every one of them independently retries after exactly 60 seconds. That keeps the request rate pinned at the limit forever instead of letting it drain —
attempts: :unlimitedstays unlimited, but now the wait between retries grows with the number of attempts instead of staying flat.How this was tested
I don't have a Ruby/Rails environment in my sandbox, so I could not run
rails test:allorrails lint:alllocally. This is a one-line change to aretry_onkeyword argument — no new logic. The existing test (test/jobs/attach_cover_image_from_discogs_job_test.rb, "should retry the job when api request has been rate limited") already exercises the retry-on-rate-limit path and doesn't assert on the wait duration, so it should be unaffected by this change, but I'd appreciate CI/a maintainer confirming.Closes #414
AI disclosure: This PR (diff, commit message, and this description) was authored by an AI coding agent (Claude). I found no CONTRIBUTING.md or AI-usage policy in this repo to check against.