Description
HorizonPayments::payments_after in crates/ingest/src/horizon.rs treats any non-2xx, non-404 response (including a 429 Too Many Requests, which public Horizon instances issue under load) identically to a genuine server error — both map to HorizonError::Request, and Ingestor::run's retry loop just waits the normal poll interval before trying again with no awareness that the previous failure was specifically a rate limit that Horizon expects callers to back off from (often advertised via a Retry-After header).
Requirements and Context
- Add a
HorizonError::RateLimited { retry_after: Option<Duration> } variant, populated by reading the Retry-After header (seconds, per HTTP spec) when Horizon returns 429.
- In
Ingestor::run's poll loop, when poll_once fails with IngestError::Horizon wrapping a rate-limit specifically, sleep for the advertised retry_after (or a sane default backoff like 5s if the header is absent/unparseable) instead of just falling through to the normal poll interval — this may require IngestError to carry enough information to distinguish this case, or for the check to happen inside run directly against a richer return type.
Suggested Execution
Branch: fix/ingest/handle-horizon-429-backoff
Implement Changes
- Update
HorizonError in crates/ingest/src/horizon.rs and the response-status handling in payments_after.
- Update
IngestError in crates/ingest/src/lib.rs if needed to propagate the distinction, and update Ingestor::run's loop accordingly.
Test and Commit
payments_after_maps_429_to_rate_limited_with_retry_after_header.payments_after_defaults_retry_after_when_header_is_absent_or_unparseable.run_backs_off_by_the_advertised_duration_on_rate_limit (using a mock Horizon server, asserting the loop's next request happens no sooner than the advertised backoff).- Run
cargo test -p octo-ingest locally before committing.
Example Commit Message
fix(ingest): back off on Horizon 429 responses using the advertised Retry-After
A Horizon rate-limit response was previously indistinguishable from any
other server error, so the ingest loop just retried at its normal poll
interval — the opposite of what a 429 asks callers to do. Adds a distinct
RateLimited variant honoring Retry-After, with a sane default backoff when
the header is missing.
Guidelines
- Use a mock HTTP server (see the ingest mock-server testing issue for the same pattern) to simulate the 429 +
Retry-After header rather than relying on the live public Horizon rate limit, which is not reliably triggerable in CI.
- Reference this issue with
Closes #<issue-number> in the PR description.
Description
HorizonPayments::payments_afterincrates/ingest/src/horizon.rstreats any non-2xx, non-404 response (including a 429 Too Many Requests, which public Horizon instances issue under load) identically to a genuine server error — both map toHorizonError::Request, andIngestor::run's retry loop just waits the normal pollintervalbefore trying again with no awareness that the previous failure was specifically a rate limit that Horizon expects callers to back off from (often advertised via aRetry-Afterheader).Requirements and Context
HorizonError::RateLimited { retry_after: Option<Duration> }variant, populated by reading theRetry-Afterheader (seconds, per HTTP spec) when Horizon returns 429.Ingestor::run's poll loop, whenpoll_oncefails withIngestError::Horizonwrapping a rate-limit specifically, sleep for the advertisedretry_after(or a sane default backoff like 5s if the header is absent/unparseable) instead of just falling through to the normal poll interval — this may requireIngestErrorto carry enough information to distinguish this case, or for the check to happen insiderundirectly against a richer return type.Suggested Execution
Branch:
fix/ingest/handle-horizon-429-backoffImplement Changes
HorizonErrorincrates/ingest/src/horizon.rsand the response-status handling inpayments_after.IngestErrorincrates/ingest/src/lib.rsif needed to propagate the distinction, and updateIngestor::run's loop accordingly.Test and Commit
payments_after_maps_429_to_rate_limited_with_retry_after_header.payments_after_defaults_retry_after_when_header_is_absent_or_unparseable.run_backs_off_by_the_advertised_duration_on_rate_limit(using a mock Horizon server, asserting the loop's next request happens no sooner than the advertised backoff).cargo test -p octo-ingestlocally before committing.Example Commit Message
Guidelines
Retry-Afterheader rather than relying on the live public Horizon rate limit, which is not reliably triggerable in CI.Closes #<issue-number>in the PR description.