feat(ethflow-watcher): cap backoff: retries (COW-1083)#59
Closed
brunota20 wants to merge 1 commit into
Closed
Conversation
…OW-1083)
The strategy's `apply_submit_retry` previously wrote an empty
`backoff:{uid}` marker on every retriable submit failure (including
the `TryNextBlock` fallback for unparseable orderbook envelopes). The
marker was a presence flag with no payload, so on every supervisor
reconnect / engine restart the same dead placement would retry
indefinitely — bounded only by log re-delivery frequency.
This change persists a per-UID retry count in the marker's value
(ASCII `u32`) and upgrades to `dropped:` after `MAX_BACKOFF_RETRIES =
5` consecutive retries. The upgrade emits a Warn-level log line so
the operator sees the structural issue (flaky CDN, indexer hiccup,
poisoned envelope) rather than silently accumulating retries.
## Changes
- `modules/ethflow-watcher/src/strategy.rs`:
- New const `MAX_BACKOFF_RETRIES = 5`.
- New helper `read_backoff_count` that reads + parses the marker
payload; pre-COW-1083 empty markers decode to 0 so previously-set
backoff: rows still get a fresh attempt (no premature drop on
rollout).
- `apply_submit_retry`'s retriable branch now reads the prior
count, increments, and either writes the new count or upgrades
to `dropped:` (clearing the stale `backoff:`) at the cap.
- Cap-upgrade log line carries the retry-count and message: "...
after 5 retries on transient/unparseable rejection ...".
## Tests
- 19/19 ethflow-watcher tests pass.
- New `submit_transient_error_at_cap_upgrades_to_dropped_warn`:
seeds `backoff:{uid} = "4"`, triggers a `data: None` rejection
(the unparseable case the issue names explicitly), asserts:
* `dropped:{uid}` is now set
* `backoff:{uid}` is cleared (single outcome marker at rest)
* exactly one Warn log line containing "ethflow dropped" +
"retries"
- New `submit_transient_error_with_legacy_empty_marker_resets_counter`:
backwards-compat — a pre-COW-1083 empty `b""` marker is treated
as count 0, bumped to "1" on first retry rather than prematurely
dropping. Protects in-flight backoffs across the rollout.
- Existing `submit_transient_error_writes_backoff_marker_and_returns`
extended with an assertion that the first retry persists
`backoff:{uid} = "1"`.
- `cargo clippy -p ethflow-watcher --all-targets -- -D warnings`
clean.
## Why this is M4
Surfaced by jeffersonBastos's PR #55 (M3 mirror) review, thread on
`crates/shepherd-sdk/src/cow/error.rs:82`. Latent in normal
operation (the host forwards parseable envelopes after COW-1075, so
`classify_api_error` returns `Drop` for permanent rejections), but
the gap fires when the orderbook returns a non-JSON 4xx body
(e.g. an HTML error page from a CDN) or if a future host change
accidentally drops the envelope again. Bounded retry semantics
close the latent risk without changing the safe-default
classification (still `TryNextBlock` on `None` data — that part is
explicitly out of scope per the issue).
AI-assisted authoring with Claude (Opus 4.7); reviewed end-to-end
and validated against the existing ethflow-watcher strategy tests
before push.
This was referenced Jun 22, 2026
Author
|
Closing as superseded by the observe-not-submit redesign in PR #62 (M2 layer) + PR #63 (BLEU-855 / M3-M4 layer). This PR caps the Background on the redesign and the empirical evidence (2026-06-22 Sepolia soak: 3 EthFlow placements No code from this PR is retained. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
apply_submit_retrypreviously wrote an emptybackoff:{uid}presence flag on every retriable submit failure — including theTryNextBlockfallback for unparseable orderbook envelopes (the case the issue names explicitly). With no payload, the same dead placement would retry indefinitely on every log re-delivery, bounded only by re-delivery frequency. This change persists a per-UID retry count in the marker's value (ASCIIu32) and upgrades todropped:with a Warn log afterMAX_BACKOFF_RETRIES = 5consecutive retries.b""markers decode to count 0 so previously-set backoffs get a fresh attempt instead of being prematurely dropped on rollout. Garbage marker values reset to 0 (defensive against manual store edits).TryNextBlockonNonedata stays correct for transient flakiness) — that part is explicitly out of scope per the issue. This is bounded retry semantics on top of the existing classification.Why this is M4
Surfaced by jeffersonBastos's PR #55 (M3 mirror) review, thread on
crates/shepherd-sdk/src/cow/error.rs:82. Latent in normal operation (host forwards parseable envelopes after COW-1075, soclassify_api_errorreturnsDropfor permanent rejections), but the gap fires when the orderbook returns a non-JSON 4xx body (e.g. an HTML error page from a CDN) or if a future host change accidentally drops the envelope again.Tests
submit_transient_error_at_cap_upgrades_to_dropped_warn— Nth retry of an unparseable rejection upgrades to dropped: + Warn logsubmit_transient_error_with_legacy_empty_marker_resets_counter— pre-COW-1083 empty marker bumps to "1", no premature dropcargo clippy -p ethflow-watcher --all-targets -- -D warningscleanStacked on
PR #58 (
feat/chain-forward-revert-data-cow-1082) → PR #57 → upstream M4 epic (TBD).AI-assistance disclosure
Authored with Claude (Opus 4.7); reviewed end-to-end and validated against the existing ethflow-watcher strategy tests before push.