feat(reliability): bounded provider retry + circuit breaker (folds #928)#933
Open
seonghobae wants to merge 2 commits into
Open
feat(reliability): bounded provider retry + circuit breaker (folds #928)#933seonghobae wants to merge 2 commits into
seonghobae wants to merge 2 commits into
Conversation
… failures Request-path provider calls previously had zero retry: one flaky connection, timeout, 429, or provider 5xx became a user-facing failure (search embedding, summary extraction, translation, reply drafting). Adds services/retry.py — a stdlib-only retry_transient helper (no new dependency; requirements are hash-pinned): retries ONLY transient errors (APIConnectionError, APITimeoutError, RateLimitError, InternalServerError) with exponential backoff + jitter, capped at 2 retries / 8s delay. Auth and 4xx validation errors propagate immediately, preserving fail-closed behavior. Wired into the four provider call sites (embeddings.create, extraction parse, translation, reply drafting); error mapping and client lifecycle unchanged. Tests: tests/test_retry.py (5) — success passthrough, transient-then-success with exponential delays, budget exhaustion re-raises original error, non-transient fails immediately, custom retryable tuple. Regressions green locally: llm_service/embedding/search/llm_api/import_fixtures (92 passed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
…n retry) Retry handles transient blips; nothing stopped the backend from hammering a provider that is hard-down — every user request stacked full retry+timeout latency before failing. - services/circuit_breaker.py: per-key (provider base URL) three-state breaker. Opens after N consecutive failures (default 5), fails fast with CircuitOpenError during a cooldown (default 30s), allows a single half-open probe, closes on probe success / reopens on probe failure. Injectable clock for tests. In-process state only (ceiling documented in-line). - Wired around the retry wrapper at all four provider call sites (embeddings, extraction, translation, drafting): a failure only counts against the breaker after the retry budget is exhausted. Tests (6): pass-through, opens-after-threshold + fail-fast without invoking the operation, half-open probe closes on success, probe failure reopens, per-key isolation, success resets the failure streak. Regressions green (91: retry/llm_service/embedding/search/llm_api). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
seonghobae
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gap
#928's retry handles transient blips, but nothing stopped the backend from hammering a hard-down provider — every user request stacked the full retry+timeout latency before failing (reliability audit follow-up).
What
services/circuit_breaker.py: per-key (provider base URL) three-state breaker — opens after 5 consecutive failures, fails fast withCircuitOpenErrorduring a 30s cooldown, single half-open probe, closes on success / reopens on failure. Injectable clock; in-process state only (ceiling documented — upgrade to shared store if per-replica probing matters at fleet scale).Verification (local)
6 new tests: pass-through · opens-after-threshold + fail-fast (operation not invoked) · half-open probe success closes · probe failure reopens · per-key isolation · success resets streak. Regressions 91 passed.
🤖 Generated with Claude Code