fix(llm): retry hollow responses instead of bisecting them (#2880) by rajarshidattapy · Pull Request #2888 · Graphify-Labs/graphify

rajarshidattapy · 2026-08-20T04:52:56Z

Fixes #2880 (asks 1 and 3; see Not included for 2 and 4).

The conflation

_response_is_hollow() detects two different failures, and all five _call_* sites relabelled both as finish_reason="length":

truncated — the model ran out of max_completion_tokens mid-JSON. Bisecting is the correct recovery: smaller input ⇒ shorter output ⇒ it fits.
hollow — HTTP 200 with empty/null/whitespace content, or content that parses to zero nodes and zero edges (a rate limit, a transport hiccup, a refusal, an agentic prose reply, a reasoning-first content block).

Bisecting a hollow response cannot converge — both halves are produced by the same misbehaving backend and come back hollow too — so every hollow response cost 2**max_retry_depth billed calls instead of one. At the default depth that is up to 15 calls per chunk, all failing, which matches the reporter's measured ~18x token blow-up for an identical 920-node/854-edge graph.

Ask 1 — hollow gets its own recovery

_mark_hollow() replaces the five copies of the relabel and sets finish_reason = "hollow", distinct from "length". _extract_with_adaptive_retry then:

retries the same chunk with backoff (2s, 8s — bounded, so at most 3 calls per chunk);
if it is still hollow, prints a loud failure naming the likely causes, marks the chunk's files partial so the next run re-dispatches them and they are not promoted to the semantic cache as authoritative, and returns.

Bisection still handles genuine finish_reason == "length" and context-window rejections — those are size problems, and that path is unchanged (covered by test_adaptive_retry_bisects_on_truncated_response).

Worst case per chunk: 15 calls → 3. A transient hollow that clears still recovers, in 2 calls rather than 3+.

The three stale comments the issue quoted (llm.py ~1510, ~1557, _bedrock_response_text ~1075) are updated to describe the new behaviour.

Ask 3 — expose the knob

GRAPHIFY_MAX_RETRY_DEPTH now overrides max_retry_depth, which was a Python-API kwarg only with no CLI flag, so a graphify extract operator could not lower it or set it to 0 as a mitigation. It follows the existing _resolve_max_retries / _resolve_api_timeout shape (garbage and negatives fall back to the default; 0 disables retries) and is documented in the README env table. Explicit Python-API callers still win — the resolution only fires when the kwarg is left as None.

Tests

test_adaptive_retry_does_not_bisect_a_hollow_response — asserts the sub-call chunk sizes are [4, 4, 4], i.e. no halving, and that the files come back marked partial. This is the regression that would catch a future re-conflation.
test_adaptive_retry_recovers_a_transient_hollow_response — 2 calls, all 4 nodes, nothing marked partial.
test_adaptive_retry_bisects_on_truncated_response — the repurposed end-to-end test, now driven by a real truncation, so the bisect path stays covered.
test_call_openai_compat_labels_*_hollow ×3 and test_call_openai_compat_keeps_real_truncation_as_length — the two labels stay distinct at the backend boundary.
test_max_retry_depth_reads_the_env_var.

Full suite: no new failures (the 13 that fail on this machine fail identically on a clean v8 — missing optional grammars and platform-specific install/watch tests).

Not included

Ask 2 (a sub-call budget and a global ceiling). With hollow off the bisect path, the remaining fan-out is driven only by genuine truncation, where each split does make the input smaller and 2**max_retry_depth is already a real bound — and that bound is now operator-controllable via ask 3. A run-level call/token ceiling that aborts is a new config surface with its own semantics (what to do with the partial graph), so it seemed better as its own change than smuggled into this fix.
Ask 4 (emit tokens: on abnormal termination). Separable observability work in cli.py, not part of this defect. Happy to open it separately.

…Labs#2880) `_response_is_hollow()` collapsed two very different failures into one, and every `_call_*` then relabelled the result as `finish_reason="length"` so adaptive retry bisected the chunk: - truncated — the model ran out of max_completion_tokens mid-JSON. Bisecting is the correct recovery: smaller input, shorter output. - hollow — HTTP 200 with empty/null/whitespace content, or content that parses to zero nodes and zero edges (a rate limit, a transport hiccup, a refusal, an agentic prose reply, a reasoning-first content block). Bisecting a hollow response cannot converge: both halves go to the same misbehaving backend and come back hollow too, so one bad response cost 2**max_retry_depth billed calls — up to 15 per chunk at the default depth, all of them failing. A nightly run measured ~18x the input tokens for an identical 920-node/854-edge graph. Label hollow distinctly instead. The five relabel sites collapse onto one `_mark_hollow()` helper, and `_extract_with_adaptive_retry` gives hollow its own recovery: retry the SAME chunk with backoff (2s, 8s), then fail the chunk loudly, marking its files partial so the next run re-dispatches them and they are not promoted to the semantic cache as authoritative. Worst case per chunk goes from 15 calls to 3. Bisection still handles genuine `finish_reason == "length"` and context-window rejections, which are real size problems. Also expose `GRAPHIFY_MAX_RETRY_DEPTH` (ask 3): max_retry_depth was a Python-API kwarg only, so a `graphify extract` operator had no way to lower it — or set it to 0 — as a mitigation.

graphify-labs

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.

Formal verification. 1 change(s) tested, no difference found (not proven).

Graphify review — findings

Replaces the "hollow response → relabel as length → bisect" recovery with a dedicated hollow finish reason that retries the same chunk with bounded backoff. Centralizes the per-backend hollow detection in a new _mark_hollow helper (used by openai-compat, claude, claude-cli, azure, bedrock) and adds _resolve_max_retry_depth honoring a new GRAPHIFY_MAX_RETRY_DEPTH env var (0 disables adaptive retry). Documents the knob in the README.

Worth a look

GRAPHIFY_MAX_RETRY_DEPTH=0 semantics claim to disable retries but code returns 0 depth which may not disable hollow same-chunk retries — graphify/llm.py:436 · Escalate · medium
- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review Execution auto-disposal is off for this run; enable it (with sandbox isolation) to have Graphify try to confirm or refute this automatically.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 782 functions depend on the 306 functions this change touches.

Health — this change adds coupling hotspots:

new: deduplicate_entities() — 63 callers, 21 callees
new: build_merge() — 46 callers, 14 callees
new: extract_files_direct() — 17 callers, 20 callees
new: extract_corpus_parallel() — 26 callers, 11 callees
new: _call_claude_cli() — 31 callers, 9 callees
new: dispatch_command() — 2 callers, 117 callees
new: _extract_with_adaptive_retry() — 20 callers, 10 callees
new: _call_llm() — 11 callers, 18 callees
…and 15 more — each is listed as a finding

Verification — 782 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

verification_scope: 513 function(s) in the blast radius were not formally verified this run

Formal verification

No difference found (not proven): No behavior difference found in \_bedrock\_response\_text (not a proof).

The verifier ran both versions of \_bedrock\_response\_text on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

Verification did not run: Verification did not run for \_call\_azure.

The verification could not execute (an environment/toolchain issue, not a statement about the code).

Guarantee: No guarantee, the check itself did not complete.

Note: Detail: harness produced no verdict (rc=124): timeout after 30s

Could not verify: Could not verify \_call\_bedrock.

The verifier did not have enough to check \_call\_bedrock, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 200 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly ParamValidationError — names the real obstacle, not a sampling gap)

Verification did not run: Verification did not run for \_call\_claude.

The verification could not execute (an environment/toolchain issue, not a statement about the code).

Guarantee: No guarantee, the check itself did not complete.

Note: Detail: harness produced no verdict (rc=124): timeout after 30s

Could not verify: Could not verify \_call\_claude\_cli.

The verifier did not have enough to check \_call\_claude\_cli, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 200 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly RuntimeError — names the real obstacle, not a sampling gap)

Verification did not run: Verification did not run for \_call\_openai\_compat.

The verification could not execute (an environment/toolchain issue, not a statement about the code).

Guarantee: No guarantee, the check itself did not complete.

Note: Detail: harness produced no verdict (rc=124): timeout after 30s

· 23 more finding(s) on lines outside this diff (see the check run).

Review catch: `0` was documented as disabling retries entirely, but it only bounded the bisection depth. A hollow response still took the same-chunk retry path, so a chunk cost 3 calls in a run whose operator had explicitly asked for none — the opposite of what the knob is set for, since ask 3 in the issue was a spend mitigation. Gate the hollow backoff on max_depth > 0, and report the attempt count that actually applied rather than a hardcoded 3. `0` now means one call per chunk, full stop; the chunk still fails loudly and its files are still marked for re-dispatch on the next run. Docstrings and the README row say exactly this now, and a test pins both sides: one call at depth 0, three at the default.

graphify-labs

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Formal verification. 1 change(s) tested, no difference found (not proven).

Graphify review — findings

Adds a GRAPHIFY_MAX_RETRY_DEPTH env knob (via _resolve_max_retry_depth) and splits hollow responses from truncation: a new _mark_hollow helper tags empty/zero-result HTTP 200s with finish_reason="hollow" across all backends (_call_openai_compat, _call_claude, _call_claude_cli, _call_azure, _call_bedrock) instead of relabeling them as "length". _extract_with_adaptive_retry now retries hollow chunks in place with bounded backoff (_HOLLOW_BACKOFF_S) rather than bisecting them, and 0 disables every retry path. README documents the new variable.

No blocking issues surfaced. 10 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 788 functions depend on the 312 functions this change touches.

Health — this change adds coupling hotspots:

new: deduplicate_entities() — 63 callers, 21 callees
new: build_merge() — 46 callers, 14 callees
new: extract_files_direct() — 17 callers, 20 callees
new: extract_corpus_parallel() — 26 callers, 11 callees
new: _call_claude_cli() — 31 callers, 9 callees
new: dispatch_command() — 2 callers, 117 callees
new: _extract_with_adaptive_retry() — 22 callers, 10 callees
new: _call_llm() — 11 callers, 18 callees
…and 15 more — each is listed as a finding

Verification — 788 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

verification_scope: 519 function(s) in the blast radius were not formally verified this run