A reply that keeps repeating itself comes back cut instead of running to the token ceiling

[QuintinBotes]

Closes #1301.

Why

At temperature 0, a small local model sometimes falls into a repetition loop: the same statement row, written again and again until MAX_COMPLETION_TOKENS. On qwen3-8b that was about 3% of extraction calls, and each held a slot for about 1.8 hours. As discussed in #1301, the ceiling itself stays, because on endpoints that count reasoning inside max_tokens it is a thinking budget. So the stream is stopped once its tail repeats.

What changes

• repeats_at_tail checks whether the same span of 30 to 600 bytes fills the end of the reply five times in a row, and whether that run's shortest period is also at least 30 bytes. Every statement starts with a verbatim quote, and a quoted dot leader or a row of empty <td></td> cells repeats with a short period. A real loop repeats a whole row.
• chat_at_streaming runs the check after each SSE frame, once the answer has grown by another 256 bytes. It checks per frame because one network chunk can carry the whole loop.
• On a hit, the reply comes back with Reply::repeating set, kept separate from finish_reason, so hit_token_ceiling() still means the endpoint said so. The response stream is dropped, which closes the connection and stops the endpoint generating. The warning 回复在复读，按截断收下 is logged with the model, the call label and the length reached.
• extraction_open treats a repeating reply as cut. It keeps the complete statements before the loop and records "the reply started repeating itself; kept up to the last complete item", or "response started repeating itself and could not be parsed; this chunk did not enter the graph", under the existing truncated_reply and chunk_unextracted codes.
• Logs: the extraction parse-failure warning keeps its hit_token_ceiling field and adds repeating. The phrase and type alignment warnings, which already log finish_reason, log repeating too; a guard cut there now shows finish_reason: None.
• Docs: docs/design/extraction.md gets a paragraph, "Where a looping answer stops", next to "Where a long answer stops".
• Scope: the guard sits in chat_at_streaming, so every call that streams through chat_retrying_rate_limits_at gets it: extraction, errata, implication, phrase and type alignment, time resolution and the ontology agent. Only extraction records the cut as a drop. The raw and tool streams are untouched. This stays on the response-reading side, clear of Retry with max_completion_tokens / suppress temperature when the endpoint rejects them #1232's retry changes to the request.

Tests

Two tests in utopia-llm, using the crate's existing raw-socket stream server (an_http_response), so utopia-llm gains no new dependency:

• a_reply_that_keeps_repeating_comes_back_cut: 200 identical rows with no finish come back with repeating set, hit_token_ceiling() false, and fewer than 20 rows read. Without the guard the stream is Interrupted.
• a_quoted_dot_leader_comes_back_whole: a statement quoting a 200-dot table-of-contents leader comes back whole. Without the shortest-period check it is cut at the leader.

How it was checked

On an M1 Max (macOS), at the PR head on top of dev 177ad0b:

• cargo fmt --all --check: passes.
• cargo clippy --workspace --all-targets -- -D warnings: no issues.
• cargo test --workspace --no-fail-fast with UTOPIA_DATABASE_URL pointing at a database created and migrated for this branch alone: 1,611 passed, 0 failed, 6 ignored.

On the deployment that reported #1301, the rule without the shortest-period check ran in front of Ollama overnight. It cut 57 replies, all real loops judging by their tails. Five of them, loops of rows under 30 bytes and one run of spaces, would not be cut by this rule (details in the review reply).

Before review

• Every commit is signed off (git commit -s)
• cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass

[WaylandYang]

Thank you, the guard is in the right place: checking per frame is cheap (at most a few hundred short compares every 256 bytes), the first test fails without it, and it stays clear of #1232.

Two problems:

1. It cuts real answers. Every statement starts with a verbatim quote, and any run of 150 bytes or more with a short period also repeats with a span of 30 or more. Streaming extraction-shaped replies through the same check and cadence: a single quote of a table-of-contents line with a 200-dot leader cut the reply in 7 of 16 frame sizes, and an HTML row with 20 empty <td></td> cells cut it too; whether it trips depends only on where the 256-byte check lands, and everything after the quote is lost. A real loop repeats a whole row, so it's enough to count only a tail whose shortest period is at least 30 bytes, i.e. skip it when it also matches itself shifted by less than 30 (&& !(1..30).any(|p| tail[p..] == tail[..tail.len() - p]) catches your looped row and neither of these). Please replace a_long_reply_whose_rows_differ_is_not_cut, which passes without the change, with a test that streams a reply quoting a dot leader and checks it comes back whole.
2. The cut is reported as the token ceiling. Setting finish_reason to length (lib.rs:743) makes hit_token_ceiling() true, whose doc says the endpoint said so, and extraction_open.rs then records "the open reply hit the token ceiling" (line 163, and 123 when parsing fails). Anyone reading the drops will go looking at the ceiling, which #1301 decided to keep. Please mark a reply cut by the guard separately (for example, a field on Reply), have extraction treat it as cut, and say "the reply started repeating itself; kept up to the last complete item". Keep the truncated_reply code, since the frontend keys on it.

[QuintinBotes]

Thanks, both are in.

1. Only a whole-row loop counts now, with your expression: a tail is skipped when it also matches itself shifted by less than 30 bytes.
   • a_long_reply_whose_rows_differ_is_not_cut is replaced by a_quoted_dot_leader_comes_back_whole. It streams a 200-dot leader across the 256-byte check, and it fails without the shift check.
   • For scale, the earlier rule, without the shortest-period check, ran overnight in front of Ollama on the deployment from Extraction asks for 65,536 completion tokens at temperature 0, so a local model that starts repeating runs for hours #1301. Here is what it did. It cut 57 replies, and from their tails all 57 were real loops: whole statement or entity rows repeated verbatim. Under the new rule, 5 of them would no longer be cut and would run to the ceiling instead: four loops of rows shorter than 30 bytes (for example ["NuGet", "tool", 1], , 22 bytes) and one run of spaces. That seems the right side to err on. I'm mentioning it in case you'd rather revisit the bound later.
2. A guard cut is reported as repetition, not the ceiling:
   • It is now Reply::repeating, separate from finish_reason, so hit_token_ceiling() stays the endpoint's word.
   • extraction_open takes it as cut and says "the reply started repeating itself; kept up to the last complete item". When nothing parses, it says "response started repeating itself and could not be parsed; this chunk did not enter the graph". Both keep their reason codes, truncated_reply and chunk_unextracted.
   • The loop test now also checks repeating and !hit_token_ceiling().
   • The extraction warning keeps its hit_token_ceiling log field and adds repeating.
   • chat_at_streaming sits behind chat_retrying_rate_limits_at, so the guard also covers errata, implication, phrase and type alignment, time resolution and the ontology agent. The description now says so. The phrase and type alignment warnings, which log finish_reason, now log repeating as well, since a guard cut no longer shows up there as length.

docs/design/extraction.md gets a short "Where a looping answer stops" paragraph next to "Where a long answer stops", with the shortest-period reason and the short-row trade-off.

[QuintinBotes]

One more data point, with the details in #1301. In 11 hours with this exact rule, the guard cut 58 loops, all real. Another 8 loops ran to the cap because their rows were shorter than 30 bytes, for example ["tool", "tool", 1], (21 bytes).

The patterns the bound protects (a dot leader, <td></td>, a run of spaces, null, ) all have periods under 10 bytes. If you'd like, I can lower the bound to 16 bytes so those short-row loops are cut too, with a test streaming one. Otherwise I'll leave it at 30.

[WaylandYang]

Thank you, Quintin! Both points are in: the shortest-period check stops quoted dot leaders and empty cells from being cut, and the dot-leader test pins that; and repeating stays separate from finish_reason, so the drop row tells people to look at the loop, not the ceiling. The overnight numbers (58 cuts, all real) are reassuring.

On the 16-byte idea, I'd keep 30: at 16, a quoted run of <td class="num"></td> or <td align="right"></td> would be cut, and losing the rest of a reply costs more than letting a few short-row loops run to the cap. Merging.