Skip to content

feat(runtime): give ArchiveRead search, line reads, and decoded paging - #4380

Open
liugddx wants to merge 3 commits into
apache:mainfrom
liugddx:fix/archive-read-retrieval
Open

feat(runtime): give ArchiveRead search, line reads, and decoded paging#4380
liugddx wants to merge 3 commits into
apache:mainfrom
liugddx:fix/archive-read-retrieval

Conversation

@liugddx

@liugddx liugddx commented Aug 31, 2026

Copy link
Copy Markdown
Member

What

Once a large tool result is archived, retrieving content from it was awkward (issue #4355, part of #4267):

  • no search-within-archive — you could only page blindly by character offset;
  • character-only reads — poor for line-oriented terminal output;
  • read sliced the JSON-escaped serialization — pages landed on escape bytes (\n, \") rather than decoded text;
  • inspect gave no preview or coordinate space for plain-text / object payloads (only keys for objects, valueType for scalars).

This extends ArchiveRead within the existing ≤TOOL_RESULT_ARCHIVE_MAX_RESPONSE_CHARS bounded-response invariant, so a read can never itself trigger another archive:

  • search operation — a literal, case-insensitive substring scan returning matched offset, line, and a bounded context snippet per hit; capped per page (TOOL_RESULT_ARCHIVE_MAX_SEARCH_MATCHES) with nextOffset/hasMore to resume.
  • line-based readsunit: "line" pages whole lines by line offset/limit, trimmed by binary search to the response budget, with nextLineOffset. unit defaults to "char" so existing callers are unaffected.
  • decoded paging — string (terminal/text) payloads page the decoded string so offsets land on real characters; structured payloads still page their canonical JSON.
  • inspect positional index + preview — text payloads report totalChars/totalLines and a short preview; object/scalar payloads report totalChars and a preview (objects with an items[] manifest keep their existing shape).

Why it's safe

  • Search is literal, never a regex — no pathological backtracking, no ReDoS.
  • Every response path (search, line, char, inspect) is bounded to TOOL_RESULT_ARCHIVE_MAX_RESPONSE_CHARS — search stops adding matches, and line reads binary-search their line count, before the envelope overflows.
  • Scope is retrieval ergonomics only: the archive threshold behavior (active-tool-result-prune.ts) and the upstream Shell capture/preservation contract are deliberately untouched, per the audit.

Tests

Six regression tests in tool-result-archive-resource.test.ts:

  1. inspect surfaces a positional index + preview for text payloads.
  2. read pages the decoded string, not the escaped JSON.
  3. read by line range returns whole lines and resumes with nextLineOffset.
  4. search locates a case-insensitive substring with offsets, line numbers, and snippets.
  5. search stays within the response budget and resumes (nextOffset) under a pathological dense-match input.
  6. search without a pattern fails closed.

Causal proof (fail-without / pass-with): with the production dispatch temporarily reverted to the old char-only / serialized-JSON behavior, all six new tests fail (# fail 8, the extra two being the pre-existing inspect-manifest tests under the crude stub); with the change restored, the suite is green (# pass 11).

Part of #4267. Closes #4355.

Once a large tool result is archived, retrieving from it was awkward:
there was no way to locate a substring without blind character paging,
reads were character-only (poor for line-oriented terminal output),
`read` sliced the JSON-escaped serialization rather than the decoded
text, and `inspect` gave no preview or coordinate space for plain
text/object payloads.

Extend ArchiveRead within the existing bounded-response invariant:

- add a `search` operation — a literal, case-insensitive substring
  scan returning matched offsets, line numbers, and bounded context
  snippets, capped per page with `nextOffset` to resume;
- add line-based reads (`unit: "line"`) alongside character offsets,
  trimmed by binary search to the response budget with `nextLineOffset`;
- page the DECODED string for text payloads so offsets land on real
  characters, not JSON escape bytes (structured payloads still page
  their canonical JSON);
- have `inspect` surface a positional index (totalChars/totalLines) and
  a short preview for text, object, and scalar payloads.

Search is literal (never a regex) so it cannot backtrack pathologically,
and every response — search, line, char, inspect — stays within
TOOL_RESULT_ARCHIVE_MAX_RESPONSE_CHARS, so reading an archive can never
itself trigger another archive.

Adds six regression tests covering each capability plus an adversarial
dense-match search that stays within budget and resumes.

Part of apache#4267. Closes apache#4355.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added the effort/M Under 500 readable lines label Aug 31, 2026
liugddx and others added 2 commits September 1, 2026 00:01
…eRead

The ArchiveRead schema now advertises the `search` operation plus the
`unit`/`pattern` parameters and updated descriptions. That tool JSON is
part of the headless-coding-v1 provider wire contract frozen by
`hosted execution freezes the headless coding provider wire contract`,
so its SHA256 legitimately changes. Update the expected tools hash to
match the new, intended contract.

Part of apache#4267. Follow-up to the ArchiveRead retrieval-ergonomics change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The archived-placeholder readInstructions string is hashed into every
model-projection transitionId (buildModelProjectionTransition digests
the replacement projection, which embeds readInstructions). Extending
this string perturbs those content-derived ids and flips the relative
ordering of the two rival transitions in the conversation-copy fold
test, which asserts on which SHA256 sorts larger.

ArchiveRead's new search/line-unit operations are already advertised to
the model through the tool description and parameter schema, so the
placeholder hint does not need to change. Restore it to the exact
upstream bytes to leave transition identities untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/M Under 500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

harness: ArchiveRead search / line reads / section index ergonomics (item 3 of #4267)

1 participant