Skip to content

Search the scrollback without turning it into text - #47

Merged
tomlm merged 2 commits into
tomlm:mainfrom
JohnCampionJr:scrollback-search
Aug 28, 2026
Merged

Search the scrollback without turning it into text#47
tomlm merged 2 commits into
tomlm:mainfrom
JohnCampionJr:scrollback-search

Conversation

@JohnCampionJr

@JohnCampionJr JohnCampionJr commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

No longer stacked — #45 merged and this is rebased down to the feature alone. Two commits, conflict-free against main, full suite green. Ready to review on its own.

Ghostty's most-requested feature until 1.3. The design note with the full reasoning is written up separately; the short version turns on one measurement.

The measurement

A 10,000-line scrollback at 240 columns, searched for error, both ways, same 25,325 hits:

approach time allocated
a string per line, then IndexOf 9.7 ms 16.3 MiB
codepoints read out of the cells 3.7 ms 0.0 MiB

A find box searches per keystroke, so the obvious implementation is ~80 MiB to type "error" — in a library that just spent #38 getting per-character allocation to zero. So BufferSearch walks cells and never builds a string.

What walking cells buys beyond the allocation

The offset-mapping problem dissolves. A string search returns character offsets that then have to be mapped back onto cells — wide characters, clusters, wraps. Here the position is the (row, column) cursor, and a match that crosses a wrap simply produces two runs because the cursor changed row part way through. The runs share a MatchId — the same shape as an OSC 8 link that wrapped — and TryMoveNext steps by match, so a wrapped one is one stop.

Logical lines, not physical. A search that stops at the right edge misses exactly the matches on long output, which are the ones worth finding.

Width-0 cells are stepped over, not compared. The placeholder behind a wide glyph and an orphaned combining mark aren't characters; comparing them would fail a needle against text that reads exactly like it.

Results that survive the buffer moving

The ring drops lines as output arrives, and a stale row index fails silently — the highlight just lands somewhere else. BufferSearch subscribes to Trimmed and shifts, as SelectionManager already does, dropping what fell off the top. There's a test that scrolls a match through a trim and checks the row it lands on still says needle.

Shaped for the renderer

HitsOnRow(row) answers by binary search over row-ordered hits and returns a span in place — asked once per row per frame, and the answer is almost always nothing. The cap (10,000 runs) is announced through Truncated rather than silently applied: a count that quietly stops being true reads as a bug in the search rather than a limit on it.

Deliberately no regex

Regex takes a string, so supporting it means materialising the buffer as text — the 16 MiB this design exists to avoid, and paid by every search rather than only the ones that asked. Substring, case-insensitive by default, whole-word as an option. If regex ever lands it should materialise one logical line at a time.

A measurement trap worth recording

This change's own zero-allocation test initially failed at 66 MB — against code that was allocating nothing. GC.GetTotalAllocatedBytes is process-wide and xUnit runs test classes in parallel, so it counted the rest of the suite's allocations on other threads against the search. GetAllocatedBytesForCurrentThread cannot be polluted. Filed here because it's the same trap anyone adding an allocation assertion to this suite will hit.

Tests

1191 pass, thirteen new: wrap-straddling matches, whole-word across a wrap, ring trimming, the cap admitting it bit, and the zero-allocation claim asserted per thread.

The renderer half — highlight in the render path, scroll-to-match, find-next forwarded to the control — is already written as Iciclecreek#88, and follows once this merges and a prerelease carries it. The terminal-side PRs deliberately come after the emulator ones: they cannot compile until the package exists.

🤖 Generated with Claude Code

JohnCampionJr added a commit to JohnCampionJr/XTerm.NET that referenced this pull request Aug 28, 2026
The perf gate on tomlm#47 flagged alt-redraw at +6.0% against a 6.0% gate with 2% noise,
and it was right. NoteLink carried a remark saying "guarded at the CALL as well as
inside" -- and the three call sites were bare. The guard lived only inside the
helper, so every printed character and every batched chunk paid a call to be told
there is no link and never has been. The exact shape that cost alt-redraw 12% in
the placeholder work, caught this time by the CI gate that was calibrated on it.

The three writers now test `_linkUrl is not null || line.HasLinks` themselves and
call NoteLinkRun directly; the helper is gone, because three guarded call sites
plus a helper whose comment lied is worse than no helper.

1191 tests pass unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JohnCampionJr

Copy link
Copy Markdown
Collaborator Author

The compare failure was real, and it was not this PR — it was #45 underneath, and the gate did exactly what it was calibrated to do.

alt-redraw at +6.0% against a 6.0% gate, ±2% noise. The cause: NoteLink carried a remark saying "guarded at the CALL as well as inside" — and the three call sites were bare. The guard lived only inside the helper, so every printed character and every batched chunk paid a call to be told there is no link and never has been. The exact unguarded-call shape that cost alt-redraw 12% in the placeholder work, caught this time by CI instead of by hand.

Fixed in 40da2d4 on #45 and merged through: the three writers test _linkUrl is not null || line.HasLinks themselves, and the helper is gone — three guarded call sites plus a helper whose comment lied is worse than no helper. 1191 tests unchanged; compare re-running on the push.

@JohnCampionJr

Copy link
Copy Markdown
Collaborator Author

some stacking here, sorry about that. should be the last time. this is the last of what I wanted to land.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Confirmed correctness issues in the new search and hyperlink-span maintenance logic need fixes before the new APIs can be relied on.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds scrollback searching that operates directly on buffer cells (avoiding per-line string materialization/allocation) and expands line-level state to anchor OSC 8 hyperlinks and OSC 133 shell-integration marks to BufferLines for stable hit-testing and prompt navigation.

Changes:

  • Introduces BufferSearch (and SearchHit/SearchOptions) to find matches across logical (wrapped) lines without building strings.
  • Anchors OSC 133 marks (LineMark) and OSC 8 hyperlink spans (LineHyperlink) to BufferLine, with print-path bookkeeping and prompt navigation helpers on Terminal.
  • Adds comprehensive tests covering wrapping, trimming, caps/truncation, hyperlink/mark anchoring, and allocation assertions.
File summaries
File Description
src/XTerm.NET/Terminal.cs Adds prompt navigation helpers based on anchored OSC 133 marks.
src/XTerm.NET/Search/SearchTypes.cs Adds search result and option types (SearchHit, SearchOptions).
src/XTerm.NET/Search/BufferSearch.cs Implements allocation-free scrollback search over buffer cells.
src/XTerm.NET/InputHandler.cs Anchors shell marks to lines and records hyperlink spans during printing.
src/XTerm.NET/Buffer/LineMark.cs Defines line-anchored shell-integration mark structure.
src/XTerm.NET/Buffer/LineHyperlink.cs Defines line-anchored hyperlink span structure.
src/XTerm.NET/Buffer/BufferLine.cs Stores/updates marks and hyperlink spans on lines; clears on reuse.
src/XTerm.NET.Tests/ShellIntegrationMarkAnchorTests.cs Tests mark anchoring and prompt navigation behavior.
src/XTerm.NET.Tests/HyperlinkAnchorTests.cs Tests hyperlink span anchoring and mutation under overwrites.
src/XTerm.NET.Tests/BufferSearchTests.cs Tests search behavior (wraps, stepping, trimming, cap, allocations).
Review details

Suppressed comments (1)

src/XTerm.NET/Search/BufferSearch.cs:332

  • Whole-word boundary check on the "after" side doesn't skip width-0 placeholder cells, so a match that ends before a wide glyph's placeholder may see the placeholder as the next character and incorrectly treat it as a boundary.
        if (!Advance(lines, endRow, ref r, ref c, skipZeroWidth: false))
            return false;   // end of the logical line
  • Files reviewed: 10/10 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/XTerm.NET/Search/BufferSearch.cs
Comment thread src/XTerm.NET/Buffer/BufferLine.cs Outdated
Comment thread src/XTerm.NET/Search/BufferSearch.cs
JohnCampionJr and others added 2 commits August 28, 2026 12:42
Ghostty's most-requested feature until 1.3, and the design turns on one measurement:
asking each line for a string and running IndexOf costs 9.7 ms and 16.3 MiB per
search over a 10,000-line buffer, while reading codepoints out of the cells costs
3.7 ms and allocates nothing, for the same 25,325 hits. A find box searches per
keystroke, so the first of those is roughly 80 MiB to type "error" -- in a library
that went to some trouble to stop allocating per character.

So BufferSearch walks cells. The walk moves a (row, column) cursor and compares
codepoints, which also dissolves the offset-mapping problem: the position IS the
cursor, and a match that crosses a wrap simply produces two runs because the cursor
changed row part way through. The runs share a MatchId, the same shape as an OSC 8
link that wrapped, and stepping moves by match so a wrapped one is one stop.

Logical lines, not physical: a search that stops at the right edge misses exactly
the matches on long output, which are the ones worth finding. Width-0 cells are
stepped over rather than compared -- the placeholder behind a wide glyph and an
orphaned combining mark are not characters, and comparing them would fail a needle
against text that reads exactly like it.

Results follow the buffer. The ring drops lines as output arrives and a row index
that is not adjusted goes stale silently -- the highlight lands somewhere else, and
nothing about it looks wrong. Subscribes to Trimmed and shifts, as SelectionManager
does, dropping what fell off the top.

HitsOnRow answers by binary search over row-ordered hits and hands back a span in
place, because the renderer asks once per row per frame and the answer is almost
always nothing. The cap (10,000 runs) is announced through Truncated rather than
silently applied: a count that quietly stops being true reads as a bug in the search
rather than a limit on it.

No regular expressions, deliberately. Regex takes a string, so supporting it means
materialising the buffer as text -- the 16 MiB this design exists to avoid, paid by
every search rather than the ones that asked. Substring, case-insensitive by
default, whole-word as an option.

One measurement trap found by this change's own test: it first asserted with
GC.GetTotalAllocatedBytes, which is process-wide, and xUnit runs test classes in
parallel -- so it counted 66 MB of other tests' allocations against the search and
failed a design that was allocating nothing. GetAllocatedBytesForCurrentThread
cannot be polluted. The bisection that found this took longer than the fix.

Thirteen tests: wrap-straddling matches, whole-word across a wrap boundary, ring
trimming, the cap admitting it bit, and the zero-allocation claim asserted per
thread.

1191 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The third finding -- SplitLinksOver appending the right fragment -- was fixed in
b1fcc00 before this review posted; Copilot reviewed the pre-merge head.

The needle is now codepoints, not chars. A cell stores a codepoint, and comparing
it to UTF-16 chars one at a time can never match anything outside the BMP: an emoji
in the needle is two chars and neither equals the cell. The needle folds once per
Find; astral case pairs are left exact, which is honest rather than lossy. A cell
holding a multi-codepoint cluster is still compared by its leading codepoint, and
the remarks now say why: matching full cluster text means materialising it per
cell, the same cost this class exists to avoid and the same reason it offers no
regular expressions. The limitation is one-directional -- base-character searches
still find cells carrying combining marks.

And the whole-word left scan steps past width-0 cells. The placeholder behind a
wide glyph is not a character, and treating it as the neighbour let a whole-word
match sit flush against a CJK letter.

Two regression tests. 1197 pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JohnCampionJr
JohnCampionJr marked this pull request as ready for review August 28, 2026 16:45
@tomlm
tomlm merged commit b0b98e6 into tomlm:main Aug 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants