Skip to content

Commit 70b2aee

Browse files
abarreclaude
andcommitted
perf(tokenizer): scan with indexOf in fastForwardTo
`fastForwardTo` advances to its sentinel one character at a time. A single `String.indexOf` compiles to a vectorised memchr instead: roughly a hundred times faster per character, against a fixed call overhead of about 12ns versus 2.75ns. That overhead means `indexOf` loses below a skip distance of four or five, and on real documents about 28% of calls land there. They do not matter: those calls carry 1% of the characters crossed, while the longest 0.5% - script and style bodies, long attribute values - carry 46%. Measured over 258 real-world pages, the tokenizer drops 31.6% on attribute-heavy markup, 43.6% on script-heavy, 45.4% on text-heavy. Probing a few characters before calling `indexOf` looks like free insurance against the short calls and is not: it charges the 71% of long calls to save at most ~9ns on the short ones, and every probe length modelled comes out slower than plain `indexOf`. So there is no probe. The needle table covers 0-127 only. Above that `FastForwardNeedles[c]` is undefined and `indexOf` would stringify it and search for the literal word "undefined" - a wrong position, silently, with nothing thrown. Every current call site passes a markup character ({34, 39, 45, 60, 62, 63, 93}), but `c` is a plain number and nothing in the type says so, hence the guard. Its measured cost is nil: the 12ns is the native call, not the lookup. The differential harness added in the previous commit covers the swap: identical on every case it checks, including start positions outside the buffer, the empty buffer, non-zero offsets and astral-plane content. Known limit: on markup where nearly every skip is three characters or fewer - `<b>a</b><i>b</i>` repeated - the entry cost dominates and this costs about 5%. The tipping point is around 80% short calls, which is a generated-file profile rather than a page one; the densest page in the corpus sits at 28% and still gains. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 169b341 commit 70b2aee

1 file changed

Lines changed: 29 additions & 4 deletions

File tree

‎src/Tokenizer.ts‎

Lines changed: 29 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,15 @@ const specialStartSequences = new Map<number, Uint8Array>([
175175
[Sequences.XmpEnd[2], Sequences.XmpEnd],
176176
]);
177177

178+
/**
179+
* One-character needles for `fastForwardTo`, so its `indexOf` call does not
180+
* allocate a string per invocation. ASCII only: every sentinel the tokenizer
181+
* looks for is a markup character, and `fastForwardTo` guards the rest.
182+
*/
183+
const FastForwardNeedles = Array.from({ length: 128 }, (_, code) =>
184+
String.fromCharCode(code),
185+
);
186+
178187
/**
179188
* Tokenizer implementation used by `Parser`.
180189
*/
@@ -374,14 +383,30 @@ export default class Tokenizer {
374383
/**
375384
* When we wait for one specific character, we can speed things up
376385
* by skipping through the buffer until we find it.
386+
*
387+
* `indexOf` scans roughly a hundred times faster per character than a
388+
* `charCodeAt` loop, at the cost of a fixed call overhead that makes it
389+
* lose below a distance of about four. That trade is overwhelmingly worth
390+
* taking: on real documents a quarter of the calls fall below the
391+
* crossover but carry one percent of the characters, while the longest
392+
* half-percent - `script` and `style` bodies, long attribute values -
393+
* carry nearly half of them. Probing a few characters first to protect
394+
* the short calls costs more than it saves, so we do not.
377395
* @param c Current character code point.
378396
* @returns Whether the character was found.
379397
*/
380398
private fastForwardTo(c: number): boolean {
381-
while (++this.index < this.buffer.length + this.offset) {
382-
if (this.buffer.charCodeAt(this.index - this.offset) === c) {
383-
return true;
384-
}
399+
/*
400+
* Above 127 the table has no entry, and `indexOf(undefined)` would
401+
* search for the literal string "undefined": a wrong position, with
402+
* nothing thrown to notice it.
403+
*/
404+
const needle = c < 128 ? FastForwardNeedles[c] : String.fromCharCode(c);
405+
const found = this.buffer.indexOf(needle, this.index - this.offset + 1);
406+
407+
if (found !== -1) {
408+
this.index = found + this.offset;
409+
return true;
385410
}
386411

387412
/*

0 commit comments

Comments
 (0)