Commit 70b2aee
perf(tokenizer): scan with indexOf in fastForwardTo
`fastForwardTo` advances to its sentinel one character at a time. A single
`String.indexOf` compiles to a vectorised memchr instead: roughly a hundred
times faster per character, against a fixed call overhead of about 12ns
versus 2.75ns.
That overhead means `indexOf` loses below a skip distance of four or five,
and on real documents about 28% of calls land there. They do not matter:
those calls carry 1% of the characters crossed, while the longest 0.5% -
script and style bodies, long attribute values - carry 46%. Measured over
258 real-world pages, the tokenizer drops 31.6% on attribute-heavy markup,
43.6% on script-heavy, 45.4% on text-heavy.
Probing a few characters before calling `indexOf` looks like free insurance
against the short calls and is not: it charges the 71% of long calls to
save at most ~9ns on the short ones, and every probe length modelled comes
out slower than plain `indexOf`. So there is no probe.
The needle table covers 0-127 only. Above that `FastForwardNeedles[c]` is
undefined and `indexOf` would stringify it and search for the literal word
"undefined" - a wrong position, silently, with nothing thrown. Every
current call site passes a markup character ({34, 39, 45, 60, 62, 63, 93}),
but `c` is a plain number and nothing in the type says so, hence the guard.
Its measured cost is nil: the 12ns is the native call, not the lookup.
The differential harness added in the previous commit covers the swap:
identical on every case it checks, including start positions outside the
buffer, the empty buffer, non-zero offsets and astral-plane content.
Known limit: on markup where nearly every skip is three characters or
fewer - `<b>a</b><i>b</i>` repeated - the entry cost dominates and this
costs about 5%. The tipping point is around 80% short calls, which is a
generated-file profile rather than a page one; the densest page in the
corpus sits at 28% and still gains.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>1 parent 169b341 commit 70b2aee
1 file changed
Lines changed: 29 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
175 | 175 | | |
176 | 176 | | |
177 | 177 | | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
178 | 187 | | |
179 | 188 | | |
180 | 189 | | |
| |||
374 | 383 | | |
375 | 384 | | |
376 | 385 | | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
377 | 395 | | |
378 | 396 | | |
379 | 397 | | |
380 | 398 | | |
381 | | - | |
382 | | - | |
383 | | - | |
384 | | - | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
385 | 410 | | |
386 | 411 | | |
387 | 412 | | |
| |||
0 commit comments