|
| 1 | +# Investigation Index |
| 2 | + |
| 3 | +## Findings (confirmed) |
| 4 | +- find-jit-hugemethodlimit | Generated DFA_SWITCH/NFA methods over 8000 bytecodes (HugeMethodLimit) run INTERPRETED — proven 20.8x/17.8x with -XX:-DontCompileHugeMethods; root: 100-state buckets sized vs 64KB hard limit + inline accept-anchor blocks; corpus blast radius 5 classes, sweeps ~0, real-traffic tail | [jit, hugemethodlimit, dfa-switch, codegen] | MEASURED prod impact: logs-processing JDK regex 5.60% CPU ($413K/yr, ~100% grok-driven, re2j 0.00% not landed in prod), prof-analyzer 1.00%, apm-processing 0.38% (its heavy regex is Rust dd_sds — not replaceable); f revised to ~2x central -> ~$210K/yr central ($207-280K range), ~89% in logs-processing; notebook 15576172 | [prod-measurement, continuous-profiler, cloud-cost, savings-estimate, grok] |
| 5 | +- find-refusal-set-parity | reggie vs rust refusal sets DISJOINT on the 513 corpus: 8 reggie-only (1.6%, all correctness guards: 4x alternation-priority, 2x anchor-dilution, nullable-capture, anchor-in-quantifier), 2 shared, 12 rust-only that reggie serves; with allowJdkFallback ZERO functional refusals (fallback is R1-prefiltered too); LIFT LANDED (6caf5db): 3 alternation-priority rules re-route to PikeVM, refusals 10->7 (native 506/513); anchor-dilution NOT lifted (measured diverger in class); user correction absorbed: logs-backend = grok-on-JDK, rust serves apm — parity question only matters for a rust-surface replacement; PIKEVM-JUDGED: 4 of 8 lift candidates (0 JDK-oracle findings each); 4 honest (nullable-capture/anchor-in-quantifier/empty-class — need backtracking for byte-identical spans) | [refusals, coverage, migration, jdk-fallback] |
| 6 | +- find-rust-engine-crossover | REAL-corpus 3-engine smoke (513 logs-backend patterns x 6 lines, FFM incl. marshal, zero divergences): rust 3-5x faster than reggie at median (11-54x NOMATCH totals), reggie 2.1x faster than JDK on matched pairs (median) -> f revised; rust counted-quantifier wall real ({0,256} fails even at 256MB NFA); dd_sds replacement would RAISE cpu (architectural case only) | [rust-regex, real-corpus, smoke-test, f-revision, dd_sds] |
| 7 | + |
| 8 | +## Hypotheses |
| 9 | +- hyp-unanchored-find-prefilter | CONFIRMED: full arc delivered and measured — R1 (38eeb62) -> R2 (b5cf63e) -> R2b/R1b (97ec426) -> 1-char intrinsic (293d0e6); no-match sweep 15.6ms -> 636us (24x), reggie 0.25us/pair vs rust 0.66 / jdk 11.97 in the final 4-engine matrix; acceptance gate (RealCorpusScanBenchmark) passed | [prefilter, memchr, single-pass, confirmed] |
| 10 | + |
| 11 | +## Dead ends |
| 12 | +- dead-bitstate-lazy-fastconsume | BitState lazy char-class loop fast-consume: gates green but box JMH flat — corpus lazy families don't consume (stack-frame's .*? exits empty via optional tail; cost = unanchored-seed DFS overhead); local probe wins were JVM noise | [bitstate, lazy-loop, perf, measured-negative] |
| 13 | +- dead-group-boundary-fusion | REFUTED/REVERTED (measured net-negative): eager + deferred fusion both built+tested; removes only ~10 pops on realistic paths while a 5th opsStack array taxes every push/pop; helps only degenerate 5-char; reverted to 677417b; short-input floor needs a hybrid engine, not fusion | [fusion, dfs, perf, measured-negative] |
| 14 | + |
| 15 | +## Evidence |
| 16 | +- ev-r1b-char-intrinsic | 293d0e6: 1-char prefilter facts now scan with the char indexOf intrinsic (String overload's first-char scan is 5.7x slower on x86; mac probes mask this — NEON); LdapNoMatch LONG outlier 276 -> 2,026 ops/ms (7.3x, re2j 358) — reggie now leads every shape in the 4-engine matrix; corpus no-match -1.7% raw / ~9% relative | [prefilter, intrinsic, simd, 293d0e6] |
| 17 | +- ev-4lane-bench-results | e61fdd9: 4-engine matrix (re2j corpus lane + rust IAST lanes + corpus-copy sync) — per-pair reggie 0.49/0.25µs m/nm vs rust 0.62/0.66, jdk 1.81/11.97, re2j 6.58/5.14 (re2j slowest on matched, 13x reggie: the never-landed decision quantified); ONE engine-beats-reggie outlier: LdapNoMatch LONG re2j 361 vs reggie 276 ops/ms (jdk 90, rust 17) — follow-up target | [benchmark, re2j, rust, 4-engine, e61fdd9] |
| 18 | +- ev-hybrid-context-rematch | ff34f90: CI divergence-gate 29>28 failure root-caused (11 branch-introduced hybrid divergences: standalone-string span re-match fired $/\Z/\z out-of-context + trusted an inverted pruned-DFA end) and fixed — PikeVM/BitState halves now search from the DFA's leftmost start in the full input; raw-finding cross-check vs main's oracle was the decisive method; fuzz 29->15 (0 regressions), corpus 0-divergence + 102 hybrids unchanged, box matched 180.2µs flat | [hybrid, anchors, context, ff34f90, ci] |
| 19 | +- ev-jit-gate-landed | 5a826bd: exact classfile method-size scan + fallback-only JIT gate (2 monster patterns decline to JDK under fallback; strict contract unchanged); full build+jacocoVerify green; changelog updated; REMAINING = merge to main + release.sh (SSM) + logs-backend PR | [jit, gate, release, 5a826bd] |
| 20 | +- ev-rd-hybrid-landed | 5e9abbf: RD give-back originals hybridize (PikeVM capture half) — matched 182.4us, no-match 641.9 (−7.7%, RD backtracking gone); 102 hybrids; fresh fuzz seeds: zero new findings, pre-existing out-of-corpus divergences recorded | [rd, give-back, 5e9abbf] |
| 21 | +- ev-alternation-retry-landed | ae3a2c4: alternation-priority retry (mode-gated block bypass + needsFallback guard + END-anchor-in-alternation decline) — matched 225.3->187.6us (−17%), 1.21x ahead of rust; hybrid substring re-match = general anchor-context hazard (fuzz 48879); maven semver honestly declined (atomic charset overlap) | [alternation, retry, ae3a2c4] |
| 22 | +- ev-leftmost-pruning-landed | 502181d: RE2 leftmost-first thread pruning (anchor-free lazy retries only; HybridMatcher.lazyFind routes matches() to the NFA half) — matched 282.9->225.3us (−20%), reggie FIRST TIME faster than rust matched (225 vs 232); two lazy tranches net blended −5.7%/pair | [pruning, re2, 502181d] |
| 23 | +- ev-lazy-hybrid-landed | 7a73126: lazy-aware captureless DFA retry + central leftmost-first certification — 9 more hybrids (66), matched 313.7->282.9us (−9.8%) but no-match 665->693.7 (+4.3%), blended per-pair FLAT; stack-frame chain genuinely uncertifiable (13 real priority conflicts, 530/266k divs caught pre-gate) — needs priority-pruning DFA; probe artifact: 513-pattern JVM overstates per-pattern no-match 20-200x vs isolated | [lazy, certification, 7a73126] |
| 24 | +- ev-lane1-jit-hybrid-landed | 6b583a8: JIT-sized DFA_SWITCH codegen + start-anchored hybrid re-admission + BitState nfa-half — box matched 313.7us WIN and no-match 665us (29% faster, first simultaneous win; 10b1a43 regression inverted) | [jit, hybrid, 6b583a8] |
| 25 | +- ev-word-boundary-find-fix | 10b1a43: NEW find-span parity battery (513x527 real pairs, ~2s, committed RealFindParityTest) found 2 PRE-EXISTING find() divergences — MULTI_GROUP_GREEDY accepted across word boundaries (name:(\S+) on "@peer.hostname..." matched), RECURSIVE_DESCENT (.*)end no-match on "appendend"; fixed via targeted PatternAnalyzer declines (blanket ->PikeVm reverted: broke PINNED_BACKREFERENCE backref routing); 266k find pairs 0 divergences after; hybrid exclusion kept | [word-boundary, find, parity-gate, 10b1a43] |
| 26 | +- ev-rust-bench-lane-standardized | 60db93c: Rust regex = 4th benchmark lane — JNI shim (Java-21-safe, marshal in timed path) + RustRegexEngine (scan + full-match semantics, graceful unavailability) + buildRustEngine task + MatchOperationBenchmark lanes + RealCorpusScanBenchmark (513 real patterns committed, MATCH/NOMATCH split; reproduces smoke exactly 491/10/12, rust no-match ~303ns/pair) — the R1/R2 acceptance gate | [benchmark, jni, 4th-lane, real-corpus] |
| 27 | + |
| 28 | +- ev-r1-prefilter-landed | 38eeb62: R1 SHIPPED — sound AST literal extraction + universal rejection wrapper; workspace-jb measured no-match sweep 33.9ms -> 3.7ms (9.1x), reggie/rust 10.6x -> 1.17x, matched unchanged, jdk/rust controls flat; audit 287/513 literals, 63% instant-reject, 0 divergences; R2 (single-pass .* scan) remains | [r1, prefilter, workspace-jb, 9x, 38eeb62] |
| 29 | + |
| 30 | +- ev-r2-rd-landed | b5cf63e: R2 first half — RD .*-prefix find() linear (anchor-start + fast give-back; slot-collision bug caught by span battery); box: no-match 2.66ms (R1+R2 total 12.7x, reggie 1.19x AHEAD of rust), matched 508us, blended corpus fastest (~16x jdk); R2b = HybridMatcher lane (~40% of residual); detour: BytecodeDebugger fixed to report ACTUAL routing (6e9b8eb) | [r2, give-back, anchor-start, b5cf63e] |
| 31 | + |
| 32 | +- ev-r2b-landed | 97ec426: R2b = ci-facts (global (?i) facts from stripped pattern, ASCII-ci exact scan) + R1b 1-char required facts (requiredChar soundness-by-construction walk); same-day box series no-match 15,571 -> 2,475 (R1) -> 2,070 (R2) -> 983us (15.8x, reggie 1.79x AHEAD of rust); matched flat 338us (1.48x behind rust, 2x ahead jdk); hybrid no-match collapsed 582->58us; METHODOLOGY: generated methods need 10k+ invocations for C2, short-rep probes overstate reggie ~6x; box drift ~1.85x between days — same-run controls mandatory | [r2b, r1b, ci-facts, prefilter, 97ec426] |
| 33 | + |
| 34 | +- ev-lazydfa-nfa-delegate-limit | Analyzer recommends LAZY_DFA but generation reuses NFABytecodeGenerator for span methods -> 64KB method limit at ~6800 NFA states -> JAVA_FALLBACK; effectively unreachable for the large-NFA patterns it targets; 0 corpus impact today | [lazydfa, fallback, method-size] |
| 35 | + |
| 36 | +- ev-real-input-parity | 32a9e26: real-input parity battery (527 lines harvested from logs-backend grok tests x 513 corpus, drop-in contract) found 3 divergences invisible to the synthetic corpus — nullable-tail group span (tagged DFA), VARIABLE_CAPTURE_BACKREF suffix/end-anchor bugs, parse-refusal fallback bypass — ALL FIXED; now 270,351 pairs / 8,179 matches / ZERO divergences; battery committed (RealInputParityTest, ~1.2s); supplier drafted for the shadow seam; real-traffic shadow rollout remains (logs-backend PR + reggie 0.4.0) | [real-inputs, parity, drop-in, 32a9e26] |
| 37 | + |
| 38 | +## Questions |
| 39 | +- q-generated-nfa-findfrom-broken | FIXED (1283fec): findFrom's indexOf scan-start jump now gated on verified match prefix (everyMatchStartsWith walk); sound indexOf==-1 rejection kept; corpus 513->528 with 15 synthetic guards (+9 inputs, 4 new hybrids -> 106); box matched 180.7/no-match 636.4 flat; previously ROOT-CAUSED — findFrom's indexOf scan-start jump assumes the required literal sits at offset 0 of the match; requiredLiterals only guarantees somewhere-in-match ((.c)+ on "-cc" → start 1; (.0){3,} → -1). Reachability census: 0/513 corpus standalone OPTIMIZED_NFA, hybrid halves gated off since ff34f90, 0 fuzz findings, 0 constructed public-API repros. Fix = first-set prefix gating when a lane routes captureless patterns there | [optimized-nfa, findfrom, required-literal, latent-bug] |
| 40 | +- q-fuzz-preexisting-divergences | OPEN: 9 divergences from fresh seeds 90210/424242 all reproduce pre-change (backref {0} boolean, c{1,1} group span, $|\z alternation span + others); out-of-corpus, canaries for future tranches | [fuzz, pre-existing] |
| 41 | +- q-hybrid-anchored-admission | OPEN: admitting anchored PIKEVM/BITSTATE into hybrid regressed matched 340->687us (10b1a43, reverted 2db164b) — whole-line anchored .*-$ shapes gain nothing from narrowing (DFA span = full line; 68us LazyDFA/RD scan + 45us PikeVM rescan on a 162-char line) and the LazyDFA/RD lane runs ~420ns/char on alternation shapes (WHY = uninvestigated, entry points in node); re-admission needs a DFA-half-cheap + span-narrows rule; `.+)` vs `.*)` in the last alternation branch decides the alternation-priority flag (probe lied with `.*`) | [hybrid, anchors, matched, lazydfa] |
| 42 | +- q-lazydfa-findfrom-leftmost | FIXED 222c8e6: plain-closure findFrom restarts at matchStart+1 (was death+1, skipping viable starts inside the dead span); self-anchoring closures (PikeVM findStep/rejectStep, BitState rejectStep) moved to findFromUnion (union covers dead-span starts, pos+1 sound); trap x(?:a+b+|b+a+){75} on "xax"+"ba"*75 -> 2 not 3 | [lazydfa, leftmost, fixed] |
| 43 | +- q-caret-midpattern-anchor | OPEN, PRE-EXISTING (reproduces pre-222c8e6): (?:[^a-caa]|c)^|.\\z\\z on "1\\n0\\n_cbcc_" — jdk [9,10) via .\\z\\z branch vs reggie [0,1) via the ^-branch; findAll 1 vs 7; suspect mid-pattern non-multiline ^ in alternation; fuzz seed 131071 reproduces (FuzzProbe argv[0]) | [anchor, caret, alternation, fuzz] |
| 44 | +- q-lazydfa-findfrom-leftmost | OPEN, LATENT (no corpus pattern routes LAZY_DFA): LazyDFACache.findFrom restarts at death+1 without re-walking — a later viable start can begin inside a dead attempt's span (ab|b on xaab returns -1 vs true leftmost 2) and ran-out-of-input attempts also skip viable later starts; fix before routing hybrid/other lanes through it; test shape: (?:aXb|Xb)-large on aaXb | [lazydfa, leftmost, soundness] |
| 45 | +- q-z-anchor-span-bug | FIXED 222c8e6 (DFA_UNROLLED greedy-walk \\Z early-return deleted; accepting-state recording with full anchor conditions already handles before-terminator acceptance and falls through to consume; [^a]*$\\Z on x\\n now [0,2)) | was OPEN, PRE-EXISTING (on 677417b): [^_-ab-c]*$\Z on "ccc0_1\n00a\n" — jdk span [10,11) vs reggie [10,10); greedy newline consumption under $ + \Z; fix separately | [anchor, span, greedy, fuzz] |
| 46 | +- q-real-input-validation | PARTIAL (local half done, 32a9e26): remaining = prod shadow rollout (supplier drafted; needs reggie 0.4.0 published + logs-backend PR; Bastien Lemale's branch is the wiring precedent) | was OPEN: all corpus numbers use synthetic lines — validate fleet f against REAL grok traffic via the shadow seam (logs.processing.grok.shadow.*, ReggieRegexPatternSupplier) before quoting beyond the ~2x floor | [inputs, grok, shadow-seam, validation] |
0 commit comments