Repository navigation
0.4.0: backend regex replacement readiness — literal-prefilter arc, hybrid engines, JIT sizing, parity batteries + review hardening (42 findings addressed) - #129
Merged
Conversation
jbachorik
marked this pull request as ready for review
September 18, 2026 12:14
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 293d0e6b8e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
jbachorik
added a commit
that referenced
this pull request
Sep 18, 2026
…129) Settles the cairn investigation for the dd-backend-check arc into doc/investigations/backend-ready: 14 nodes, 16 evidence files. Wires the nine orphaned evidence nodes flagged by the consolidation pass (R1/R2 tranche evidence onto the prefilter hypothesis and the hybrid-admission question), confirms hyp-unanchored-find-prefilter (arc delivered: no-match sweep 15.6ms -> 636us, 24x; acceptance gate passed), and records the JMH method-selector gotcha for rerunning the LdapNoMatch lane.
jbachorik
force-pushed
the
feat/dd_backend_check
branch
from
September 18, 2026 12:22
c316dd3 to
e8783f2
Compare
![datadog-official[bot]](https://avatars.githubusercontent.com/in/1358509?s=60&v=4){kind=small}
There was a problem hiding this comment.
Large counted quantifiers do not have the same behavior in the compile-time and runtime paths. The runtime path also ignores lookaround data and retains nested counters across outer loop iterations.
🤖 Datadog Autotest · Commit 293d0e6 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
jbachorik
force-pushed
the
feat/dd_backend_check
branch
from
September 18, 2026 12:39
e8783f2 to
25ec2c7
Compare
jbachorik
changed the title
Bound compile work for bounded quantifiers (semver hang, OOM bombs),
NUL literal no longer silently dropped (epsilon sentinel collision),
bare '}' is a literal (JDK leniency, unblocks 19 consumer patterns),
groupCount no longer counts '(' inside character classes, optional
capturing groups no longer leak failed-attempt spans, POSIX-style
property classes, CASE_INSENSITIVE for pure-ASCII patterns, and
(?U)/ReggieFlags.UNICODE_CHARACTER_CLASS.
-Dreggie.compile.totalDeadlineMs (default 10s) with the two coverage gaps closed, huge-charset transitions merged into boolean[] lookup tables, 2152-check input-side NUL differential audit, spotless normalization.
Bounded quantifiers x{n,m} lower to counted loops,
BackrefBacktrackMatcher DFS is allocation-free on the hot path; the
counted-loop pattern family reaches re2j parity.
JNI shim (Java-21-safe, marshalling in the timed path), RustRegexEngine with scan + full-match semantics, buildRustEngine task, MatchOperationBenchmark lanes; cairn investigation records for the arc (cost baseline, fusion dead end, rust crossover).
R1: sound AST literal extraction + universal rejection wrapper for unanchored find() (no-match sweep 9.1x). R2: linear find() for .*-prefix recursive-descent patterns. R1b: 1-char required facts + ASCII case-insensitive facts. Alternation-priority conflicts re-route to PikeVM instead of refusing (refusals 10 -> 7). BytecodeDebugger reports the actual pipeline routing.
Greedy \Z spans, LazyDFA leftmost start, three real-input parity fixes (nullable-tail group spans, backref suffix/end-anchor, parse-refusal fallback bypass) found by the new real-input parity battery, two \b find() divergences, misplaced-^ anchor guard.
JIT-sized DFA_SWITCH codegen (methods over HugeMethodLimit ran interpreted), hybrid admission for start-anchored, unanchored and lazy-aware patterns with RE2 leftmost-first thread pruning, priority-aware alternation retry, RD give-back hybridization, JIT method-size gate under ALLOW_JDK_FALLBACK. Corpus matched 340 -> 180us.
…gate Hybrid nfa-halves context-search from the DFA leftmost start (span re-match as fallback); findFrom's literal jump is gated on a verified match prefix; parity corpus extended 513 -> 528 patterns.
…intrinsic re2j real-corpus lane + rust IAST lanes (the 4-engine benchmark matrix); 1-char prefilter facts scan with the char indexOf intrinsic (the String overload's first-char scan is 5.7x slower on x86): IAST LdapNoMatch LONG 276 -> 2,026 ops/ms, corpus no-match ~9% relative.
jbachorik
force-pushed
the
feat/dd_backend_check
branch
from
September 21, 2026 07:03
25ec2c7 to
b185c90
Compare
Cairn investigation settled into doc/investigations/backend-ready: 14 nodes, 16 evidence files; prefilter hypothesis confirmed (arc delivered, no-match sweep 24x, acceptance gate passed).
jbachorik
force-pushed
the
feat/dd_backend_check
branch
from
September 21, 2026 10:55
b185c90 to
9771efb
Compare
Collaborator
Author
|
@arp review |
jbachorik
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Makes the regex engine (reggie) ready to replace
java.util.regexin high-volume backend workloads. 10 commits across four areas — all measured on an idle 16-core Linux box with same-run jdk/rust controls:Commits are squashed to one commit per logical area:
Performance — real-corpus scan (528 real production patterns × 6 log lines,
RealCorpusScanBenchmark):find()for.*-prefix recursive-descent patterns + R1b 1-char and ASCII-ci required facts: no-match 3.7ms → 983µs (15.8x vs 0.3.0 baseline; 1.79x ahead of rust-regex)indexOfintrinsic (the String overload's first-char scan is 5.7x slower on x86): corpus no-match ~9% relative; IASTLdapNoMatchLONG 276 → 2,026 ops/ms (7.3x)Correctness:
\bfind() divergences, greedy\Zspan, LazyDFA leftmost-start, generated-NFA findFrom literal jump, hybrid span re-match anchor-contextALLOW_JDK_FALLBACK(2 monster patterns)Benchmarks: re2j corpus lane + rust (JNI) IAST lanes added; benchmark corpus synced with the parity-test corpus (same TSV).
Docs: changelog for the full 0.4.0 arc.
Review hardening (adversarial multi-round review)
The whole branch was re-reviewed by a 5-round adversarial review pipeline (5 specialist roles + hypothesis-driven gap analysis, mutation/adversary verification; 305 regions, 83 files), producing 42 valid findings (0 critical, 0 high after verification). All 42 are addressed on this branch — 28 implemented in this pass, 14 verified already handled by earlier commits:
Runtime / resource lifecycle:
LITERAL_CACHE,COUNTED_LOOP_NFA_CACHE,HYBRID_CACHE) are size-capped via a shared insertion helper (clear-on-overflow,-Dreggie.compile.cacheMaxEntries, default 10k)ROUTING_NOTEThreadLocal cleared in the compilefinally;describeRouting()documented debug-only, capture cleared infinallyBackrefBacktrackMatcherper-thread workspace shrinks arena/stack when the last call used <1/4 of capacity; workspace ownership of returned capture arrays documented; memo load-factor safetyPrefilteringMatcherrequired-literal rejection now covers all entry points (match,matchesBounded,matchBounded,findBoundsFrom)Codegen correctness / robustness:
DFAUnrolledBytecodeGenerator: lookup-table emission enforced terminally (generateLookupTables+ completeness assert beforevisitEnd()); per-class 64KB table memory documentedDFASwitchBytecodeGenerator: generation-time assertion that boolean/MatchResult bounded flavors emit identical bucket rangesDeterministicChainBytecodeGenerator: documented proof that the remaining retried regions cannot capture (literal/class-only bodies perDeterministicChainInfo)RecursiveDescentBytecodeGenerator: R2 give-back buffer now amortized-doubling (BytecodeUtil.growIterEnd) instead of O(input-length) per callSubsetConstructor: OOM →StateExplosionExceptionand total-deadline clamp shared by both determinization entry points; deadline check on a dedicated counter immune to mixed increments; dead throwingcanReachGroupExitremovedParser / parity:
(?iu)/(?iU)\p{...}property paths now reject loudly viacheckUnicodeCaseFold; in-class\bstays U+0008 under(?U); escaped]inside classes no longer corruptscountGroupsHybridMatcher.findMatchFromenforces the leftmost invariant (NFA-half start must equal DFA start, else span re-match fallback)FallbackPatternDetectortestsEpsilonNodebeforeLiteralNodeat all sites;CharSetBMP-only(?U)limitation documented; stale calibration comments fixedMatchBudgetExceededExceptioncontract documented onReggieMatchercomputeRequiredLiteralparses at most twice (was three)Benchmark / rust lane:
IastRegexpBenchmarkno longer crashes the whole class when the rust library is unbuilt;MatchOperationBenchmarkdistinguishes "pattern refused" from "library missing" and gained a native-handle@TearDown;RealCorpusScanBenchmarknow fails fast on reggie/JDK oracle divergence// SAFETY:comments citingjni.hindices,rxLastErrorthread-local diagnostics for compile refusals, production-reuse warning on the 256MB NFA limit, deadfind()/embeddedmain()removed, Windows probing documentedTests: new
ReviewParityFixesTestpins escaped-bracket group counting,(?U)[\b]backspace parity,(?iu)/(?iU)property rejection, prefilter entry-point coverage, workspace-shrink stability and bounded-cache behavior;HybridFreshInstanceTestfails on timed-out workers;RequiredLiteralAuditTestbattery mismatches are hard assertions; deadline tests document their process-global property constraint.Verification:
./gradlew spotlessApply && ./gradlew buildgreen (3198+ tests, 0 failures),cargo checkgreen; implementation cross-check converged in 2 rounds.Motivation
java.util.regexremains a measurable CPU consumer in regex-heavy backend services, and the earlier re2j migration attempt never landed. This branch makes reggie the fastest option on every measured shape: 13x faster than re2j on matched and 2.3x on no-match at the corpus, and 2-4x ahead of the JDK end-to-end — with JDK-identical semantics enforced by the parity batteries.Related Issue(s)
None.
Change Type
Checklist
./gradlew build, incl.jacocoVerify)Performance Impact
Box-verified (idle 16-core Linux, JMH, same-run controls; full tables in
doc/temp/bench-*):Arc totals vs the 0.3.0 baseline: no-match sweep 15.6ms → 636µs (24x), matched 340µs → 180µs. Fuzz: 15 standing findings, 0 regressions; all four parity batteries at 0 divergences.
Additional Notes
doc/agents-fallback-and-limitations.md): 7 honest refusals (nullable-capture, anchor-in-quantifier, anchor-dilution, empty-class), matched-side span families (PikeVM capture-lists / BitState greedy spans) as the next work item.release.sh minor(0.4.0 publish) and consumer-side rollout PRs now that the engine is ready.