feat: lazy min>0 scan loops in deterministic chain with capture snapshot/restore

.gitignore

@@ -70,3 +70,6 @@ reggie-benchmark/com.datadoghq.reggie.benchmark*/
 .cachebro/
 .worktrees/
 reggie-benchmark/rust-regex-engine/target/
+
+# Transient cairn session wrap-ups (never committed)
+notes/

.sphinx/teach-report.json

@@ -1,41 +1,37 @@
 {
-  "generated_at": "2026-06-28T00:00",
-  "source": "pr:89",
+  "source": "DataDog/java-reggie#136 review feedback (chatgpt-codex-connector[bot] P1 inline comment; datadog-datadog-prod-us1[bot] Bits P2 inline comment)",
+  "generated_at": "2026-09-29T15:20:00Z",
   "improvements": [
     {
-      "id": "3487413781",
-      "source_comment": "**P1** Avoid silently dropping per-config backref branches\n\nFor assertion-free backref NFAs this path is now enabled unconditionally by `perConfigEligible()`, but overflow still jumps to `full` and drops the configuration. Patterns with more than 256 live branches, such as a large alternation backref like `(a|aa|...|a{300})\\1` on an input requiring the long branch, can therefore miss the only successful configuration and return a silent false negative instead of falling back or growing the worklist.",
-      "source_file": "reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/codegen/NFABytecodeGenerator.java",
-      "source_line": 419,
+      "status": "accepted",
       "classification": "project_lens",
-      "rationale": "The pattern requires knowledge of this project's per-config worklist cap (256), `perConfigEligible()` predicate, and the JDK fallback delegation contract — none of which are expressible without project-specific types.",
+      "source_file": "reggie-processor/src/main/java/com/datadoghq/reggie/processor/ReggieMatcherBytecodeGenerator.java",
+      "source_line": 469,
+      "source_comment": "For DFA_UNROLLED_WITH_ASSERTIONS, emitting this helper only repairs findMatchFrom/findAll; DFAUnrolledBytecodeGenerator.generateFindBoundsFromMethod implements a separate inline scan and never calls the helper. Consequently literal replaceAll and split still silently return incorrect results (a+(?=b) on aab -> aab instead of _b).",
+      "rationale": "The original fix audited only the emitter whose missing-helper call crashed; a sibling emitter in the same strategy family had its own inline reimplementation of the same operation and kept the wrong-result bug. Generated-code dispatchers have many parallel emitters per strategy arm; a helper fix is complete only when every emitter that references the helper directly OR reimplements its operation inline is covered, and pinned by a differential rich-API test.",
       "proposed": {
-        "theme": "perconfig-worklist-overflow-silent-drop",
-        "anti_pattern": "When the per-config worklist overflows its bounded capacity, execution jumps to the full-match label and silently drops the overflowed configuration, returning a false negative instead of delegating to the JDK fallback.",
-        "fix_example": "On overflow, emit a JDK-delegation bytecode path (or throw a controlled signal) rather than jumping to `full`, so no matching configuration is silently discarded.",
-        "root_cause": "The overflow guard was written before the per-config path was ungated; once `perConfigEligible()` enables the path unconditionally, any pattern that generates more than PER_CONFIG_CAP branches can trigger the silent drop on inputs that require a branch beyond the cap.",
+        "theme": "helper-fix-must-audit-sibling-emitters-with-inline-reimplementations",
+        "anti_pattern": "A missing-shared-helper fix in a generated-code dispatcher is validated against the one code path that crashed (here findMatchFrom/findAll), while a sibling generator in the same strategy family (DFAUnrolledBytecodeGenerator.generateFindBoundsFromMethod) keeps its own inline implementation of the same operation with an independent bug (assertion-blind scan that aborts on the first failed lookahead), so replaceAll/split silently return wrong results after the crash is fixed.",
+        "fix_example": "After repairing the dispatcher emission, enumerate every emitter in the affected strategy arms that either calls the helper or reimplements its operation inline, route each assertion-bearing one through the shared assertion-correct primitive (generateFindBoundsFromLongestEnd delegating to findLongestMatchEnd), and pin each with a differential JDK-parity test case (a+(?=b) on aab) in RichApiDifferentialTest.",
+        "root_cause": "Generated-code strategy dispatchers multiply each semantic operation across several independent emitters; the crash surfaced only the missing-emission instance, and the wrong-result instance lived in a sibling class the crash trace never mentions.",
         "confidence": "high",
-        "source": "teach"
-      },
-      "status": "accepted"
+        "languages": [
+          "java"
+        ]
+      }
     },
     {
-      "id": "3487413783",
-      "source_comment": "**P1** Continue scanning nullable prefixes for overlap\n\nThis `break` only checks the immediate next non-anchor sibling, so an overlapping greedy prefix is missed when another nullable prefix sits before the capture. For example `a*b*(a+)\\1` now routes to `VARIABLE_CAPTURE_BACKREF` with no fallback; the generated prefix matcher greedily consumes all `a`s in `a*` and never backtracks across `b*`, so `matches(\"aaaa\")` fails even though JDK can leave `aa` for the capture/backref. Keep scanning through nullable/disjoint prefix nodes, or compute the first charset of the whole suffix before allowing this strategy.",
-      "source_file": "reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/analysis/FallbackPatternDetector.java",
-      "source_line": 1186,
-      "classification": "generic_lens",
-      "rationale": "A break that exits a scan loop on the first non-target sibling instead of continuing until the target is found is a language-agnostic algorithmic error expressible without any project-specific types.",
+      "status": "accepted",
+      "classification": "consul_pass",
+      "source_file": "reggie-codegen/src/main/java/com/datadoghq/reggie/codegen/codegen/NFABytecodeGenerator.java",
+      "source_line": 9940,
+      "source_comment": "For assertion-bearing NFAs with competing valid match lengths, delegating bounds to findMatchFrom selects the longest end instead of Java's leftmost-first alternative ((a|aa)(?=a)) on aaa therefore replaces to _a rather than __a.",
+      "rationale": "The fix delegated bounds extraction to an existing shared primitive (findMatchFrom) whose candidate-end scan picks the longest end; the delegated primitive cannot express Java's leftmost-first priority for competing match lengths, so the fix traded one divergence (no-match) for another (longest-first). A reviewer pass is needed: any fix that delegates to a shared search/matching primitive must differentially verify the primitive's tie-breaking semantics, and a primitive that structurally cannot express the required priority must lead to refusal-plus-documentation rather than silent divergence.",
       "proposed": {
-        "theme": "scan-loop-break-on-non-target-sibling",
-        "anti_pattern": "A loop that searches a sequence for a target element uses `break` (or `return`) on the first element that is not an explicit skip condition, exiting before reaching the actual target and producing a false 'not found' result.",
-        "fix_example": "Replace `break` with `continue` (or remove it) so the loop advances past intermediate non-target elements and tests every element in the sequence before concluding the target is absent.",
-        "root_cause": "Confusing 'this element is not the target' with 'the target cannot exist later in the sequence' causes the scan to terminate prematurely, missing elements that follow unrelated intermediate nodes.",
-        "confidence": "high",
-        "languages": ["java"],
-        "frameworks": []
-      },
-      "status": "accepted"
+        "pass_name": "verify-delegation-tie-breaking-semantics",
+        "pass_text": "When a fix delegates to a shared search/matching primitive, differentially verify the primitive's tie-breaking semantics (leftmost-first vs leftmost-longest, shortest vs longest group split) against the reference implementation with adversarial inputs that have competing valid matches. If the primitive structurally cannot express the required priority, do not accept the delegation: prefer an analyzer-level refusal (documented divergence) or a priority-aware engine, and record the limitation in-code at the delegation site.",
+        "source": "user"
+      }
     }
   ]
-}
+}

doc/2026-09-01-deterministic-chain-bytecode-design.md

@@ -49,7 +49,8 @@ Elem     := Lit | Class1 | GreedyLoop | LazyLoop | OptChain | LitAlternation | C
 Lit      := string literal (1+ chars)
 Class1   := single char-class consume (exactly one char)
 GreedyLoop:= CharClass (min>=1 | min>=0), max=-1, greedy
-LazyLoop := CharClass (min=0, max=-1), lazy
+LazyLoop := CharClass (min>=0, max=-1), lazy — min > 0 ({@code x+?}) pre-rolls min mandatory
+            chars before the first tail try; k = min, min+1, … (JDK lazy order)
 OptChain := non-capturing sub-chain with quantifier {0,1}     -- two-attempt, all-or-nothing
 LitAlternation := alternation of Lits                          -- sequential tries, priority order
 Capture  := capturing group around a contiguous sub-chain (start/end recorded)
@@ -72,7 +73,11 @@ Admission rules (each is a linearity or correctness requirement, checked at dete
    tail's first chars), else advance. Tail tries fail in O(1) on the gate; each scan position is
    visited once → O(n) per try-start. `.*?` (ANY-class) is admitted; captures in the tail are
    rewritten by each try (last write wins — correct because the winning try rewrites start and
-   end before returning).
+   end before returning). **Every failed try restores the capture snapshot taken at loop entry**
+   (the lazy-predicate rejection — POS_EQ_LEN / END_ANCHOR failing after a structurally
+   successful tail — bypasses the tail's own OPT retry/restore hygiene, so `emitLazyLoop`
+   snapshots at entry and restores on `tailFail`; without it `^(\S+?)(?::(\d+))?$` on
+   `relative:12x` leaks g2=`12` from the predicate-rejected try into the winning skip path).
 4. **Chain min-width ≥ 1 for unanchored branches** (v1): empty-matching branches
    (`(a*b*c*d*e*)`) decline — BitState keeps them (already fast-pathed 2.3x).
 5. **No backreferences, no lookaround, no `\b`** — hard declines (RecursiveDescent /

docs/sphinx/specs/2026-09-30-address-pr-138-multi-lazy-loop-and-benchmark-input.md

@@ -0,0 +1,43 @@
+# Spec: address-pr-138-multi-lazy-loop-and-benchmark-input
+
+## Problem
+Review feedback on PR #138 (lazy-min scan-loop work) flagged two defects: (1) the
+capture-heavy benchmark input `LazyScanLoopBenchmark.DOTTED_TEXT` places its first dot at
+index 1, so `^(.+?)\.` succeeds on its first tail try and never executes
+`emitRestoreCaptures` — the benchmark cannot catch regressions in the O(g)-per-failed-try
+capture-restore path it claims to measure; (2) `parseChainQuantifier` admits every unbounded
+lazy char-class quantifier independently, so the deterministic-chain detector routes patterns
+with multiple lazy scan loops (e.g. `a+?a+?b`) off the linear BitState route onto chain
+bytecode where every position of the outer scan re-runs the inner scan to the end — quadratic
+per try, and the `matches`/`match` entry points carry no work budget because `LAZY_LOOP`
+never sets `hasGiveBack`.
+
+## Correct behaviour
+- `DOTTED_TEXT = "abcdefghijklmnopqrstuvwxyz".repeat(8) + "."` (single 208-char dot-free
+  token + trailing dot): every lazy tail try k = 1..207 for `^(.+?)\.` fails and pays the
+  per-try capture snapshot/restore before the winning try at k = 208; the field javadoc states
+  the new intent; the JMH methods and annotations are unchanged.
+- `detectDeterministicChain` enforces the grammar's one-lazy-loop invariant per ChainBranch
+  tree (branch seq, nested OPT/CAPTURE seqs, ALT_CHAIN alternative seqs; LOOP_ALT bodies
+  counted for future-proofing): a branch tree containing 2+ `LAZY_LOOP` elements declines the
+  pattern (returns null), keeping it on the linear BitState/PikeVM route. One lazy loop per
+  branch stays admitted (`a+?x|b+?y` remains routed to the chain generator; each branch try is
+  a single O(n·w) scan and find-family methods always carry the work budget + PikeVM
+  fallback).
+- The `detectDeterministicChain` javadoc records that the invariant is enforced and why (a
+  second lazy loop inside a lazy tail is O(n^2) on the budget-free matches/match paths).
+- Pre-existing detector and routing pins hold unchanged: `a+?x`, `a*?x`, `a{1024,}?x`,
+  `a+?b`, `a*?b`, `^(.+?)\.([^/]+)` admitted; `a{2,4}?x`, possessive/atomic shapes declined;
+  `a*?b`/`a+?b` still route to DETERMINISTIC_CHAIN_BYTECODE.
+
+## Test plan
+- `DeterministicChainDetectorTest`: new `multipleLazyScanLoopsDeclined*` tests —
+  `a+?a+?b`, `a*?b*?c`, `^(a+?)b+?c` decline (each fails on pre-fix HEAD, where the detector
+  admits these chains); `a+?x|b+?y` admitted with exactly one LAZY_LOOP per branch (scope pin
+  against an over-broad whole-pattern regression).
+- Existing suite green: `DeterministicChainDetectorTest`, `StrategySelectionTest`,
+  `StrategySelectionExtendedTest`, `IastPatternRoutingTest`, then `./gradlew build`.
+
+## Out of scope
+- Generator-side budgeting for multi-lazy patterns (declined by detection instead).
+- The stale `.sphinx/address/benchmark-evidence.md` javadoc reference (transient notes path).

docs/sphinx/specs/2026-09-30-lazy-min-chain-review-followups.md

@@ -0,0 +1,79 @@
+# Spec: lazy-min-chain-review-followups
+
+## Problem
+The uncommitted lazy-min extension (LAZY_LOOP `min > 0` admitted into the deterministic-chain
+family; capture snapshot/restore + mandatory pre-roll in `emitLazyLoop`) passed review with 12
+findings: one file-placement violation (`notes/` untracked but not git-ignored), two
+performance/benchmark-evidence requests for the routing flip, and the rest test gaps or missing
+documentation around the new admission boundary, first-set/emptyPrefix contract, min-width
+accumulation, and capture-hygiene behavior. Production behavior is correct (all existing tests
+pass; JDK ground truth verified); the work is tests, comments, gitignore, and benchmark evidence.
+
+## Correct behaviour
+- `notes/` is git-ignored; the transient wrap-up file is never committed.
+- The admission boundary is pinned by tests: `min == MAX_CHAIN_LOOP_BOUND` (1024) admitted with
+  elem min 1024 / branch minWidth 1025 and strategy DETERMINISTIC_CHAIN_BYTECODE;
+  `min == MAX_CHAIN_LOOP_BOUND + 1` declined (detector null); `min == 0` still admitted.
+- First-set/emptyPrefix contract pinned: `a+?b` → branch firstSetAscii = {a} only;
+  `a*?b` → {a, b}; `a{2,}?x` → elem min 2, branch minWidth 3.
+- Runtime parity vs java.util.regex (both the real `Reggie.compile` route and the direct
+  compileChain harness) for: `\d+?` (full successive-find span sequence + no-match input),
+  `a+?b` (tail-try-at-min both directions: "ab", "aab", "xaab"),
+  `^([^\s]+?)(?::([0-9]+))?$` (capture hygiene: "relative:12x" → g2 null; "relative:12" →
+  g1="relative", g2="12"), `^(.+?)\.([^/]+)` (capture-wrapped empty-tail shape).
+- `emitLazyLoop` documents (a) why pre-roll failures jump to failLabel without restoring the
+  capture snapshot (every branch try is preceded by emitResetCaptures, so stale slots are
+  unobservable), and (b) why the full-slot restore per failed tail try is acceptable cost.
+- Benchmark evidence exists for the re-routed lazy shapes (`\d+?`, `<.+?>`, `^(.+?)\.`):
+  reggie vs JDK throughput, before (BITSTATE route) and after (chain route), with the numbers
+  recorded; the chain route must stay within 1.5x of the JDK baseline per shape.
+
+## Constraints
+- Existing route pins (`a*?b`, `<.+?>`, `a+?b`, `\d+?` → DETERMINISTIC_CHAIN_BYTECODE) unchanged;
+  admission condition `q.min >= 0 && q.min <= MAX_CHAIN_LOOP_BOUND` unchanged.
+- Repo rules: `spotlessApply` before commit; `pushInt` for int constants; tests live in the
+  existing suites (DeterministicChainDetectorTest, StrategySelectionExtendedTest,
+  PikeVmCaptureRegressionTest, DeterministicChainV3BytecodeTest).
+- Benchmark is a new JMH class in reggie-benchmark, run manually via the `jmh` gradle task with
+  short iterations; before-run measured with ONLY the two production files stashed.
+
+## Scope
+
+### Primary fixes
+- F-2c170e02c9b1: `.gitignore` — add `notes/` (transient session wrap-ups must not be committed).
+- F-ac9d70c1902d: PatternAnalyzer.java:10839 boundary — tests at min=1024 (admitted) /
+  min=1025 (declined) / min=0 (admitted) + strategy pin for the admitted boundary case.
+- F-77e4bcc2a40a: PatternAnalyzer.java:10841 — targeted JMH benchmark + before/after evidence.
+- F-656d11bf48c5: PatternAnalyzer.java:11149 — reviewer's fall-through claim is false (code
+  breaks); fix = comment documenting the mandatory-consume break contract + first-set test.
+- F-8e5397f23c99: PatternAnalyzer.java:11072/11152 — first-set/emptyPrefix output tests for
+  `a+?b` / `a*?b`.
+- F-85b1d768cfb6: PatternAnalyzer.java:11279 — min-width test for `a{2,}?x` (min==2,
+  minWidth==3) + runtime parity on a short input.
+- F-ab12c5895b5e: DeterministicChainBytecodeGenerator.java:2381 — behavioral parity test for
+  the capture-hygiene shape (g2 null on "relative:12x"; groups vs JDK), both routes.
+- F-f45daa6be9e7: DeterministicChainBytecodeGenerator.java:2389 — pre-roll parity test for
+  `a+?b` on min+tail / min+1+tail inputs.
+- F-f3f00ed53217: DeterministicChainBytecodeGenerator.java:2413 — comment documenting the
+  failLabel invariant + the capture-hygiene test covers the outer-retry-succeeds scenario.
+- F-c3d19c25acd2: DeterministicChainBytecodeGenerator.java:2444 — cost-analysis comment at the
+  restore site + shared benchmark evidence (full restore kept; per-slot optimization declined).
+- F-8b693a4cec8e: StrategySelectionExtendedTest.java:306 — runtime behavioral tests for the
+  empty-tail lazy shapes (placed in PikeVmCaptureRegressionTest; see tester plan).
+- F-93c9337a4dd9: PikeVmCaptureRegressionTest.java:176 — `\d+?` behavioral parity +
+  predicate-rejection capture case via the real route.
+
+### Auto-expanded sibling fixes
+None identified. Audit performed: the three analyzer LAZY_LOOP sites (checkChainDisjoint ~11072,
+computeChainFirst ~11147, chainSeqMinWidth ~11278) are the only `e.min` consumers of the new
+admission and are all covered by the planned tests; the generator's only LAZY_LOOP emitter is
+`emitLazyLoop`; ChainElem's structural hash already mixes `e.min` (PatternAnalyzer ~8325, pinned
+by the existing min=1/min=0 collision test); GREEDY_LOOP/LOOP_ALT `min > 0` handling predates
+this change and is exercised by existing greedy/loop-alt tests.
+
+## Assumptions
+- A targeted JMH subset (new LazyScanLoopBenchmark, short iterations) is acceptable benchmark
+  evidence; the full 322-benchmark suite before AND after is out of scope for this pass.
+- JDK per-digit successive-find semantics for `\d+?` (measured: [0,1) [1,2) [2,3) [6,7) [7,8)
+  [8,9) on "123abc456") is the parity target, not the reviewer's digit-run-span guess.
+- `notes/` stays on disk locally; only the .gitignore rule is added.