fix: linear backref must reject groups without a completed span (\1 OOM, #122)

[jbachorik]

What does this PR do?

Fixes #122. A backreference evaluated by the LINEAR_BACKREFERENCE strategy against a group that has no completed span (self- or forward-reference, e.g. (\1), (a\1)) computed a negative group length: groupEnds[group] was still the -1 initializer, the pos + groupLen > len bounds check passed, regionMatches vacuously succeeded, and pos += groupLen moved the match position backward. The matcher then produced bogus matches at every position and findAll never advanced — the reported ReDoS-like hang ending in OutOfMemoryError.

The fix: generateBackrefCheck now also fails when the group end is unset — a backref may only consult a completed group span, which is exactly java.util.regex's Pattern$BackRef behavior. (\1) and friends now return no match like the JDK; the negative-span path is unreachable so the backward-position corruption cannot recur.

Motivation

Issue #122 ((\1) OOM), reported against 0.3.0, blocking PCRE-parity expectations. Also fixes a silent wrong-answer divergence ((a\1) returned true where the JDK returns false) found while reproducing the OOM.

Related Issue(s)

Fixes #122. Related: #39 (self-referencing backreference support), #48.

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactoring (no functional change)
• Documentation
• Test improvement
• Build/CI change

Checklist

• I have read the CONTRIBUTING.md guidelines
• All existing tests pass (./gradlew build) — full build + cleanTest test (codegen/runtime/processor) + integration tests green
• I have added tests for my changes — BackrefSelfReferenceTest (JDK-differential: expected values computed from java.util.regex at runtime; covers self-refs, forward refs, quantified self-refs, nested, and valid-backref controls)
• I have updated documentation (if applicable) — doc/agents-fallback-and-limitations.md self-reference section
• My commits are signed

Performance Impact

None on valid backref patterns — verified by JMH BackreferenceBenchmark A/B on workspace-jb (Temurin 21.0.12-tem): all 17 reggie entries within ±0.6% of baseline. The end-group guard is emitted only for self-/forward-referencing backrefs and quantifier-body backrefs; plain backward references compile to byte-identical code.

Additional Notes

AI-assisted change.

Scope notes:

• The (\1) family routed to LINEAR_BACKREFERENCE — the issue's root cause is in LinearPatternBytecodeGenerator, not the RECURSIVE_DESCENT self-ref path.
• Pre-existing, out of scope (issue [pcre] Self-referencing backreferences not supported (e.g. (a\1?){4}) #39 audit item): under RECURSIVE_DESCENT (C-01/C-03 partial-open sentinel), a required self-reference ((\1+), (a\1+)) resolves to a zero-length match where the JDK (which resolves mid-iteration backrefs against the last completed span) returns no match. Optional self-references (\1?) behave equivalently to the JDK for match/no-match. Changing C-03 unilaterally would break the canonical ^(a\1?){4}$ acceptance cases on inputs like "aaaaaa". Documented in doc/agents-fallback-and-limitations.md.
• (\1) is also correctly rejected (JAVA_FALLBACK refusal) when it routes through nullable-group guards — unchanged behavior.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 81.81818% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.4%. Comparing base (b1937ae) to head (049de68).

Files with missing lines	Patch %	Lines
...odegen/codegen/LinearPatternBytecodeGenerator.java	81.8%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##              main    #134     +/-   ##
=========================================
- Coverage     85.4%   85.4%   -0.1%     
  Complexity       1       1             
=========================================
  Files          163     163             
  Lines        48740   48759     +19     
  Branches      6966    6970      +4     
=========================================
+ Hits         41635   41647     +12     
- Misses        5098    5104      +6     
- Partials      2007    2008      +1     

Files with missing lines	Coverage Δ
...odegen/codegen/LinearPatternBytecodeGenerator.java	91.2% <81.8%> (-0.8%)	⬇️

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1937ae...049de68. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-28T12:52:05.185940Z	049de68	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 049de688ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

More details

Completed-group tracking emits the unset-end guard for self, forward, and quantified backreferences while preserving the existing path for completed backward references.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 049de68 · @DataDog review to ask questions}