[Omkar-Wagholikar]

Fixes #543.

Problem

In the C/POSIX locale sed uses byte mode, which rejected every regex containing a back-reference:

    $ printf 'aba\n' | LC_ALL=C sed -n '/\(a\)b\1/p'
    sed: ... back-references are not supported in byte mode

GNU sed prints aba.

Fix

As @keith-hall suggested on #545, this uses the bytes support added in fancy-regex 0.19 instead of mapping bytes to chars and translating offsets:

• Bump fancy-regex 0.18 → 0.19 and adapt to Captures<'t, S> / CaptureMatches<'r, 't, S> (this is why Update Rust crate fancy-regex to 0.19.0 #503 fails CI).
• Add a Regex::FancyBytes variant compiled with BytesMode::Ascii and searched on chunk.as_bytes(). Ascii mode (. = any byte, ASCII-only classes) matches the semantics of the existing regex::bytes + unicode(false) byte path, so behaviour is consistent with or without back-references.
• Offsets come back as byte offsets into the original buffer, so no translation layer is needed.

Limitation

fancy-regex currently has no syntax for a raw non-ASCII byte: in bytes mode \xE9 is read as U+00E9 and matched as its UTF-8 encoding, and [\x80-\xFF] fails to parse in Ascii mode. So a back-reference pattern that also contains bytes ≥ 0x80 still errors in byte mode, now with an explicit message ("back-references combined with non-ASCII bytes are not supported in byte mode") rather than producing wrong matches. ASCII patterns work on any input, including invalid UTF-8. I'm happy to raise this upstream.

Testing

• New unit tests for FancyBytes (is_match, find offsets, captures with an unmatched optional group, captures_iter, and the non-ASCII rejection).
• New integration tests: the issue's address case, substitution, invalid UTF-8 input, and the non-ASCII rejection. They replace subst_backref_rejected_in_c_locale.
• cargo test, cargo clippy --all-targets -D warnings, cargo fmt --check, and cargo check in fuzz/ all pass.
• Byte-for-byte differential runs against GNU sed 4.9 with LC_ALL=C (back-references in addresses and substitutions, g/N-th/I flags, multi-line pattern space, invalid UTF-8 input): all identical.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.98%. Comparing base (3b464d1) to head (b046ee3).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #602      +/-   ##
==========================================
+ Coverage   94.85%   94.98%   +0.12%     
==========================================
  Files          14       14              
  Lines        7230     7357     +127     
  Branches      432      433       +1     
==========================================
+ Hits         6858     6988     +130     
+ Misses        371      368       -3     
  Partials        1        1              

Flag	Coverage Δ
macos_latest	96.50% <100.00%> (+0.10%)	⬆️
ubuntu_latest	96.43% <100.00%> (+0.10%)	⬆️
windows_latest	0.00% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed]

Merging this PR will regress 1 benchmark

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 7 improved benchmarks
❌ 1 regressed benchmark
✅ 3 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
❌	access_log_append	1 s	1.1 s	-2.64%
⚡	number_fix	1,293.4 ms	635.5 ms	×2
⚡	access_log_no_del	900 ms	831.5 ms	+8.23%
⚡	access_log_all_del	932.2 ms	863.8 ms	+7.92%
⚡	remove_cr	432.7 ms	407.4 ms	+6.19%
⚡	access_log_subst	2.2 s	2.1 s	+5.86%
⚡	access_log_no_subst	908.8 ms	861.9 ms	+5.45%
⚡	genome_subst	389.9 ms	381.2 ms	+2.28%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing Omkar-Wagholikar:fix/543-byte-mode-backrefs (b046ee3) with main (3b464d1)}

[Omkar-Wagholikar]

Will look into the performance regression

[sylvestre]

perf-dfa-full is unrelated to back-references, please split that into another pr.
it also impacts the binary size, please give numbers there

[Omkar-Wagholikar]

This isn't exactly unrelated, the fancy-regex 0.19 dropped regex-automata/dfa (0.18 enabled it), so the bump essentially disables regex's full DFA.

Without adding the perf-dfa-full flag, a 1M+line long s/[0-9]+.[0-9]+…/IP/ run goes from ~0.75s to ~10s.

Binary size (release, x86_64 Linux): main 3,214,528 B; this PR without the flag 3,128,960 B; with the flag 3,302,656 B (+88 KB vs main which is ~3% bigger).

[sylvestre]

The CodSpeed regression you said you'd look into: please post what you found, or a fix. :)

[sylvestre]

@codspeedbot fix this regression

[codspeed]

@codspeedbot fix this regression

I fixed the regression and pushed two commits to this branch: a merge of main and b046ee3.

Cause

The genome_subst slowdown wasn't in the new byte-mode code, which only runs for patterns with back-references. It came from re_or_saved_re() in processor.rs, which made a full copy of the current regex for every input line so it could be reused later as the empty regex (//). Adding the FancyBytes variant made that copy and its cleanup a bit more expensive, and the compiler stopped inlining the copy in the hot loop. That added about 2–3% to genome_subst.

The per-line copy was already expensive before this PR. For regex::bytes::Regex, each copy allocates a new matcher cache pool on every line.

Fix (b046ee3)

Script regexes (Address::Re, Substitution::regex) and ProcessingContext::saved_regex are now Rc<Regex>. Saving the last regex now just increments a reference count, with no allocation or copy. Behaviour is unchanged: empty // still uses the last regex used at run time.

Results (CodSpeed simulation, compared with current main on the same machine)

Benchmark	main	this PR	Change
number_fix	574.7 ms	269.9 ms	×2.1
access_log_no_del	426.8 ms	392.4 ms	+8.8%
access_log_all_del	443.5 ms	409.3 ms	+8.4%
remove_cr	207.0 ms	191.3 ms	+8.2%
access_log_no_subst	429.9 ms	398.8 ms	+7.8%
genome_subst	194.7 ms	187.7 ms	+3.7%
access_log_subst	1,015 ms	984.9 ms	+3.1%

The other 4 benchmarks are unchanged, and none regressed.

cargo test, cargo clippy --all-targets -D warnings, cargo fmt and cargo check in fuzz/ all pass. I also checked that the empty regex picks up the right regex across several commands and matches GNU sed.

The earlier CodSpeed report also compared runs on different CPUs, which makes small differences less reliable. The CI run on this push should show the new numbers.

If you'd rather have the Rc change in its own PR, comment @codspeedbot move the Rc change to a separate PR and I'll do it.

Summary

• Warming up the performance wizard...
• Investigate genome_subst regression — Cause: the per-line Regex clone in re_or_saved_re got more expensive and was no longer inlined once the FancyBytes variant was added
• Fix the regression — Merged main and changed script REs to Rc, so saving the last RE is now a reference count increment
• Validate with CodSpeed simulation — Compared with main: 7 benchmarks faster (number_fix ×2.1, genome_subst +3.7%), 0 regressions
• Push and report — Pushed b046ee3
• Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}

[Omkar-Wagholikar]

Hi @sylvestre! I was wondering if you’d be comfortable chatting a bit more informally on Discord, Twitter, or another platform you prefer. I hope you don’t mind me asking!

[sylvestre]

Sure I am on discord uutils or element

[Omkar-Wagholikar]

Hi @sylvestre , messaged you on discord under the username: sylvestre__