sed: implement \\U \\L \\u \\l \\E case-conversion escapes in s/// by dedsec-terminal · Pull Request #557 · uutils/sed

dedsec-terminal · 2026-09-15T19:32:33Z

Fixes #540.

Implements GNU s/// replacement case-conversion escapes:

\U uppercase until \L/\E, \L lowercase until \U/\E
\u/\l one-shot next-character conversion
\E ends persistent conversion and clears pending one-shot

Details:

New ReplacementPart::{Upper,Lower,UpperFirst,LowerFirst,End} in command.rs; compile_replacement parses them before parse_char_escape so \u/\U in replacements are case-conversion directives rather than \uXXXX/\UXXXXXXXX Unicode escapes (matches GNU sed behavior; README updated accordingly). Escaped delimiter still wins (e.g. sU...U\UU).
Runtime append_with_case applies persistent + one-shot to literals, &, \1..\9. Empty matches leave one-shot pending (GNU s/(b?)-/\u\1x/g carry), any produced char consumes it. Fresh state per substituted occurrence so g does not propagate.
Factored out case selection into take_case(single, persistent) helper.
UTF-8 mode is Unicode-aware with invalid-byte passthrough; byte mode is ASCII-only.

Verified against GNU sed 4.9 including the issue table (ABC DEF, ABc def) plus \U&\E, \U\1-\2, empty-group g cases, \u\l/\l\u last-wins, \U\l/\L\u combos.

Tests: cargo test --lib 385 passed, cargo test --test tests 268 passed, cargo clippy --lib clean, cargo fmt --check clean.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-09-16T05:08:42Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

codecov · 2026-09-16T05:09:42Z

Codecov Report

❌ Patch coverage is 99.03537% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.13%. Comparing base (3b464d1) to head (04e82e0).

Files with missing lines	Patch %	Lines
src/sed/command.rs	99.07%	5 Missing ⚠️
src/sed/compiler.rs	98.64%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #557      +/-   ##
==========================================
+ Coverage   94.85%   95.13%   +0.27%     
==========================================
  Files          14       14              
  Lines        7230     7809     +579     
  Branches      432      457      +25     
==========================================
+ Hits         6858     7429     +571     
- Misses        371      379       +8     
  Partials        1        1

Flag	Coverage Δ
macos_latest	`96.56% <99.03%> (+0.17%)`	⬆️
ubuntu_latest	`96.50% <99.03%> (+0.17%)`	⬆️
windows_latest	`0.00% <0.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codspeed · 2026-09-16T05:11:56Z

Merging this PR will regress 1 benchmark

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 8 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`number_fix`	1.3 s	1.3 s	-2.16%
⚡	`genome_subst`	389.9 ms	367.4 ms	+6.11%
⚡	`access_log_subst`	2.2 s	2.1 s	+3.86%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing dedsec-terminal:fix-540-case-conversion-escapes (04e82e0) with main (3b464d1)}

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

dedsec-terminal · 2026-09-16T20:19:56Z

Hi @dspinellis — friendly review request on this s/// case-conversion implementation (fixes #540; GNU suite +1, utf8-ru). Just pushed a perf follow-up (fast paths when no conversion directives; 265 tests green, fmt+clippy clean) to address the Codspeed delta. Let me know if you need anything else. Thanks!

github-actions · 2026-09-17T06:53:11Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

dedsec-terminal · 2026-09-17T18:32:55Z

Follow-up pushed (23b7709): patch coverage for the fast paths - unit tests for unmatched-group/single-literal fast paths, flag-sync contract panics, LowerFirst invalid-byte handling, compiler case-directive parsing + flag, and end-to-end substitution tests; plus removal of two dead None arms preempted by the all-None early return. Full suite green (lib + integration), fmt/clippy clean. Remaining reds are not from this change: Android jobs fail in sdkmanager setup, and CodSpeed shows -3.75% on number_fix only (genome/access_log improved; likely noise, happy to dig if you want).

github-actions · 2026-09-18T05:22:20Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-09-22T06:16:03Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-09-22T08:28:38Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-10-14T04:41:11Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 20 / FAILED: 37 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 18 / FAILED: 39 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + subst-replacement
  + utf8-ru

…ixes uutils#540)

Addresses Codspeed -7.9pct regression: common case (no U/L/u/l/E) now plain memcpy via has_case_conversion flag; byte-mode hoists the branch; UTF-8 path validates once per segment; ASCII case uses to_ascii_* fast path. All 265 tests pass; fmt+clippy clean.

Add unit tests for the no-conversion fast paths (unmatched groups, single literals, flag-sync contracts), LowerFirst invalid-byte handling, compiler case-directive parsing and the has_case_conversion flag, plus end-to-end substitution coverage. Collapse two provably-dead None arms preempted by the all-None early return. Deduplicate GNU testsuite bot comments.

- revert unrelated GnuComment workflow modifications - update README to note \u/\U replacement behavior - consolidate case escape mapping in compiler - factor out take_case helper and shorten comments in command.rs

…placement

- Encapsulate ReplacementTemplate.has_case_conversion with a private field and public getter - Replace panic arm in render_parts with unreachable!() and remove artificial panic tests - Drop backslash for unrecognized escapes in replacement strings to match GNU sed - Disable GNU case-conversion escapes in replacement strings under POSIX mode - Add unit and integration regression tests for unrecognized escapes and POSIX mode

…ment

…bytes

…n escapes

…own escape backslashes

github-actions · 2026-10-14T04:54:26Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 20 / FAILED: 37 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 18 / FAILED: 39 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + subst-replacement
  + utf8-ru

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

github-actions · 2026-10-18T18:31:05Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 19 / FAILED: 38 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 18 / FAILED: 39 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

Copilot AI lite review requested due to automatic review settings September 15, 2026 19:32

Copilot AI reviewed Sep 15, 2026

dedsec-terminal mentioned this pull request Sep 15, 2026

s/// replacement: implement the \U \L \u \l \E case-conversion escapes #540

Open

Copilot AI review requested due to automatic review settings September 16, 2026 17:26

Copilot AI reviewed Sep 16, 2026

Copilot AI review requested due to automatic review settings September 16, 2026 20:19

Copilot AI reviewed Sep 16, 2026

Copilot AI review requested due to automatic review settings September 17, 2026 18:10

Copilot AI reviewed Sep 17, 2026

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread .github/workflows/GnuComment.yml Outdated

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread src/sed/compiler.rs Outdated

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread src/sed/compiler.rs Outdated

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread src/sed/command.rs Outdated

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread src/sed/command.rs

Copilot AI review requested due to automatic review settings September 22, 2026 06:10

Copilot AI reviewed Sep 22, 2026

Copilot AI review requested due to automatic review settings September 22, 2026 08:03

Copilot AI reviewed Sep 22, 2026

Copilot AI review requested due to automatic review settings September 22, 2026 09:26

Copilot AI reviewed Sep 22, 2026

sylvestre reviewed Sep 22, 2026

View reviewed changes

Comment thread src/sed/command.rs Outdated

Copilot AI lite review requested due to automatic review settings October 14, 2026 04:33

Copilot AI reviewed Oct 14, 2026

Copilot AI lite review requested due to automatic review settings October 14, 2026 04:35

Copilot AI reviewed Oct 14, 2026

dedsec-terminal and others added 12 commits October 14, 2026 10:12

sed: implement \\U \\L \\u \\l \\E case-conversion escapes in s/// (f…

fdcc6fb

…ixes uutils#540)

style: apply cargo fmt to case-conversion code

c915374

refactor: address review feedback on case-conversion escapes

cf9233e

- revert unrelated GnuComment workflow modifications - update README to note \u/\U replacement behavior - consolidate case escape mapping in compiler - factor out take_case helper and shorten comments in command.rs

perf(subst): avoid intermediate buffer allocations in substitution re…

c8b57a4

…placement

refactor(subst): address review feedback on case-conversion escapes

2a73c9b

fix(subst): preserve delimiter precedence for digits and & in replace…

2610ef9

…ment

fix(subst): preserve persistent case conversion across invalid UTF-8 …

f438e81

…bytes

refactor(subst): address maintainer review feedback on case-conversio…

17451d2

…n escapes

fix(subst): restore fast paths, encapsulate case flags, and drop unkn…

f4d6370

…own escape backslashes

dedsec-terminal force-pushed the fix-540-case-conversion-escapes branch from 704f253 to f4d6370 Compare October 14, 2026 04:43

sylvestre reviewed Oct 18, 2026

View reviewed changes

Comment thread src/sed/command.rs Outdated

sylvestre reviewed Oct 18, 2026

View reviewed changes

Comment thread src/sed/command.rs Outdated

sylvestre reviewed Oct 18, 2026

View reviewed changes

Comment thread src/sed/command.rs Outdated

sylvestre reviewed Oct 18, 2026

View reviewed changes

Comment thread src/sed/compiler.rs

refactor(subst): address latest maintainer review feedback

c2e304a

Copilot AI lite review requested due to automatic review settings October 18, 2026 17:41

Copilot AI reviewed Oct 18, 2026

Merge branch 'main' into fix-540-case-conversion-escapes

04e82e0

Copilot AI lite review requested due to automatic review settings October 18, 2026 17:44

Copilot AI reviewed Oct 18, 2026

Conversation

dedsec-terminal commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

codecov Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 1 benchmark

Performance Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

dedsec-terminal commented Sep 16, 2026

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

dedsec-terminal commented Sep 17, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

dedsec-terminal commented Sep 15, 2026 •

edited

Loading

codecov Bot commented Sep 16, 2026 •

edited

Loading

codspeed Bot commented Sep 16, 2026 •

edited

Loading