Repository navigation
lexer: encode \u escapes using locale encoding (rebase) - #1
Closed
Franklin-Qi wants to merge 20 commits into
Closed
Franklin-Qi wants to merge 20 commits into
Franklin-Qi wants to merge 20 commits into
Conversation
Expand test coverage for the parser and lexer using existing patterns: test_parser! macro assertions for AST s-expressions and token-slice comparisons for the lexer. Closes: uutils#43
* interpreter: widen instructions, immediates * lexer, parser: add tokens and nodes for built-in functions * chore: fix CI --------- Co-authored-by: yor1xd <gabrielhenriquenf1@gmail.com>
Validate namespace and literal parts of ns::name against gawk's reserved keyword list when building identifiers, including @namespace directives. Closes: uutils#37
- Track an anchor at the start of each Pratt subexpression and report errors from that anchor through the offending token, instead of highlighting only the final token. - Apply this to assignment/index/place errors, prefix operators, unclosed parentheses and brackets, and related Pratt paths. Keep narrow spans for non-associative operators (e.g. chained = ). - Add unit tests that assert the highlighted source snippets for common parse failures. Closes: uutils#42 Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>
…orced-non-keyword-names-on-namespaced-identifiers parser: reject reserved keywords in qualified identifiers
The Pratt parser now handles subtraction vs. unary negative.
Detect charset from LC_ALL/LC_CTYPE/LANG and encode \u sequences into the locale multibyte encoding (UTF-8, ISO-8859-1, ASCII-only for C/POSIX), matching gawk. Unrepresentable or invalid code points become '?'. Closes: uutils#40
cursor
Bot
force-pushed
the
cursor/pr51-locale-encoding-rebase-cfaf
branch
from
June 30, 2026 12:59
be7c222 to
4105912
Compare
|
EDIT: nvm I thought this was the main repo, got pinged somehow. |
Owner
Author
@Alonely0 Thank you for your feedback. Regarding my issue, the current log is a bit messy, I'm considering closing this PR for now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebases uutils/awk#51 onto current
mainand addresses review feedback.Changes
main(resolves merge conflicts with merged PRs Add unit tests for parser and lexer uutils/awk#49, parser: extend Pratt error spans to cover subexpressions uutils/awk#50, parser: reject reserved keywords in qualified identifiers uutils/awk#52)LC_ALL/LC_CTYPE/LANGviavar_os()with UTF-8 fallback toC.UTF-8\usequences into locale multibyte encoding (UTF-8, ISO-8859-1, ASCII-only for C/POSIX), matching gawkVecallocation)char::try_fromfor code point validationLocaleEncoding::from_locale_nameReview feedback addressed
from_locale_nameis now aLocaleEncodingconstructor methodis_asciiclosureencode_unicode_escapewrites into caller-provided buffer viaExtend<u8>char::try_fromreplacesunwrap()for validationvar_os()used for locale environment variablesTesting
cargo test -p lexer— 45 tests passedcargo test— all workspace tests passedCloses: uutils#40