Skip to content

Extend lexer for SimplicityHL formatter - #372

Open
ikripaka wants to merge 3 commits into
BlockstreamResearch:masterfrom
ikripaka:feat/fmt-parser
Open

Extend lexer for SimplicityHL formatter#372
ikripaka wants to merge 3 commits into
BlockstreamResearch:masterfrom
ikripaka:feat/fmt-parser

Conversation

@ikripaka

@ikripaka ikripaka commented Jul 15, 2026

Copy link
Copy Markdown

Add parsing of additional tokens for formatting

  • add fmt feature to separate logic for formatter parsing
  • edit the logic of the lexer to include other characters as well for formatting
  • add FmtTokens, which stores comments, newlines and whitespace data
  • mark Token, TriviaKind, FmtToken as non-exhaustive
  • modify parser to ignore comments in regular mode
  • separate tests into modules
  • add span to most of the structs in the ast
  • add tests to test correctness of both behaviours for ordinary parsing and lossless

@ikripaka
ikripaka force-pushed the feat/fmt-parser branch 2 times, most recently from 3e6427e to 0cd4ac0 Compare July 23, 2026 17:45
@ikripaka
ikripaka marked this pull request as ready for review July 24, 2026 09:15
@ikripaka
ikripaka requested a review from delta1 as a code owner July 24, 2026 09:15
* add `fmt` feature
* edit logic of lexer to include other characters (restrict behaviour to be consistent across different lexers)
* add FmtTokens, which stores comments, newlines and whitespaces data
* add `#[non_exhaustive]` to Token, TriviaKind and FmtToken
* separate tests with modules for different behaviour (lexer + parser)
* add additional spans for appropriate structs in parser
* add integration test for lossless lexer, which checks that lossless lexer didn't lose anything from context (relies onto the all .simf examples in this repo)
@ikripaka
ikripaka force-pushed the feat/fmt-parser branch 3 times, most recently from b8b59e9 to de15ca1 Compare August 5, 2026 11:40
* add tests to test such behaviour
* separate digits parsers, reject digits with leading undersrores in decimal variant
* process whitespace in lexer by chunks instead of one-by-one
* featuregate logic with including underscores
@LesterEvSe

LesterEvSe commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Let's flatten the nested test modules in lexer.rs. Having an outer mod tests creates unnecessary nesting since the file itself already scopes the tests.

Current:

mod tests {
    mod lexer { ... }
    mod fmt_lexer { ... }
}

Proposed:

mod lexer { ... }

mod fmt_lexer { ... }

UPD: Same idea in parse.rs

@LesterEvSe LesterEvSe left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 11c8937; tested locally with just check and just check_fuzz

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants