Next-generation syntax highlighting for rustdoc

[matklad]

At the moment, rustdoc uses only the lexer to do syntax highlighting (#75775). This has the benefits of simplicity & performance, and gets us 80% of the way quality wise, but it is not perfect. For example, lexer-based highlighting can't properly detect contextual keywords, glued/split operators, etc.

We can and probably should make this better, though this requires a lot of fundamental work on IDE-ification of the compiler

• the simple thing we can do is to use better heuristic to glue token (Use rustc_lexer for rustdoc syntax highlighting #75775 (comment))
• the next step is to use parser & syntax tree for highlighting. This probably should be done after parser library-ification (Form t-compiler/wg-parser-library compiler-team#338)
• the final step is to overlay semantic info, so that references are colored the same way as the things they refer to (which includes "clicking on the name should open the name's definition" as a subtask). This is already available in rust-analyzer, but will take some time to percolate to rustdoc

[camelid]

What do you mean by this?

so that references are colored the same way as the things they refer to (which includes "clicking on the name should open the name's definition as a subtask")

[matklad]

Note how modules, macros, trait and enums all have different colors in imports

EDIT: ah, I misplaced the quote, the wording should be better now

[camelid]

Oh, that's cool!

[pickfire]

And it would be cooler if those text are clickable.

[GuillaumeGomez]

@pickfire: There is a work in progress for that (started in #84176). I need to write an RFC at some point so we can stabilize it.