- tokenizing BOM Unicode character at the start of the file so it is skipped in the token stream
- moved encoding detection from PythonLexerBase to a separate component (grun4py)
- added ENCODING token
-
Type comment tokens are no longer generated.
Type comments will now be tokenized as plain comment tokens. -
Line continuation for string literals (backslash followed by a newline) is no longer resolved.
(backslash+newline is no longer removed from string literals)