in the web demo: If the text contains '<|endoftext|>', an error will be reported for recognition

[zhuzhuyule]

If the text contains '<|endoftext|>', an error will be reported for recognition

Removing the '<|endoftext|' of the '>' at the end can be recognized

Entered the full '<|endoftext|>' and did not show the colleague error

[niieani]

This is just a limitation of the demo at the moment, this works in the API if you enable which tokens you want to allow tokenizing (as per README).