Skip to content

[Tokenizer] Update ensure_ascii to False for tojson in ChatTemplate - #11134

Merged
swgu98 merged 1 commit into
PaddlePaddle:incubate/paddlenlp-fleety_20250421from
DrownFish19:fleety_20251010_update_chattemplate
Oct 10, 2025
Merged

[Tokenizer] Update ensure_ascii to False for tojson in ChatTemplate#11134
swgu98 merged 1 commit into
PaddlePaddle:incubate/paddlenlp-fleety_20250421from
DrownFish19:fleety_20251010_update_chattemplate

Conversation

@DrownFish19

@DrownFish19 DrownFish19 commented Oct 10, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Before submitting

  • Lint code. If there are lint issues, please format the code first.
# Install and register `pre-commit` in the project folder
pip install pre-commit && pre-commit install

# Process previous code files separately
pre-commit run --file XXXX.py
  • Add test cases into tests folder. If there are codecov issues, please add tests cases first.

PR types

Bug fixes

PR changes

Others

Description

  1. 在ChatTemplate，修改tojson的参数ensure_ascii默认为False，避免apply_chat_template时中文出现转换。

cherry-pick from https://github.com/PaddlePaddle/PaddleNLP/pull/10677/commits

@paddle-bot

paddle-bot Bot commented Oct 10, 2025

Copy link
Copy Markdown

Thanks for your contribution!

@DrownFish19
DrownFish19 force-pushed the fleety_20251010_update_chattemplate branch from 01a47a0 to 2a1f1f9 Compare October 10, 2025 11:42
@swgu98
swgu98 merged commit a4bd6e9 into PaddlePaddle:incubate/paddlenlp-fleety_20250421 Oct 10, 2025
6 of 7 checks passed
@DrownFish19
DrownFish19 deleted the fleety_20251010_update_chattemplate branch October 10, 2025 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@DrownFish19 @swgu98 @wtmlon