Repository navigation
[bug]: Wan folder tokenizer maps unknown characters to <s> under transformers 5 #9984
Description
What happened
For a Wan 2.2 Diffusers folder (or a standalone UMT5 folder), invokeai/backend/model_manager/load/model_loaders/wan.py:618 loads the tokenizer with AutoTokenizer.from_pretrained. Under transformers 5.10.4 this returns a T5Tokenizer that rebuilds its own pipeline instead of using the folder's tokenizer.json:
- it splits on whitespace;
- it has no normalizer;
- it hard-codes
Unigram(unk_id=2)(transformers/models/t5/tokenization_t5.py:113-127). Id 2 is<s>in UMT5's vocabulary;<unk>is 3.
Measured against the folder's own tokenizer.json (Wan 2.2 TI2V-5B Diffusers), with prompts run through the Wan node's prompt_clean:
- On 4021 fuzz strings, the folder path differs from
tokenizer.jsonin 3785 cases. - Example:
'\U0001D11E music'→ folder[273, 2, 10129, 1],tokenizer.json[273, 3, 10129, 1]. - Out-of-vocabulary characters become
<s>instead of<unk>: several emoji, about 55 % of CJK Unified code points, most Hangul syllables. - A space before a literal
</s>or<extra_id_N>also tokenizes differently.
The UMT5 GGUF tokenizer added in #9967 is built from the GGUF fields. It matches tokenizer.json exactly on the same set, so the two Wan encoder formats now condition differently on such prompts.
What you expected to happen
The folder tokenizer reproduces tokenizer.json, the tokenization Wan was trained with.
How to reproduce the problem
Load the Wan 2.2 Diffusers tokenizer/ with AutoTokenizer.from_pretrained and with PreTrainedTokenizerFast(tokenizer_file=…/tokenizer.json), and compare the ids for a prompt containing e.g. 𝄞.
Possible fix
Load the folder tokenizer from tokenizer.json as a PreTrainedTokenizerFast (or T5TokenizerFast), with the folder's special tokens.
Environment
main @ ea1056b, transformers 5.10.4.
Activity
guishiw commentedon Oct 27, 2026
Hi, I’d like to work on this issue. I’ve reproduced the tokenizer mismatch and completed a proposed fix on my personal fork:
Branch: fix/9984-wan-folder-tokenizer
Root cause
Wan 2.2 uses a UMT5 vocabulary where token ID
2is<s>and token ID3is<unk>.Under Transformers 5.10.4, loading the folder through
AutoTokenizeror theT5TokenizerFastclass declared inmodel_index.jsondoes not reliably preserve the folder’s serializedtokenizer.jsonpipeline. The T5 implementation rebuilds a Unigram tokenizer with a hard-codedunk_id=2.Consequently, out-of-vocabulary characters—including many emoji, CJK characters, Hangul syllables, and other Unicode input—are encoded as
<s>instead of<unk>. This produces conditioning that differs from the tokenizer used to train Wan and from InvokeAI’s GGUF tokenizer path.The problem affects both:
Fix strategy
tokenizer.jsonis present, load it directly as aPreTrainedTokenizerFastso the serialized Unigram pipeline andunk_id=3are preserved.spiece.model, reconstruct the tokenizer from the SentencePiece protobuf while preserving:USER_DEFINEDvocabulary pieces.WanDiffusersModelloader route.The focused tests pass with 4 tests passed. I also compared the reconstructed SentencePiece path against the official Wan serialized tokenizer across 2,030 ordinary, Unicode, unknown, special-token, and user-defined-token cases, with 0 mismatches. Ruff, format, and diff checks also pass.
Could you please assign this issue to me? If the approach looks good, I’ll open a pull request for review. Thank you!