[guishiw]

Summary

Transformers 5.10.4 rebuilds Wan's UMT5 tokenizer with unk_id=2, but Wan reserves ID 2 for <s> and uses ID 3 for <unk>. As a result, unsupported Unicode characters could be interpreted as beginning-of-sequence tokens and produce incorrect prompt conditioning.

Before:

𝄞 music → [273, 2, 10129, 1] (2 is <s>)

After:

𝄞 music → [273, 3, 10129, 1] (3 is <unk>)

This change:

• Routes both Wan Diffusers models and standalone Wan T5 encoders through a shared tokenizer loader.
• Loads tokenizer.json directly instead of rebuilding its Unigram pipeline through T5Tokenizer.
• Preserves UMT5 special-token IDs and installs the expected EOS post-processor when missing.
• Reconstructs SentencePiece-only tokenizers while preserving normalization, metaspace, unk_id=3, EOS behavior, and user-defined pieces.

Related Issues / Discussions

Closes #9984

QA Instructions

Reproduction:

1. Load a Wan 2.2 Diffusers tokenizer folder with Transformers 5.10.4.
2. Encode a prompt containing an unsupported character, such as 𝄞 music.
3. Confirm the unsupported character is encoded as <unk> ID 3 rather than <s> ID 2.

Focused regression tests:

pytest tests/backend/model_manager/load/test_wan_tokenizer.py

Result:

4 passed, 11 warnings in 1.85s

The warnings came from an unavailable SQLAlchemy warning filter in the minimal test environment and a Python 3.14 protobuf deprecation; neither is related to this change.

Static checks:

uv tool run ruff@0.11.2 check \
  invokeai/backend/model_manager/load/model_loaders/wan.py \
  invokeai/backend/wan/tokenizer.py \
  tests/backend/model_manager/load/test_wan_tokenizer.py

uv tool run ruff@0.11.2 format --check \
  invokeai/backend/model_manager/load/model_loaders/wan.py \
  invokeai/backend/wan/tokenizer.py \
  tests/backend/model_manager/load/test_wan_tokenizer.py

git diff --check

Results:

• Ruff check: passed.
• Ruff format check: 3 files already formatted.
• git diff --check: passed.
• Official Wan tokenizer.json versus the reconstructed SentencePiece-only path: 2,030 Unicode, unknown-token, special-token, and user-defined-token cases compared with zero mismatches.

Review

Material findings resolved:

• The initial implementation covered standalone UMT5 folders but not the normal Wan Diffusers tokenizer-loading path.
• The SentencePiece-only fallback still used the affected Transformers tokenizer reconstruction.
• Tokenizer folders without a serialized post-processor did not append EOS.
• User-defined SentencePiece tokens initially had incorrect metaspace behavior.

Regression coverage now exercises the real Wan Diffusers loader route, serialized tokenizer loading, EOS insertion, SentencePiece-only reconstruction, unknown-token IDs, and user-defined-token boundaries.

Remaining limitation: the focused CPU tests validate tokenizer and loader behavior, but a complete Wan GPU generation was not run locally.

Compatibility / Rollout

No persisted-data migration or public API change is required.

Prompts containing previously mis-tokenized Unicode or out-of-vocabulary characters will intentionally produce different conditioning after this fix. Existing tokenizer folders containing either tokenizer.json or spiece.model remain supported.

Rollback only requires reverting this change; no persisted state needs to be restored.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (not applicable; no user-facing API or workflow change)
• Updated What's New copy (not doing a release in this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}