Skip to content

Pull requests: huggingface/tokenizers

New pull request New
39 Open 765 Closed
39 Open 765 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Add get_special_tokens and is_special_token methods
#1945 opened Feb 5, 2026 by ArthurZucker Loading…
2 tasks done
1
Add post_process_tokens and post_process_ids methods
#1944 opened Feb 5, 2026 by ArthurZucker Loading…
3 tasks done
1
feat: add unk_token property to Unigram model
#1943 opened Feb 5, 2026 by ArthurZucker Loading…
4 tasks done
1
feat: add role_to_token field for special token metadata
#1942 opened Feb 5, 2026 by ArthurZucker Loading…
5 tasks done
1
Port to PyO3 0.27
#1941 opened Feb 5, 2026 by xanderlent Loading…
3
Fix escape HTML characters in EncodingVisualizer output
#1937 opened Jan 29, 2026 by OhashiReon Loading…
2
Fix broken source links in documentation
#1934 opened Jan 21, 2026 by Shivam-Bhardwaj Loading…
1
fix: added type hints in .py files
#1932 opened Jan 20, 2026 by ashmi8 Loading…
42
Include license file into python wheels
#1931 opened Jan 20, 2026 by justeph Loading…
Add type hint, update to pyo3 0.27, add automatic type hint generator
#1928 opened Jan 12, 2026 by ArthurZucker Loading…
15
feat: add progress_format option for machine-readable JSON output
#1921 opened Dec 26, 2025 by podarok Loading…
6 tasks done
1
Upgrade GitHub Actions for Node 24 compatibility
#1916 opened Dec 20, 2025 by salmanmkc Loading…
Use unicode-normalization instead of unicode-normalization-alignments
#1912 opened Dec 14, 2025 by IvanIsCoding Loading…
1
Add windows arm64 wheel build to python release
#1907 opened Dec 8, 2025 by finnagin Loading…
1
2
Fix undefined names in docs/source/_ext/entities.py
#1895 opened Nov 28, 2025 by cclauss Loading…
1
Python: Add ruff rules for asyncio and performance
#1894 opened Nov 28, 2025 by cclauss Loading…
3
Implement Append normalizer
#1893 opened Nov 28, 2025 by ArthurZucker Loading…
1
C and C++ bindings to Tokenizers Feature Request
#1888 opened Nov 21, 2025 by thammegowda Loading…
8
Providing byte level offsets for effective alignment in Cross-Tokenizer On-Policy Distillation Feature Request
#1880 opened Oct 30, 2025 by JqzChandler Loading…
5
Mark Python tests that need network access
#1872 opened Oct 2, 2025 by gordonmessmer Loading…
3
Add a multithreaded tokenizer test and as well as 3.14 and 3.14t CI
#1864 opened Sep 12, 2025 by ngoldbaum Loading…
7
feat: allow BPETrainer to be seeded with a set of initial tokens
#1862 opened Sep 6, 2025 by henrycharlesworth Loading…
Fix unsigned integer underflow issue with truncation
#1859 opened Sep 1, 2025 by maxdebayser Loading…
5
feat: add cli for tokenizer and training Feature Request
#1842 opened Aug 6, 2025 by b00f Loading…
6
feat: whitespace optimize Feature Request
#1841 opened Aug 6, 2025 by b00f Loading…
1
1
ProTip! Mix and match filters to narrow down what you’re looking for.