fix(deps): update dependency ai.djl.huggingface:tokenizers to v0.38.0

[renovate]

This PR contains the following updates:

Package	Type	Update	Change	OpenSSF
ai.djl.huggingface:tokenizers (source)	compile	minor	0.33.0 → 0.38.0

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

Release Notes

deepjavalibrary/djl (ai.djl.huggingface:tokenizers)

v0.38.0: DJL v0.38.0 Release

Compare Source

What's Changed

• Use 64-bit arithmetic for tensor buffer size calculations by @​ethnzhng in #​3876
• Increase build version to 0.38.0 by @​xyang16 in #​3879
• [ci] Upgrade actions version by @​xyang16 in #​3880
• [ci] Fix native library build pipeline by @​xyang16 in #​3881

Full Changelog: deepjavalibrary/djl@v0.37.0...v0.38.0

v0.37.0: DJL v0.37.0 Release

Compare Source

This is the v0.37.0 release of DJL.

Changes

• Bundled .java compilation and remote URL loading are now configurable, with tightened defaults (#​3875)
• Bumped log4j to 2.26.0 (#​3877)
• onnxruntime: allow sub-managers after NDManager.cap() (#​3870)
• Docs: fixed a typo (#​3874)

Behavior changes

Two loading behaviors changed their defaults in #​3875. Both previous behaviors remain available, but from this release each has to be enabled explicitly:

Behavior	Default	To restore
Compile bundled .java sources at model load	off	DJL_COMPILE_JAVA=true or -Dai.djl.compile_java=true
Unrestricted remote URL loading	off	DJL_ALLOW_INSECURE_URL=true or -Dai.djl.allow_insecure_url=true

Utils.openUrl now limits remote fetches to public http(s) destinations, resolves redirects explicitly with a bounded hop count, and allows file: and jar: only where they denote a local read. Loading from local directories, s3://, gs://, public http(s) archives, the djl:// model zoo and Hugging Face is unchanged. See the PR description for the full list and the flags.

Known issues / limitations

• A model whose translator is available only as a raw .java file under lib/classes/ will not have it compiled by default; an error-level log names the flag. Models shipping a precompiled .class or .jar, or supplying a translator programmatically, are unaffected.
• Loading a model over http(s) from a host that resolves to a private address now fails by default. Set the flag above if you serve models from an internal mirror.

Contributors

Thank you to the following community members for contributing to this release:

Rohit Kumar Srivastava, Vaibhav Srivastava, dev_Hakaze

v0.36.0: DJL v0.36.0 Release

Compare Source

What's Changed

• Increase build version to 0.36.0 by @​access2rohit in #​3817
• Update the Documentation (via DocString) to indicate the risks of using Utils.openUrl() with untrusted inputs by @​access2rohit in #​3819
• NDManager.create(Number) unsupported type error message improvement by @​fracpete in #​3818

New Contributors

• @​access2rohit made their first contribution in #​3816
• @​fracpete made their first contribution in #​3818

Full Changelog: deepjavalibrary/djl@v0.35.1...v0.36.0

v0.35.1: DJL v0.35.1 Release

Compare Source

What's Changed

• fix: upgrade jQuery from 2.1.1 to 3.7.1 to address security vulnerabilities by @​Lokiiiiii in #​3812
• Fix script injection vulnerabilities in GitHub Actions workflows by @​Lokiiiiii in #​3813
• Fix security vulnerabilities in QuestionAnsweringTranslator (CVSS 8.2) by @​Lokiiiiii in #​3814

New Contributors

• @​Lokiiiiii made their first contribution in #​3812

Full Changelog: deepjavalibrary/djl@v0.35.0...v0.35.1

v0.35.0: DJL v0.35.0 Release

Compare Source

What's Changed

• Engine Updates
  • XGBoost 3.0.4 #​3788

Enhancements

• Increase build version to 0.35.0 by @​xyang16 in #​3766
• Add default header for openai by @​bryanktliu in #​3772
• Update gradle to 9.0.0 by @​xyang16 in #​3777
• [XGBoost] Support json model for XGBoost by @​xyang16 in #​3786
• [api] Accept json input/output for tensor input by @​xyang16 in #​3787
• Upgrade xgboost to 3.0.4 by @​xyang16 in #​3788
• [pytorch] Add tuple of dict support for IValue by @​xyang16 in #​3797
• [XGBoost] Add back gpu support for xgboost by @​xyang16 in #​3796
• Update gradle build script to support configuration cache by @​xyang16 in #​3782
• Added percentile implementation for Pytorch engine by @​tipame in #​3800
• Implemented CSV translator that handles CSV input/output support for XGBoost by @​smouaa in #​3801
• Correct topK documentation by @​alanocallaghan in #​3808
• [api] Refactor csvTranslator by @​xyang16 in #​3805

Bug Fixes

• Fix pytorch native 2.5.1 aarch64 build by @​xyang16 in #​3767
• [fix] Fixes load tokenizer gpu shared library issue by @​xyang16 in #​3773
• Fixes #​3774 for YoloSegmentationTranslator by @​bryanktliu in #​3775
• Fixes issue #​3761 by @​bryanktliu in #​3776
• Fix BaseNDManager.debugDump by @​petebankhead in #​3780
• Fix preference of tokenizer_config.json and remove doLowerCase from TokenizerConfig by @​Soha-Agarwal in #​3785
• Fixes: #​3795, adds aggregation_strategy to token classification task by @​bryanktliu in #​3798
• Fix pytorch native build for windows by @​xyang16 in #​3802

CI/CD

• Build pytorch jni cpu precxx by @​xyang16 in #​3770
• Remove pytorch 2.1.2 and 2.3.1 publish in nightly by @​xyang16 in #​3771
• Fix sentencepiece native workflow by @​xyang16 in #​3768
• Fix fasttext native workflow by @​xyang16 in #​3769
• Fix continuous build by @​xyang16 in #​3804

New Contributors

• @​smouaa made their first contribution in #​3801
• @​alanocallaghan made their first contribution in #​3808

Full Changelog: deepjavalibrary/djl@v0.34.0...v0.35.0

v0.34.0: DJL v0.34.0 Release

Compare Source

Key Changes

• Engine Updates
  • PyTorch 2.7.1 #​3733
  • Remove TensorRT engine #​3765

Enhancements

• [api] Adds FUSE based repository support by @​raymondkhliu in #​3695
• Treat model server as a remote model by @​xyang16 in #​3704
• Align ZeroShotClassificationTranslator with Hugging Face behavior by @​raphaeldelio in #​3712
• Remove usage of UUID.randomUUID() for better performance by @​aakashb-kayzen in #​3719
• Replace UUID with NDManager.nextUid() by @​xyang16 in #​3720
• Add NDArray diagonal by @​dev-jonghoonpark in #​3724
• [api] Implements remote REST API call as a model by @​bryanktliu in #​3725
• Adds jsonlines streaming support for RpcEngine by @​bryanktliu in #​3727
• Adds genai extension by @​bryanktliu in #​3726
• [genai] tool calls to ChatInput by @​bryanktliu in #​3728
• Add utility class for genai function call by @​bryanktliu in #​3729
• Improve function call for genai extension by @​bryanktliu in #​3731
• Support convert ChatInput to GeminiInput by @​bryanktliu in #​3739
• Add partial support for HuggingFace Tokenizer to use parameters from tokenizer_config.json by @​Soha-Agarwal in #​3738
• Add Anthropic support from genai by @​bryanktliu in #​3759

Bug Fixes

• Correct CaptchaDataset options count to prevent torch.gather IndexError by @​xinhuagu in #​3694
• Fix dimension check when reshaping offsets in SeqBatcher by @​xinhuagu in #​3702
• Fix the incorrect function name and minor typo in doc by @​xinhuagu in #​3703
• Fix incorrect return value in getImageHeight method by @​xinhuagu in #​3706
• [djl_converter]: Fixes djl_converter bug by @​raymondkhliu in #​3705
• Fix the out of bounds exception when using an init image by @​luke-zhou in #​3717
• Fix earlystopping metric issue #​3722 by @​SamBSalgado in #​3736
• [tokenizers] Fix tokenizer cpu and cuda build by @​xyang16 in #​3756
• [examples] Update tfhub urls by @​xyang16 in #​3752
• Update tfhub repository url by @​xyang16 in #​3755

Documentation

• [docs] Update model zoo document by @​xyang16 in #​3690

CI/CD

• increase build version to 0.34.0 by @​siddvenk in #​3689
• Uses new recommended way to build pip wheel by @​raymondkhliu in #​3709
• Bump protobuf from 3.20.2 to 4.25.8 in /extensions/tokenizers/src/main/python by @​dependabot[bot] in #​3716
• [android] Upgrade DJL version to 0.34.0 by @​xyang16 in #​3721
• [ci] simplify nightly publish workflow by @​xyang16 in #​3732
• [ci] Update maven central publish url and credential by @​xyang16 in #​3748
• [djl-converter] Update license in pyproject.toml by @​xyang16 in #​3749
• [pytorch] Updates PyTorch to 2.7.1 by @​xyang16 in #​3733
• [ci] Remove serving-publish workflow by @​xyang16 in #​3750
• Update oss sonatype snapshot repository url by @​xyang16 in #​3754
• Upgrade dependency versions by @​xyang16 in #​3762
• [android] Upgrade pytorch version to 2.7.1 by @​xyang16 in #​3763
• [ci] Add post-publish task by @​xyang16 in #​3764
• Increase DJL version to 0.34.0 by @​Soha-Agarwal in #​3758

New Contributors

• @​raymondkhliu made their first contribution in #​3695
• @​raphaeldelio made their first contribution in #​3712
• @​luke-zhou made their first contribution in #​3717
• @​aakashb-kayzen made their first contribution in #​3719
• @​dev-jonghoonpark made their first contribution in #​3724
• @​Soha-Agarwal made their first contribution in #​3738
• @​SamBSalgado made their first contribution in #​3736

Full Changelog: deepjavalibrary/djl@v0.33.0...v0.34.0

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[bjorncs]

Let's upgrade this before adding back #36071.

[hmusum]

Fails to build with:

[ERROR] HuggingFaceTokenizerTest.bert_tokenizer:35 » NoClassDefFound Could not initialize class ai.djl.huggingface.tokenizers.jni.LibUtils

2026-03-11 09:37:31 CEST | [ERROR] HuggingFaceTokenizerTest.pads_to_max_length:109 » NoClassDefFound Could not initialize class ai.djl.huggingface.tokenizers.jni.LibUtils
2026-03-11 09:37:31 CEST | [ERROR] HuggingFaceTokenizerTest.provides_model_info:121 » NoClassDefFound Could not initialize class ai.djl.huggingface.tokenizers.jni.LibUtils
2026-03-11 09:37:31 CEST | [ERROR] HuggingFaceTokenizerTest.tokenizes_using_paraphrase_multilingual_mpnet_base_v2:52 » UnsatisfiedLink /root/.djl.ai/tokenizers/0.21.0-0.36.0-cpu-linux-aarch64/libtokenizers.so: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /root/.djl.ai/tokenizers/0.21.0-0.36.0-cpu-linux-aarch64/libtokenizers.so)
2026-03-11 09:37:31 CEST | [ERROR] HuggingFaceTokenizerTest.truncates_to_max_length:91 » NoClassDefFound Could not initialize class ai.djl.huggingface.tokenizers.jni.LibUtils

[bjorncs]

The build fails on AlmaLinux 8 because the DJL tokenizers v0.36.0 native library (libtokenizers.so) requires glibc 2.29, but AlmaLinux 8 ships glibc 2.28.

The only failing module is linguistics-components — all 5 test failures are in HuggingFaceTokenizerTest:

Test	Error
tokenizes_using_paraphrase_multilingual_mpnet_base_v2	UnsatisfiedLinkError — root cause
bert_tokenizer	NoClassDefFoundError — cascading
pads_to_max_length	NoClassDefFoundError — cascading
provides_model_info	NoClassDefFoundError — cascading
truncates_to_max_length	NoClassDefFoundError — cascading

[bjorncs]

After discussing with @arnej27959 and @toregge: the native so library requires at least glibc 2.34, which will be available once we have upgraded to AlmaLinux 9.