Take too much time twhen loading checkpoint shards

[maxin-cn]

Describe the bug

I tried to use from_pretrained function in Latte to automatically download related models from hugginface, but it took a long time to load the T5 model:

https://github.com/Vchitect/Latte/blob/main/sample/sample_t2v.py#L36

I also tried to download the model manually using wget and trained it with the from_pretrained function. It loaded in about 2s.

Reproduction

git clone https://github.com/Vchitect/Latte
cd Latte

conda env create -f environment.yml
conda activate latte

bash sample/t2v.sh

Logs

No response

System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

• diffusers version: 0.24.0
• Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
• Python version: 3.11.8
• PyTorch version (GPU?): 2.2.1 (True)
• Huggingface_hub version: 0.22.2
• Transformers version: 4.39.3
• Accelerate version: 0.29.2
• xFormers version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

No response

[sayakpaul]

It's not a diffusers related issue. I think it's probably on the Hub side, so I am gonna tag @Wauplin here and move this to Discussions.