Training stuck at "Listing all text embed cache entries"

[PCanavelli]

Hey guys,

I'm not exactly sure why, but all SimpleTuner trainings have suddenly stopped working.

The issue seems independent of the dataset, model, and virtual environment used.

More exactly, the training hang indefinitely after this line:

2024-11-04 12:57:34,332 [INFO] (Rank: 0) (id=text-embeds) Listing all text embed cache entries
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 257.49it/s]

Is this a know issue?

Please let me know if I can provide any more details.

[juneleung]

same here

[bghira]

this should be resolved for quite a while now