Intermittent errors between epochs

[rasar07]

We see these errors intermittently between epochs.  Not every test.  And, not every epoch.  In this case, it occurred once out of 8 epochs. The errors do not prevent the test from completion. We first noticed it after re-cloning to 3.0.15.

[OUTPUT] 2026-06-20T15:21:32.972188 Ending epoch 4 - 32641 steps completed in 1594.25 s
[OUTPUT] 2026-06-20T15:21:33.499306 Reshard for epoch 5: 783387 train files, 0 eval files redistributed via alltoall in 0.53s
Traceback (most recent call last):
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
finalizer()
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in call
res = self._callback(*self._args, **self._kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
rmtree(tempdir, onerror=onerror)
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
_rmtree_safe_fd(stack, onexc)
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
onexc(func, path, err)
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
return onerror(func, path, exc_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/shutil.py", line 662, in _rmtree_safe_fd
os.rmdir(name, dir_fd=dirfd)
OSError: [Errno 39] Directory not empty: '/tmp/pymp-9b8nyjch'
[OUTPUT] 2026-06-20T15:21:35.302933 Worker pre-warm complete for epoch 5 (8 workers spawned)
[OUTPUT] 2026-06-20T15:21:35.307140 Starting epoch 5: 32641 steps expected

[FileSystemGuy]

Diagnosed: the traceback comes from a recent DLIO change that disabled persistent_workers=True on the resharded DataLoader branch, so workers tear down and respawn each epoch. The cleanup of the per-worker /tmp/pymp-* temp dir occasionally races with a parent-side write into the same dir → ENOTEMPTY.

Important: this is cosmetic for scoring purposes. Verified that inter-epoch wall-clock time does not enter the AU or throughput metrics — AU is computed per-epoch from intra-epoch timestamps and then averaged, and the submission checker reads only train_au_mean_percentage / train_au_meet_expectation. The respawn/reshard gap sits outside every epoch's measurement window. So your runs and any submission from them are valid; the traceback is noise.

Tracked for upstream cleanup as #565 (with fix options and the full evidence trail). Closing this in favor of that one is reasonable once the symptom side is acknowledged.

[russfellows]

Can you re-test and reproduce on the CURRENT version?
The most recent Hash-ID is: 6553c0e