Skip to content

Honor low-disk policy before TorchInductor compilation - #3220

Merged
bghira merged 1 commit into
mainfrom
fix/3203-compile-disk-pressure
Sep 7, 2026
Merged

bghira merged 1 commit into
mainfrom
fix/3203-compile-disk-pressure

Conversation

@bghira

@bghira bghira commented Sep 7, 2026

Copy link
Copy Markdown
Owner

The existing low-disk policy runs only before checkpoint saves, so it does not pause training when the TorchInductor cache filesystem is already below the configured threshold. Apply the same policy before compiler initialization in the training context and before each graph compilation or recompilation. Check the Inductor cache directory and any separately configured Triton cache directory, and remove the callback when training exits. Both CLI and SDK entry points are covered; checkpoint checks retain their main-process behavior.

This is a preflight check, not disk-space reservation or recovery from a failed compilation. Compilation can consume the remaining space after the check, and initialization outside the training context can precede it. The supplied issue log shows CUDA allocation failure, not an established disk-full error; this change does not recover GPU OOMs or replay training work. Existing options and all OPTIONS translations now describe the scope.

Validation: reproduced the missing waits with actual tiny CPU Inductor compilation and shape-triggered recompilation before the fix. An SDK regression also failed before its context was added. Afterward, all 142 tests passed with .venv/bin/python -m unittest -v -f tests.test_compile_disk_space tests.test_dynamo tests.test_dynamo_cache tests.test_trainer. Coverage includes stop/wait/script behavior, custom/default cache locations, callback cleanup, failure propagation, and checkpoint process selection. Black and git diff --check passed. Testing used the installed Python 3.14 environment.

Related to #3203.

@bghira
bghira merged commit 1e2d014 into main Sep 7, 2026
4 checks passed
@bghira
bghira deleted the fix/3203-compile-disk-pressure branch September 7, 2026 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant