Honor low-disk policy before TorchInductor compilation - #3220
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The existing low-disk policy runs only before checkpoint saves, so it does not pause training when the TorchInductor cache filesystem is already below the configured threshold. Apply the same policy before compiler initialization in the training context and before each graph compilation or recompilation. Check the Inductor cache directory and any separately configured Triton cache directory, and remove the callback when training exits. Both CLI and SDK entry points are covered; checkpoint checks retain their main-process behavior.
This is a preflight check, not disk-space reservation or recovery from a failed compilation. Compilation can consume the remaining space after the check, and initialization outside the training context can precede it. The supplied issue log shows CUDA allocation failure, not an established disk-full error; this change does not recover GPU OOMs or replay training work. Existing options and all OPTIONS translations now describe the scope.
Validation: reproduced the missing waits with actual tiny CPU Inductor compilation and shape-triggered recompilation before the fix. An SDK regression also failed before its context was added. Afterward, all 142 tests passed with
.venv/bin/python -m unittest -v -f tests.test_compile_disk_space tests.test_dynamo tests.test_dynamo_cache tests.test_trainer. Coverage includes stop/wait/script behavior, custom/default cache locations, callback cleanup, failure propagation, and checkpoint process selection. Black andgit diff --checkpassed. Testing used the installed Python 3.14 environment.Related to #3203.