Repository navigation
Avoid float16 overflow in inverse_softplus - #765
Open
Asterisk-Hunter wants to merge 2 commits into
Open
Asterisk-Hunter wants to merge 2 commits into
Asterisk-Hunter wants to merge 2 commits into
Conversation
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Asterisk-Hunter
marked this pull request as ready for review
October 5, 2026 10:29
Contributor
|
I checked this branch out locally and compared the old and new inverse_softplus. tests/test_utils.py -k softplus: 9 passed. One thing I couldn't tell is how often this path actually runs in float16 under autocast, so I can't say how many users hit it. The change itself looks correct to me. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Finite float16 inputs such as 12, 15, and 20 overflow the positive
expm1intermediate ininverse_softplus, producing infinite outputs and NaN gradients below its default threshold.Use the equivalent negative-exponential expression while preserving the existing input clamp, linear branch, and dtype. Add regression coverage for beta 0.5, 1, and 2 below, at, and above the threshold. No new dependencies.
Validation:
pytest -vv -s --doctest-modules: 307 passed, 62 warnings, in the locked CPU development environment (Python 3.12, PyTorch 2.12.0).pytest -q tests/test_utils.py: 9 passed.pre-commit run --all-files: all hooks passed.Issue Link
Closes #764.
Type of change
Checklist before requesting a review
README changes are not needed for this internal numerical fix.
Author checklist after completed review