[General]: FST can round to infinity without reporting overflow

[shuklaayush]

Problem

FST and FSTP store an x87 value to memory, rounding it to the destination format. FSTP
also removes the value from the x87 register stack after a successful store. Rounding a
finite value can overflow the destination format and produce infinity; that must set the
overflow flag.

The source helper checks for overflow before rounding, but does not report
overflow when the rounding itself produces infinity. Its normal-result branch increments
the exponent when the fraction carries:

if rounded_up then
    if Is_All_Ones(mantissa) then
        exponent := exponent + 1[0 +: E];
        mantissa := Zero(M);
    else
        mantissa := mantissa + 1[0 +: M];
    endif;
endif;

If that increment makes every exponent bit 1 and the fraction is zero, the stored value
is infinity. The branch does not set the overflow flag.

Reproducing case

The opcode notation D9 /3 means opcode byte D9 with the register-selection field of
the following ModR/M byte set to 3; the other fields select the memory address.

Use 64-bit mode with x87 enabled: CR0.EM=0 and CR0.TS=0 allow these instructions to
execute with x87 enabled. ST(0) is the top x87 register and
ST(1) is the next. The control word selects rounding and which exceptions are masked.
“Masked” means the instruction uses the defined fallback behavior for that exception.

Store an extended value with exponent 0x407e and significand 0xffffff8000000000 to a
32-bit floating-point memory destination using FSTP m32fp (D9 /3) under control word
0x037f (round-to-nearest, exceptions masked). Use a nonempty ST(0), initially TOP=0
and cleared exception flags, with a writable four-byte destination. The input is exactly
halfway between the largest finite binary32 value (0x7f7fffff) and 2^128. The helper's
normal branch rounds to infinity but does not assign its overflow flag.

The exponent and significand above specify the input bits exactly; the sign is positive.
The destination is a 32-bit floating-point value.

Source checked: Intel SDM executable specification revision
d307f89f742765865b87c5d4d23f552b3c72e871.

Separate hardware test

A separate hardware test on an AMD EPYC-Milan processor stored 0x7f800000
(positive infinity) with both overflow and precision flags set. This agrees with
the Intel manual's overflow requirement.

Manual reference

References use Intel SDM 325462-089US, October 2025.

Intel SDM Volume 1, section 8.5.4, page 8-28 (PDF page 236) defines overflow
using the rounded result and explicitly includes an extended value
stored to single or double precision. Volume 2A, FST/FSTP, page
3-379 (PDF page 1075) requires conversion to the destination format
using the selected rounding mode.

Proposed fix

When rounding carries out of an all-ones finite significand, check whether the
incremented exponent reaches the all-ones encoding. If it does, report overflow and the
inexact result, then apply the existing masked or unmasked overflow response.

AI disclosure

Assisted-by: Codex

Codex assisted with source analysis, test review, and drafting this report.

Source links

Current helper pages:
FP87::Convert_To_Float.