[General]: FPATAN omits the denormal flag when ST(0) is denormal and ST(1) is zero

[shuklaayush]

Problem

FPATAN computes an angle from the two top x87 registers, ST(0) and ST(1). A denormal,
also called a subnormal, is a nonzero floating-point value in the range below the
smallest normal value. The manual requires the denormal-input flag (DE) when a source
operand is denormal.

For positive denormal ST(0) and zero ST(1), the source helper returns the
correct angle, zero, but never sets DE. The instruction uses the
helper's flags, so the denormal input is not reported.

The relevant branch handles both normal and denormal ST(0):

when (_, FP87_Class_Zero) => // DENORMAL/NORMAL, ZERO
    if x.Sign == 0b0 then
        result := FP87::Zero(y.Sign);
    ...

It needs to distinguish those inputs when setting the exception flag.

Reproducing case

Use 64-bit mode with x87 enabled: CR0.EM=0 and CR0.TS=0 allow these instructions to
execute with x87 enabled. ST(0) is the top x87 register and
ST(1) is the next. The control word selects rounding and which exceptions are masked.
“Masked” means the instruction uses the defined fallback behavior for that exception.

Set ST(0) to the smallest positive extended-precision subnormal (raw
00000000000000000001), ST(1) to positive zero, and the x87 control word to 0x037f
(exceptions masked). Mark both stack entries nonempty and clear pending x87 exceptions.
Execute FPATAN (D9 F3).

The source helper returns positive zero and DE=0. The result value is expected, but
the denormal-input flag should be set. This counterexample follows from the displayed
source branch and manual exception rule.

Each 20-digit hexadecimal input specifies the exact 80-bit x87 register contents.

Source checked: Intel SDM executable specification revision
d307f89f742765865b87c5d4d23f552b3c72e871.

Manual reference

References use Intel SDM 325462-089US, October 2025.

Intel SDM Volume 2A, FPATAN, pages 3-357–3-358 (PDF pages 1053–1054) lists a denormal
source operand as a denormal exception. Its description identifies
ST(0) and ST(1) as both source operands.

Proposed fix

In the (_, FP87_Class_Zero) branch, set exceptions.DE := 0b1 when x_class == FP87_Class_Denormal, before selecting the zero or π result. Preserve the existing
result and rounding behavior.

AI disclosure

Assisted-by: Codex

Codex assisted with source and manual analysis and drafting this report.

Source links

Current helper pages:
FP87::Atan.