[Bug target/100927] [sse2] floating point to integer conversion functions incorrect results w/ NaN constants + optimization

["michael.crusoe at gmail dot com" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100927

--- Comment #3 from Michael Crusoe <michael.crusoe at gmail dot com> ---
Good question, lets check the reference.

Summary: it is specified behavior that _mm_cvttpd_epi32 returns Integer
Indefinite (80000000H) for NaN inputs.

All references below are from the December 2022 edition (Order Number:
325462-078US) of "Intel® 64 and IA-32 Architectures Software Developer’s Manual
Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D, and 4" from
https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html

The formal signature of the _mm_cvttpd_epi32 intrinsic is in Table C-1 "Simple
Intrinsics" on page 2987, reminding us that the mnemonic is CVTTPD2DQ.

The formal definition of CVTTPD2DQ is given in section 5.6.1.6 "Intel® SSE2
Conversion Instructions" on page 133

> Convert with truncation packed double precision floating-point values to packed double-
word integers.

On page 106 we learn more about what truncation means in the definition of
CVTTPD2DQ

> 4.8.4.2 Truncation with Intel® SSE, SSE2, and AVX Conversion Instructions
> The following Intel SSE/SSE2 instructions automatically truncate the results of
> conversions from floating-point values to integers when the result it inexact: CVTTPD2DQ,
> CVTTPS2DQ, CVTTPD2PI, CVTTPS2PI, CVTTSD2SI, and CVTTSS2SI. Here, truncation means the
> round toward zero mode described in Table 4-8. There are also several Intel AVX2 and
> AVX-512 instructions which use truncation (VCVTT*)

Table 4.8 from section 4.8.4 states

> Rounding Mode: Round toward zero (Truncate)
> Description: Rounded result is closest to but no greater in absolute value than the infinitely precise result.

Section 11.4.1.6 ("SSE2 Conversion Instructions") states that

> The CVTTPD2DQ (convert with truncation packed double precision floating-point values to
> packed doubleword integers) instruction is similar to the CVTPD2DQ instruction except
> that truncation is used to round a source value to an integer value.

Table 11-1. "Masked Responses of SSE/SSE2/SSE3 Instructions to Invalid
Arithmetic Operations" states that

> Condition: Conversion to integer when the value in the source register is a NaN, ∞, or
> exceeds the representable range for CVTPS2PI, CVTTPS2PI, CVTSS2SI, CVTTSS2SI, CVTPD2PI,
> CVTSD2SI, CVTPD2DQ, CVTTPD2PI, CVTTSD2SI, CVTTPD2DQ, CVTPS2DQ, or CVTTPS2DQ

> Masked Response: Return the integer Indefinite

More explicitly stated is in section D.4.2.2 "Results of Operations with NaN
Operands or a NaN Result for SSE/SSE2/SSE3 Numeric Instructions" where Table
D-8 (page 455) ("CVTPS2PI, CVTSS2SI, CVTTPS2PI, CVTTSS2SI, CVTPD2PI, CVTSD2SI,
CVTTPD2PI, CVTTSD2SI, CVTPS2DQ, CVTTPS2DQ, CVTPD2DQ, CVTTPD2DQ") states that
the masked result from any type of NaN (SNaN or QNaN) will be the Integer
Indefinite (80000000H in for 32-bit values).