[Bug c/114659] New: gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[Bug c/114659] New: gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[bruno at clisp dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

            Bug ID: 114659
           Summary: gcc miscompiles a __builtin_memcpy on i386, leading to
                    wrong results for SNaN
           Product: gcc
           Version: 13.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
          Assignee: unassigned at gcc dot gnu.org
          Reporter: bruno at clisp dot org
  Target Milestone: ---

Created attachment 57912
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57912&action=edit
test case tf.c

In the two attached test cases, gcc miscompiles a __builtin_memcpy invocation.
In the first test case, the data type is a 'float' (4 bytes).
In the second test case, the data type is a 'double' (8 bytes).

A value of this data type exists in memory, given as *x and *y.
A modified copy of this value, convert_snan_to_qnan(value), exists
also in the stack, among the local variables.
gcc implements the __builtin_memcpy operation by accessing
convert_snan_to_qnan(value) instead of the original value.

How to reproduce:

$ gcc-version 13.2.0 -m32 -Wall tf.c
$ ./a.out ; echo $?
0
$ gcc-version 13.2.0 -m32 -Wall -O2 tf.c
$ ./a.out ; echo $?
1

$ gcc-version 13.2.0 -m32 -Wall td.c
$ ./a.out ; echo $?
0
$ gcc-version 13.2.0 -m32 -Wall -O2 td.c
$ ./a.out ; echo $?
1

Analysis:

$ gcc-version 13.2.0 -m32 -Wall -O2 -S tf.c

tf.c has this function:
============================================================
int
my_totalorderf (float const *x, float const *y)
{
  int xs = __builtin_signbit (*x);
  int ys = __builtin_signbit (*y);
  if (!xs != !ys)
    return xs;

  int xn = __builtin_isnan (*x);
  int yn = __builtin_isnan (*y);
  if (!xn != !yn)
    return !xn == !xs;
  if (!xn)
    return *x <= *y;

  unsigned int extended_sign = -!!xs;
  union { unsigned int i; float f; } xu = {0}, yu = {0};
  __builtin_memcpy (&xu.f, x, sizeof (float));
  __builtin_memcpy (&yu.f, y, sizeof (float));
  return (xu.i ^ extended_sign) <= (yu.i ^ extended_sign);
}
============================================================
tf.s looks like this:
============================================================
my_totalorderf:
        pushl   %ebx
        subl    $8, %esp
;;  int xs = __builtin_signbit (*x);
        movl    16(%esp), %eax
        flds    (%eax)
        fsts    (%esp)                ;; [%esp+0] := convert_snan_to_qnan(*x)
        fxam
        fnstsw  %ax
        movl    %eax, %edx
        movl    20(%esp), %eax
        andl    $512, %edx
;;  int ys = __builtin_signbit (*y);
        flds    (%eax)
        sete    %cl
        fsts    4(%esp)               ;; [%esp+4] := convert_snan_to_qnan(*y)
        fxam
        fnstsw  %ax
        testb   $2, %ah
        sete    %al
;;  if (!xs != !ys)
        cmpb    %al, %cl
        jne     .L12
;;  int xn = __builtin_isnan (*x);
        fxch    %st(1)
        fucomi  %st(0), %st
        fxch    %st(1)
        setnp   %bl
;;  int yn = __builtin_isnan (*y);
        fucomip %st(0), %st
        setnp   %al
;;  if (!xn != !yn)
        cmpb    %al, %bl
        jne     .L11
        fstp    %st(0)
        flds    (%esp)
        fucomi  %st(0), %st
        jp      .L9
        flds    4(%esp)
        xorl    %edx, %edx
        fcomip  %st(1), %st
        fstp    %st(0)
        setnb   %dl
        jmp     .L6
        .p2align 4,,10
        .p2align 3
.L12:
        fstp    %st(0)
        fstp    %st(0)
.L6:
        addl    $8, %esp
        movl    %edx, %eax
        popl    %ebx
        ret
        .p2align 4,,10
        .p2align 3
.L11:
        fucomip %st(0), %st
        setp    %dl
        addl    $8, %esp
        xorl    %ecx, %edx
        popl    %ebx
        movzbl  %dl, %edx
        movl    %edx, %eax
        ret
        .p2align 4,,10
        .p2align 3
.L9:
        fstp    %st(0)
        negl    %edx                  ;; computes -xs
        movl    (%esp), %eax          ;; fetches convert_snan_to_qnan(*x)
instead of *x
        movl    4(%esp), %ebx         ;; fetches convert_snan_to_qnan(*y)
instead of *y
        sbbl    %edx, %edx            ;; computes extended_sign = -!!xs;
        xorl    %edx, %eax            ;; computes (xu.i ^ extended_sign)
        xorl    %ebx, %edx            ;; computes (yu.i ^ extended_sign)
        cmpl    %eax, %edx            ;; compares (xu.i ^ extended_sign) and
(xu.i ^ extended_sign)
        setnb   %dl
        movzbl  %dl, %edx
        jmp     .L6
============================================================
As you can see, (%esp) and 4(%esp) contain *not* the original
*x and *y respectively, but the result of an flds/fsts instruction pair,
that is, convert_snan_to_qnan(*x) and convert_snan_to_qnan(*y), respectively.

See https://lists.gnu.org/archive/html/bug-gnulib/2023-10/msg00060.html
for some background about these instructions on i386.

The analysis of td.c is similar; here the value is stored to
memory through an fldl/fstl pair.

[Bug c/114659] gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[bruno at clisp dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

--- Comment #1 from Bruno Haible <bruno at clisp dot org> ---
Created attachment 57913
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57913&action=edit
test case td.c

[Bug c/114659] gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[bruno at clisp dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

Bruno Haible <bruno at clisp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Build|                            |x86_64-linux-gnu
               Host|                            |x86_64-linux-gnu
             Target|                            |x86_64-linux-gnu

--- Comment #2 from Bruno Haible <bruno at clisp dot org> ---
Note: "gcc-version 13.2.0" just invokes gcc-13.2.0, which I built from source.

[Bug c/114659] gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[bruno at clisp dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

--- Comment #3 from Bruno Haible <bruno at clisp dot org> ---
Also reproducible in 64-bit mode, with '-mfpmath=387':

$ gcc -mfpmath=387 -Wall tf.c
$ ./a.out ; echo $?
0
$ gcc -mfpmath=387 -Wall -O2 tf.c
$ ./a.out ; echo $?
1

$ gcc -mfpmath=387 -Wall td.c
$ ./a.out ; echo $?
0
$ gcc -mfpmath=387 -Wall -O2 td.c
$ ./a.out ; echo $?
1

[Bug c/114659] gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[bruno at clisp dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

--- Comment #4 from Bruno Haible <bruno at clisp dot org> ---
Related: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58416

[Bug c/114659] gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[bruno at clisp dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

--- Comment #5 from Bruno Haible <bruno at clisp dot org> ---
Related: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93271

[Bug target/114659] gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|middle-end                  |target

--- Comment #6 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
I doubt there is not much to be done here. It is a x87 issue where we do the
store of the float register stack register to the stack to get 32bits (or
64bit) version. And then load it into a GPR.




  float t = *x;
  float t1 = *y;

 __builtin_memcpy (&xu.f, &t, sizeof (float));
 __builtin_memcpy (&xu.f, &t1, sizeof (float));

Produces exactly the same issue.

[Bug target/114659] gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://gcc.gnu.org/bugzill
                   |                            |a/show_bug.cgi?id=56831,
                   |                            |https://gcc.gnu.org/bugzill
                   |                            |a/show_bug.cgi?id=57484

--- Comment #7 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Much more related to PR 56831 and PR 57484 rather than the other two ...

[Bug target/114659] gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[bruno at clisp dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

--- Comment #8 from Bruno Haible <bruno at clisp dot org> ---
(In reply to Andrew Pinski from comment #6)
> I doubt there is not much to be done here.

I see it as an incorrect modelization of the x87 hardware, together with a
missing distinction in the common expression elimination / aliasing analysis.
In detail:

* Incorrect modelization of the x87 hardware: The compiler seems to assume that
    flds MEM_LOCATION_1
    fsts MEM_LOCATION_2
  will result in MEM_LOCATION_2 having the same value as MEM_LOCATION_1. This
is wrong;
  this is not how the x87 hardware behaves. The actual result is:
    *MEM_LOCATION_2 = convert_snan_to_qnan(*MEM_LOCATION_1).

* In the common expression elimination / aliasing analysis, the compilers seems
to keep
  track of a set of memory locations MEM_LOCATION_1, ..., MEM_LOCATION_n which
have the
  same value. In fact, this set needs to be partitioned into two sets: a subset
which
  contains the same value, and the complementary subset which contains
  convert_snan_to_qnan(value).

  In other words, each element of the set needs to be annotated with a bit that
tells
  whether the value has been subject to the convert_snan_to_qnan.

[Bug target/114659] gcc miscompiles a __builtin_memcpy on i386, leading to wrong results for SNaN

[bruno at clisp dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114659

--- Comment #9 from Bruno Haible <bruno at clisp dot org> ---
(In reply to Andrew Pinski from comment #7)
> Much more related to PR 56831 and PR 57484 rather than the other two ...

Well, bug #56831 is more about function calls and the ABI, whereas this bug
here and bug #58416 and bug #93271 are about the compiler picking a memory
location which holds convert_snan_to_qnan(value) rather than a memory location
which holds the original value.