[Bug target/97653] New: Incorrect long double calculation with -mabi=ibmlongdouble

["redi at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97653

            Bug ID: 97653
           Summary: Incorrect long double calculation with
                    -mabi=ibmlongdouble
           Product: gcc
           Version: 11.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: redi at gcc dot gnu.org
                CC: bergner at vnet dot ibm.com
  Target Milestone: ---
            Target: powerpc64le-unknown-linux-gnu

GCC configured with --with-long-double-format=ieee produces incorrect results
when compiling with -mabi=ibmlongdouble

This code (reduced from 26_numerics/random/knuth_b.cc in the libstdc++
testsuite) fails the assertion:

#include <assert.h>
int printf(const char*, ...);

unsigned long min() { return 1; }
unsigned long max() { return 2147483646; }

unsigned long k = 256;

unsigned long y = 1465645203;

unsigned long f()
{
  long double r = ((y - min()) / (max() - min() + 1.0L));
  assert( r < 1 );
  unsigned long j = k * ((y - min()) / (max() - min() + 1.0L));
  assert( j < k );
  return j;
}

int main()
{
  printf("%lu\n", f());
}


$ ~/gcc/ieee128/bin/gcc k.c -mno-gnu-attribute -mabi=ibmlongdouble && ./a.out
a.out: k.c:16: f: Assertion `j < k' failed.
Aborted (core dumped)


$ ~/gcc/ieee128/bin/gcc -v
Using built-in specs.
COLLECT_GCC=/home/test/gcc/ieee128/bin/gcc
COLLECT_LTO_WRAPPER=/home/test/gcc/ieee128/libexec/gcc/powerpc64le-unknown-linux-gnu/11.0.0/lto-wrapper
Target: powerpc64le-unknown-linux-gnu
Configured with: ../gcc/configure --prefix=/home/test/gcc/ieee128/
--enable-libstdcxx-debug --disable-bootstrap --disable-multilib
--disable-libvtv --with-system-zlib --without-isl
--with-long-double-format=ieee --enable-languages=c,c++
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.0.0 20201030 (experimental) (GCC) 

This is an up to date Fedora 33 system:

# rpm -q glibc
glibc-2.32-1.fc33.ppc64le

# lscpu
Architecture:                    ppc64le
Byte Order:                      Little Endian
CPU(s):                          160
On-line CPU(s) list:             0-159
Thread(s) per core:              4
Core(s) per socket:              20
Socket(s):                       2
NUMA node(s):                    2
Model:                           2.2 (pvr 004e 1202)
Model name:                      POWER9, altivec supported
Frequency boost:                 enabled
CPU max MHz:                     3800.0000
CPU min MHz:                     2166.0000
L1d cache:                       1.3 MiB
L1i cache:                       1.3 MiB
L2 cache:                        10 MiB
L3 cache:                        200 MiB
NUMA node0 CPU(s):               0-79
NUMA node8 CPU(s):               80-159
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Mitigation; RFI Flush, L1D private per thread
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Mitigation; RFI Flush, L1D private per thread
Vulnerability Spec store bypass: Mitigation; Kernel entry/exit barrier (eieio)
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization, ori31
speculation barrier enabled
Vulnerability Spectre v2:        Mitigation; Indirect branch cache disabled,
Software link stack flush
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected