[Bug math/14048] New: fmod() incorrectly returns NaN for (some?) denormalized inputs

[Bug math/14048] New: fmod() incorrectly returns NaN for (some?) denormalized inputs

[jkummerow at chromium dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

             Bug #: 14048
           Summary: fmod() incorrectly returns NaN for (some?)
                    denormalized inputs
           Product: glibc
           Version: 2.15
            Status: NEW
          Severity: normal
          Priority: P2
         Component: math
        AssignedTo: unassigned@sourceware.org
        ReportedBy: jkummerow@chromium.org
    Classification: Unclassified


I've come across a case where fmod() does not return the expected result.
Reduced repro:
------
#include <math.h>
#include <stdio.h>
#include <iostream>

int main() {
  double x = 2.225073858507201e-308;
  double y = 5e-324;
  double z = fmod(x, y);
  // printf("result: %g\n", z);  // see [1] below.
  std::cout << "result: " << z << std::endl;
  return 0;
}
------
Expected result: 0
Actual result: -nan

I can repro the bug on both a Gentoo (gcc-4.5.3, kernel 3.3.4) and an Ubuntu
Precise (gcc-4.6.3, kernel 3.2.5) system, which both have glibc-2.15, and both
are x86_64. It works correctly on Ubuntu Lucid (glibc-2.11, gcc-4.4.2, kernel
2.6.38.8).

Further, it works correctly when compiling with either of -O1, -O2, -O3. It
also works correctly when removing the "std::cout << ..." and "#include
<iostream>" lines, and using the "printf(..." line (marked [1]) instead.

I've looked at the generated machine code. In the buggy case, glibc's fmod is
called directly:
main():
  400744:    55                       push   rbp
  400745:    48 89 e5                 mov    rbp,rsp
  400748:    48 83 ec 20              sub    rsp,0x20
  40074c:    48 b8 ff ff ff ff ff     movabs rax,0xfffffffffffff
  400753:    ff 0f 00 
  400756:    48 89 45 f8              mov    QWORD PTR [rbp-0x8],rax
  40075a:    b8 01 00 00 00           mov    eax,0x1
  40075f:    48 89 45 f0              mov    QWORD PTR [rbp-0x10],rax
  400763:    f2 0f 10 4d f0           movsd  xmm1,QWORD PTR [rbp-0x10]
  400768:    f2 0f 10 45 f8           movsd  xmm0,QWORD PTR [rbp-0x8]
  40076d:    e8 de fe ff ff           call   400650 <fmod@plt>
  400772:    f2 0f 11 45 e8           movsd  QWORD PTR [rbp-0x18],xmm0
  400777:    f2 0f 10 45 e8           movsd  xmm0,QWORD PTR [rbp-0x18]
  40077c:    bf dc 08 40 00           mov    edi,0x4008dc
  400781:    b8 01 00 00 00           mov    eax,0x1
  400786:    e8 75 fe ff ff           call   400600 <printf@plt>
  40078b:    b8 00 00 00 00           mov    eax,0x0
  400790:    c9                       leave  
  400791:    c3                       ret


When I do any of the changes that make it work (e.g. remove the iostream
include), g++ decides to inline an FPU-based implementation of fmod (which
seems to work as expected) and only calls out to glibc as a fallback:
main():
  400604:    55                       push   rbp
  400605:    48 89 e5                 mov    rbp,rsp
  400608:    48 83 ec 40              sub    rsp,0x40
  40060c:    48 b8 ff ff ff ff ff     movabs rax,0xfffffffffffff
  400613:    ff 0f 00 
  400616:    48 89 45 f8              mov    QWORD PTR [rbp-0x8],rax
  40061a:    b8 01 00 00 00           mov    eax,0x1
  40061f:    48 89 45 f0              mov    QWORD PTR [rbp-0x10],rax
  400623:    dd 45 f8                 fld    QWORD PTR [rbp-0x8]
  400626:    dd 45 f0                 fld    QWORD PTR [rbp-0x10]
  400629:    d9 c0                    fld    st(0)
  40062b:    d9 c2                    fld    st(2)
  40062d:    d9 f8                    fprem  
  40062f:    df e0                    fnstsw ax
  400631:    f6 c4 04                 test   ah,0x4
  400634:    75 f7                    jne    40062d <main+0x29>
  400636:    dd d9                    fstp   st(1)
  400638:    dd 5d d8                 fstp   QWORD PTR [rbp-0x28]
  40063b:    f2 0f 10 45 d8           movsd  xmm0,QWORD PTR [rbp-0x28]
  400640:    66 0f 2e c0              ucomisd xmm0,xmm0
  400644:    7a 06                    jp     40064c <main+0x48>
  400646:    66 0f 2e c0              ucomisd xmm0,xmm0
  40064a:    74 17                    je     400663 <main+0x5f>
  40064c:    dd 5d c8                 fstp   QWORD PTR [rbp-0x38]
  40064f:    f2 0f 10 4d c8           movsd  xmm1,QWORD PTR [rbp-0x38]
  400654:    dd 5d c8                 fstp   QWORD PTR [rbp-0x38]
  400657:    f2 0f 10 45 c8           movsd  xmm0,QWORD PTR [rbp-0x38]
  40065c:    e8 af fe ff ff           call   400510 <fmod@plt>
  400661:    eb 04                    jmp    400667 <main+0x63>
  400663:    dd d8                    fstp   st(0)
  400665:    dd d8                    fstp   st(0)
  400667:    f2 0f 11 45 e8           movsd  QWORD PTR [rbp-0x18],xmm0
  40066c:    f2 0f 10 45 e8           movsd  xmm0,QWORD PTR [rbp-0x18]
  400671:    bf 7c 07 40 00           mov    edi,0x40077c
  400676:    b8 01 00 00 00           mov    eax,0x1
  40067b:    e8 70 fe ff ff           call   4004f0 <printf@plt>
  400680:    b8 00 00 00 00           mov    eax,0x0
  400685:    c9                       leave  
  400686:    c3                       ret


So, it looks to me like glibc's fmod() doesn't handle this case (max
denormalized double modulo min denormalized double) correctly. Am I barking up
the wrong tree?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

Rich Felker <bugdal at aerifal dot cx> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugdal at aerifal dot cx

--- Comment #1 from Rich Felker <bugdal at aerifal dot cx> 2012-05-02 15:31:45 UTC ---
Using the -fno-builtin option to gcc should make it easier to test/reproduce
this.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[jkummerow at chromium dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

--- Comment #2 from jkummerow at chromium dot org 2012-05-03 20:38:51 UTC ---
Indeed, with -fno-builtin the provided test case fails regardless of -O{1,2,3}.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[jsm28 at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

Joseph Myers <jsm28 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |glibc_2.15
                 CC|                            |carlos at systemhalted dot
                   |                            |org

--- Comment #3 from Joseph Myers <jsm28 at gcc dot gnu.org> 2012-06-01 19:09:50 UTC ---
Fixed for 2.16 by:

commit c5bfe3d5ba29d36563f1e4bd4f8d7336093ee6fc
Author: Joseph Myers <joseph@codesourcery.com>
Date:   Fri Jun 1 19:05:46 2012 +0000

    Fix fmod for subnormals (bug 14048).

Carlos, I think this should go on 2.15 branch as well.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[jkummerow at chromium dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

--- Comment #4 from jkummerow at chromium dot org 2012-06-02 12:56:07 UTC ---
Thanks!

I have recompiled glibc-2.15 on my machine with that patch applied manually and
can confirm that it fixes the issue I was seeing.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[jsm28 at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

--- Comment #5 from Joseph Myers <jsm28 at gcc dot gnu.org> 2012-06-08 10:34:58 UTC ---
Carlos, the 2.15 backport should also include

commit 1b671feb6115afbc90a7b37be4929d3e0394f311
Author: Adhemerval Zanella <azanella@linux.vnet.ibm.com>
Date:   Tue Jun 5 21:31:24 2012 -0300

    Fix for wrong ldbl128-ibm fmodl commit

commit 34ae0b3270c67cae0c54ac98b693fdf7d010a206
Author: Adhemerval Zanella <azanella@linux.vnet.ibm.com>
Date:   Tue Jun 5 10:13:41 2012 -0300

    Fix ldbl128ibm fmodl for subnormals.

to avoid the new tests failing for powerpc.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[jsm28 at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

--- Comment #6 from Joseph Myers <jsm28 at gcc dot gnu.org> 2012-06-25 21:39:19 UTC ---
Created attachment 6481
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6481
Backport 1/3

Carlos, please review this series of three backports for 2.15 branch.  Tested
x86_64.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[jsm28 at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

--- Comment #7 from Joseph Myers <jsm28 at gcc dot gnu.org> 2012-06-25 21:41:03 UTC ---
Created attachment 6482
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6482
Backport 2/3

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[jsm28 at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

--- Comment #8 from Joseph Myers <jsm28 at gcc dot gnu.org> 2012-06-25 21:42:42 UTC ---
Created attachment 6483
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6483
Backport 3/3

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[carlos_odonell at mentor dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

Carlos O'Donell <carlos_odonell at mentor dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos_odonell at mentor
                   |                            |dot com

--- Comment #9 from Carlos O'Donell <carlos_odonell at mentor dot com> 2012-06-25 21:52:23 UTC ---
(In reply to comment #6)
> Created attachment 6481 [details]
> Backport 1/3
> 
> Carlos, please review this series of three backports for 2.15 branch.  Tested
> x86_64.

All three backports look good to me for 2.15.

My only worry is that we break the testsuite for yet another target.

Shall we drop the testsuite addition from the backport?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[joseph at codesourcery dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

--- Comment #10 from joseph at codesourcery dot com <joseph at codesourcery dot com> 2012-06-25 22:09:16 UTC ---
On Mon, 25 Jun 2012, carlos_odonell at mentor dot com wrote:

> Shall we drop the testsuite addition from the backport?

The most conservative backport would be just patch 1/3, without the 
testsuite addition - 1/3 is fixing the regression in 2.15 (a regression 
introduced by the addition of the wordsize-64 version), the other patches 
are fixing a failure that showed up for powerpc with the new testcases but 
as far as I know that powerpc failure is not a regression.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[carlos_odonell at mentor dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

--- Comment #11 from Carlos O'Donell <carlos_odonell at mentor dot com> 2012-06-25 22:22:18 UTC ---
(In reply to comment #10)
> On Mon, 25 Jun 2012, carlos_odonell at mentor dot com wrote:
> 
> > Shall we drop the testsuite addition from the backport?
> 
> The most conservative backport would be just patch 1/3, without the 
> testsuite addition - 1/3 is fixing the regression in 2.15 (a regression 
> introduced by the addition of the wordsize-64 version), the other patches 
> are fixing a failure that showed up for powerpc with the new testcases but 
> as far as I know that powerpc failure is not a regression.

The Power failure is still a failure and should be fixed.

Please checkin all three patches *without* the testsuite addition.

I want to avoid people working with a stable branch having testsuite failures
show up out of thin air because we fixed a bug. That's OK for trunk, but not OK
for a stable branch. The new testsuite failure is difficult and time-consuming
for non-experts to diagnose. We hide this by fixing the bug, and using the
regression test to verify the fix, but then dropping the testcase from the
backport.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[jsm28 at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14048

Joseph Myers <jsm28 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #12 from Joseph Myers <jsm28 at gcc dot gnu.org> 2012-06-25 23:16:00 UTC ---
Fixed 2.15 (without testcase) by:

commit b640404bd8c9a281502ccc87ab2ed640c9b4c085
Author: Adhemerval Zanella <azanella@linux.vnet.ibm.com>
Date:   Tue Jun 5 21:31:24 2012 -0300

    Fix for wrong ldbl128-ibm fmodl commit
    (cherry picked from commit 1b671feb6115afbc90a7b37be4929d3e0394f311)

commit c95a1e5f35fa099eba9b2820b461edaaa7765539
Author: Adhemerval Zanella <azanella@linux.vnet.ibm.com>
Date:   Tue Jun 5 10:13:41 2012 -0300

    Fix ldbl128ibm fmodl for subnormals.
    (cherry picked from commit 34ae0b3270c67cae0c54ac98b693fdf7d010a206)

commit 2676061a91c99fa0b2633ceee881ea5bc31de4c2
Author: Joseph Myers <joseph@codesourcery.com>
Date:   Fri Jun 1 19:05:46 2012 +0000

    Fix fmod for subnormals (bug 14048).
    (cherry picked from commit c5bfe3d5ba29d36563f1e4bd4f8d7336093ee6fc)

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[jackie.rosen at hushmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14048

Jackie Rosen <jackie.rosen at hushmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jackie.rosen at hushmail dot com

--- Comment #13 from Jackie Rosen <jackie.rosen at hushmail dot com> ---
*** Bug 260998 has been marked as a duplicate of this bug. ***
Seen from the domain http://volichat.com
Page where seen: http://volichat.com/adult-chat-rooms
Marked for reference. Resolved as fixed @bugzilla.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[schwab at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=14048

Andreas Schwab <schwab at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|jackie.rosen at hushmail dot com   |

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug math/14048] fmod() incorrectly returns NaN for (some?) denormalized inputs

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14048

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.