[Bug libfortran/54736] New: GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[Bug libfortran/54736] New: GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[shart6 at utk dot edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

             Bug #: 54736
           Summary: GFORTRAN_CONVERT_UNIT causes malloc error on several
                    platforms
    Classification: Unclassified
           Product: gcc
           Version: 4.7.1
            Status: UNCONFIRMED
          Severity: major
          Priority: P3
         Component: libfortran
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: shart6@utk.edu


I have a code that has multiple large data files that are written in big 
endian format.  To deal with this on many platforms, a script was written 
that, among various other things, sets a few units to read as big endian 
by using GFORTRAN_CONVERT_UNIT='native;big_endian:60-70,80-89'.  However, 
when the Fortran program is called from the script I get:

scale: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) 
(((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct 
malloc_chunk, fd)))) && old_size == 0) || 
((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof 
(struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 
* (sizeof(size_t))) - 1))) && ((old_to
p)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.

This reliably happens on any Linux platform I've tested it on *except* 
Fedora/RHEL.

I've been able to reliably reproduce the problem on several 32-bit 
machines (Arch Linux (GCC 4.7.1), OpenSUSE 12.2, Ubuntu 12.10 Beta 1) by 
creating and running these two programs:

Write program:

[shane@shane-laptop ~/temp/testgfortran]$ cat test_write.f90
program test_write
implicit none

integer, parameter :: NUM = 10
integer :: i

open(unit=88,form='unformatted',convert='big_endian')
do i = 1,NUM
  write (88) i
end do

close(88)

end program test_write

Read Program:

[shane@shane-laptop ~/temp/testgfortran]$ cat test_read.f90
program test_write
implicit none

integer, parameter :: NUM = 10
integer :: readInt
integer :: i

open(unit=88,form='unformatted')
do i = 1,NUM
  read (88) readInt
  write (*,*) readInt
end do

close(88)

end program test_write

And testing:

[shane@shane-laptop ~/temp/testgfortran]$ ./test_write
[shane@shane-laptop ~/temp/testgfortran]$ ./test_read
    16777216
At line 10 of file test_read.f90 (unit = 88, file = 'fort.88')
Fortran runtime error: End of file
[shane@shane-laptop ~/temp/testgfortran]$ 
GFORTRAN_CONVERT_UNIT='native;big_endian:88' ./test_read
test_read: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) 
(((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct 
malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= 
(unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))
+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && 
((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' 
failed.
Aborted
[shane@shane-laptop ~/temp/testgfortran]$ 
GFORTRAN_CONVERT_UNIT='big_endian' ./test_read
           1
           2
           3
           4
           5
           6
           7
           8
           9
          10


Obviously the first invocation of test_read fails because the endianess 
is wrong, but the second one should work.  Running gdb yields:

Program received signal SIGABRT, Aborted.
0xb7fdd424 in __kernel_vsyscall ()

However, the above test program does work on all of the 64-bit machines 
I've tested.

I found that by commenting out the lines 582-583 in libgfortran/runtime/
environ.c I can get it to work.

I don't really know why it's accessing an element in the elist structure 
that hasn't been allocated yet.  This loop doesn't seem to do anything to 
me.  Since I don't know the reprocussions of just commenting out stuff 
willy nilly I'll leave the implementation details to someone with more 
knowledge, but this seems like a good place to start.

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[tkoenig at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

Thomas Koenig <tkoenig at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
   Last reconfirmed|                            |2012-09-28
                 CC|                            |tkoenig at gcc dot gnu.org
         AssignedTo|unassigned at gcc dot       |tkoenig at gcc dot gnu.org
                   |gnu.org                     |
     Ever Confirmed|0                           |1

--- Comment #1 from Thomas Koenig <tkoenig at gcc dot gnu.org> 2012-09-28 18:38:13 UTC ---
Mine.

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[tkoenig at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

Thomas Koenig <tkoenig at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |wrong-code
                URL|                            |http://gcc.gnu.org/ml/gcc-p
                   |                            |atches/2012-09/msg01962.htm
                   |                            |l

--- Comment #2 from Thomas Koenig <tkoenig at gcc dot gnu.org> 2012-09-30 16:38:10 UTC ---
Patch posted.

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[shart6 at utk dot edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

--- Comment #3 from Shane Hart <shart6 at utk dot edu> 2012-09-30 19:12:48 UTC ---
The patch does get rid of memory corruption.  However, there seem to be some
problems with search_unit returning true if an exception is found when there is
only one exception.

If n_elist = 1, then high = low = 0, and the funtion will always return 0, even
if the unit passed in to search for is in the exception list.

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[tkoenig at netcologne dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

--- Comment #4 from tkoenig at netcologne dot de <tkoenig at netcologne dot de> 2012-09-30 20:24:03 UTC ---
Am 30.09.2012 21:12, schrieb shart6 at utk dot edu:
> If n_elist = 1, then high = low = 0, and the funtion will always return 0, even
> if the unit passed in to search for is in the exception list.

If there are n_elist exceptions, then they can be found in
elist[0], ..., elist[n_elist-1].

If there is one exception, then it can be found at elist[0].

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[shart6 at utk dot edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

--- Comment #5 from Shane Hart <shart6 at utk dot edu> 2012-09-30 20:56:39 UTC ---
(In reply to comment #4)
> Am 30.09.2012 21:12, schrieb shart6 at utk dot edu:
> > If n_elist = 1, then high = low = 0, and the funtion will always return 0, even
> > if the unit passed in to search for is in the exception list.
> 
> If there are n_elist exceptions, then they can be found in
> elist[0], ..., elist[n_elist-1].
> 
> If there is one exception, then it can be found at elist[0].

That is true, but the function will not return 1 (true, a match was found in
elist) when there is only one exception stored.  It will skip over the while
loop, set *ip to 0, and return 0 (false, the exception was not found!).

This can be illustrated:

1) Call the program, setting only one exception (in this case unit 21):

[shane@shane-laptop ~/temp/testgfortran]$
GFORTRAN_CONVERT_UNIT='native;big_endian:21' gnu_wrap gdb ./test_read

2) Set a break point at where the library enquires as to the endianness of the
file to be opened:

Breakpoint 1, _gfortrani_get_unformatted_convert (unit=21) at
../../../libgfortran/runtime/environ.c:848

3) We will step into search_unit (with unit = 21).  Everything is great.
however, since n_elist = 1, high = low = 0, and the while loop is skipped over:

(gdb) step
471       while (high - low > 1)
(gdb) step
485       if (unit > elist[high].unit)

4) And 0 is returned (unit exception not found!)

(gdb) step
490       return 0;

5) The library is told to open unit 21 (which is in the exception list at 0) as
default endianness:

_gfortrani_get_unformatted_convert (unit=21) at
../../../libgfortran/runtime/environ.c:856
856         return def;

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[shart6 at utk dot edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

--- Comment #6 from Shane Hart <shart6 at utk dot edu> 2012-10-01 19:29:24 UTC ---
Latest patch fixed it.

Thanks a lot for the time you put into this!

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[tkoenig at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

--- Comment #7 from Thomas Koenig <tkoenig at gcc dot gnu.org> 2012-10-06 13:04:38 UTC ---
Author: tkoenig
Date: Sat Oct  6 13:04:35 2012
New Revision: 192158

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=192158
Log:
2012-10-06  Thomas König  <tkoenig@gcc.gnu.org>

    PR libfortran/54736
    * runtime/environ.c (search_unit):  Correct logic
    for binary search.
    (mark_single):  Fix index errors.


Modified:
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/runtime/environ.c

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[tkoenig at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

--- Comment #8 from Thomas Koenig <tkoenig at gcc dot gnu.org> 2012-10-12 18:56:23 UTC ---
Author: tkoenig
Date: Fri Oct 12 18:56:16 2012
New Revision: 192408

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=192408
Log:
2012-10-06  Thomas König  <tkoenig@gcc.gnu.org>

    PR libfortran/54736
    Backport from trunk
    * runtime/environ.c (search_unit):  Correct logic
    for binary search.
    (mark_single):  Fix index errors.


Modified:
    branches/gcc-4_7-branch/libgfortran/ChangeLog
    branches/gcc-4_7-branch/libgfortran/runtime/environ.c

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[tkoenig at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

--- Comment #9 from Thomas Koenig <tkoenig at gcc dot gnu.org> 2012-10-12 19:38:09 UTC ---
Author: tkoenig
Date: Fri Oct 12 19:38:04 2012
New Revision: 192411

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=192411
Log:
2012-10-12  Thomas König  <tkoenig@gcc.gnu.org>

    PR libfortran/54736
    libgfortran/Changelog:  Fix date of last commit.

Modified:
    branches/gcc-4_7-branch/libgfortran/ChangeLog

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[tkoenig at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

--- Comment #10 from Thomas Koenig <tkoenig at gcc dot gnu.org> 2012-10-21 13:43:40 UTC ---
Author: tkoenig
Date: Sun Oct 21 13:43:32 2012
New Revision: 192653

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=192653
Log:
2012-10-21  Thomas König  <tkoenig@gcc.gnu.org>

    PR libfortran/54736
    Backport from trunk
    * runtime/environ.c (search_unit):  Correct logic
    for binary search.
    (mark_single):  Fix index errors.


Modified:
    branches/gcc-4_6-branch/libgfortran/ChangeLog
    branches/gcc-4_6-branch/libgfortran/runtime/environ.c

[Bug libfortran/54736] GFORTRAN_CONVERT_UNIT causes malloc error on several platforms

[tkoenig at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=54736

Thomas Koenig <tkoenig at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #11 from Thomas Koenig <tkoenig at gcc dot gnu.org> 2012-10-21 13:44:12 UTC ---
Also fixed on 4.6, closing.