[Bug fortran/61628] New: A program that reads from a file with stream access and uses pack() suddenly stops

[Bug fortran/61628] New: A program that reads from a file with stream access and uses pack() suddenly stops

[arjen.markus895 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

            Bug ID: 61628
           Summary: A program that reads from a file with stream access
                    and uses pack() suddenly stops
           Product: gcc
           Version: 4.8.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: fortran
          Assignee: unassigned at gcc dot gnu.org
          Reporter: arjen.markus895 at gmail dot com

Created attachment 33016
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=33016&action=edit
Example data file as used in the program, compressed

The following program suddenly stops after the statement with pack() -- without
any message -- but it continues if you comment out the read statement:

! readprint.f90 --
!     Read and print some data from a binary file
!
program readprint
    implicit none

    integer                            :: noseg, nx, ny, dummy, i
    integer, dimension(:), allocatable :: matrix

    open( 10, file = 'binary_data', form = 'unformatted', access = 'stream' )
    read( 10 ) nx, ny, (dummy, i = 1,4)

    allocate( matrix(nx*ny) )

!
! If the READ statement is commented, then the program continues, otherwise
! it stops without any message
!
    read( 10 ) matrix
    write(*,*) nx, ny, size(matrix)
    write(*,*) pack( matrix, matrix /= 1 )
    write(*,*) matrix(1:1000)
    write(*,*) 'done'

end program readprint

I have not tried this on another data file, but the behaviour is odd enough
that I want to report this.

Note: gfortran -v gives:

Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=c:/mingw/bin/../libexec/gcc/mingw32/4.8.1/lto-wrapper.exe
Target: mingw32
Configured with: ../gcc-4.8.1/configure --prefix=/mingw --host=mingw32
--build=mingw32 --without-pic --enable-shared --enable-static --with-gnu-ld
--enable-lto --enable-libssp --disable-multilib
--enable-languages=c,c++,fortran,objc,obj-c++,ada --disable-sjlj-exceptions
--with-dwarf2 --disable-win32-registry --enable-libstdcxx-debug
--enable-version-specific-runtime-libs
--with-gmp=/usr/src/pkg/gmp-5.1.2-1-mingw32-src/bld
--with-mpc=/usr/src/pkg/mpc-1.0.1-1-mingw32-src/bld --with-mpfr=
--with-system-zlib --with-gnu-as --enable-decimal-float=yes --enable-libgomp
--enable-threads --with-libiconv-prefix=/mingw32 --with-libintl-prefix=/mingw
--disable-bootstrap LDFLAGS=-s CFLAGS=-D_USE_32BIT_TIME_T
Thread model: win32
gcc version 4.8.1 (GCC)

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jvdelisle at gcc dot gnu.org

--- Comment #1 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Works fine for me on x86-64-linux.  I suspect this is a mingw specific issue on
Windows.  What version of windows are you running? Unfortunately this can make
a difference, though it shouldn't.  I do not have mingw setup on any of my duel
boot machines.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

Dominique d'Humieres <dominiq at lps dot ens.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2014-06-27
     Ever confirmed|0                           |1

--- Comment #2 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
Arjen,

Are you sure that you have attached the right file? When using tar I get "tar:
A lone zero block at 2498" and the test succeeds as for Jerry.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #4 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
The reason I asked about the windows version is that at work we just got all
upgraded to Windows 7 as a standard (vs XP) and we have been experiancing all
sorts of strange behavior, including the first blue screen of death I have seen
in years.  But, this is off topic.

I would need to get set up on my dual boot machine to help debug this further.
Where did you get your gfortran from and I will try.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[arjen.markus895 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #5 from Arjen Markus <arjen.markus895 at gmail dot com> ---
I am not entirely certain, but I think it was simply part of the MinGW
installation. If not, I used the
https://gcc.gnu.org/wiki/GFortranBinaries site to locate one.

Regards,

Arjen

2014-06-29 16:39 GMT+02:00 jvdelisle at gcc dot gnu.org
<gcc-bugzilla@gcc.gnu.org>:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628
>
> --- Comment #4 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
> The reason I asked about the windows version is that at work we just got all
> upgraded to Windows 7 as a standard (vs XP) and we have been experiancing all
> sorts of strange behavior, including the first blue screen of death I have seen
> in years.  But, this is off topic.
>
> I would need to get set up on my dual boot machine to help debug this further.
> Where did you get your gfortran from and I will try.
>
> --
> You are receiving this mail because:
> You reported the bug.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #6 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
I went to gfortran binaries and grabbed the mingw installer and installed it on
my Win7 side.

Compile the program, Un compressed the data file and ran it.

C:\Users\jerry\prs\pr61628>gfortran pr61628.f90

C:\Users\jerry\prs\pr61628>a
         681         469      319389

It appears to work fine.  This is with the gfortran 4.10 verrsion as of today.

Maybe try removing gfortran and re-install with todays.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #8 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
OK my misunderstanding.  I thought the problem was that the program just hung,
which I did not see. Well, I can say I have something to look at anyway.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[arjen.markus895 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #9 from Arjen Markus <arjen.markus895 at gmail dot com> ---
Hi Jerry,

no problem - I can imagine that you did not see anything wrong. It
took me sometime to realise myself what was going on when I discovered
this problem.

Regards,

Arjen

2014-07-01 3:55 GMT+02:00 jvdelisle at gcc dot gnu.org
<gcc-bugzilla@gcc.gnu.org>:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628
>
> --- Comment #8 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
> OK my misunderstanding.  I thought the problem was that the program just hung,
> which I did not see. Well, I can say I have something to look at anyway.
>
> --
> You are receiving this mail because:
> You reported the bug.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #10 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
Just to be sure I am using the right file, its md5 signature is
MD5 (binary_data) = 681215e5351ab45a494c044719776690
I am also attaching the output file which looks OK, but a double check does not
hurt.
Valgrind reports a quite large memory leak:

==2066== HEAP SUMMARY:
==2066==     in use at exit: 1,302,891 bytes in 383 blocks
==2066==   total heap usage: 501 allocs, 118 frees, 11,311,380 bytes allocated
==2066== 
==2066== 1,277,556 bytes in 1 blocks are definitely lost in loss record 77 of
77
==2066==    at 0x7011: malloc (vg_replace_malloc.c:295)
==2066==    by 0x10000087F: MAIN__ (pr61628.f90:13)
==2066==    by 0x100000D08: main (pr61628.f90:25)
==2066== 
==2066== LEAK SUMMARY:
==2066==    definitely lost: 1,277,556 bytes in 1 blocks
==2066==    indirectly lost: 0 bytes in 0 blocks
==2066==      possibly lost: 0 bytes in 0 blocks
==2066==    still reachable: 0 bytes in 0 blocks
==2066==         suppressed: 25,335 bytes in 382 blocks
==2066== 
==2066== For counts of detected and suppressed errors, rerun with: -v
==2066== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4934 from 60)

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #11 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
Created attachment 33038
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=33038&action=edit
gzipped output of the executable on x86_64-apple-darwin13 r212179

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[arjen.markus895 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #12 from Arjen Markus <arjen.markus895 at gmail dot com> ---
Hi Dominique,

that is the sort of output I would expect indeed. The memory leak is
to be expected: the program does not explicitly release the allocated
array/matrix.

It inspired me to check the same program with Cygwin - it works fine
under Cygwin (details of gfortran below), so it might be a
MinGW-specific problem, as Jerry suggested - ugh.

Regards,

Arjen

----
gfortran under Cygwin:
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-cygwin/4.8.3/lto-wrapper.exe
Target: x86_64-pc-cygwin
Configured with:
/cygdrive/i/szsz/tmpp/cygwin64/gcc/gcc-4.8.3-2/src/gcc-4.8.3/configure
--srcdir=/cygdrive/i/szsz/tmpp/cygwin64/gcc/gcc-4.8.3-2/src/gcc-4.8.3
--prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin
--libexecdir=/usr/libexec --datadir=/usr/share --localstatedir=/var
--sysconfdir=/etc --libdir=/usr/lib --datarootdir=/usr/share
--docdir=/usr/share/doc/gcc --htmldir=/usr/share/doc/gcc/html -C
--build=x86_64-pc-cygwin --host=x86_64-pc-cygwin
--target=x86_64-pc-cygwin --without-libiconv-prefix
--without-libintl-prefix --enable-shared --enable-shared-libgcc
--enable-static --enable-version-specific-runtime-libs
--enable-bootstrap --disable-__cxa_atexit --with-dwarf2
--with-tune=generic
--enable-languages=ada,c,c++,fortran,lto,objc,obj-c++
--enable-graphite --enable-threads=posix --enable-libatomic
--enable-libgomp --disable-libitm --enable-libquadmath
--enable-libquadmath-support --enable-libssp --enable-libada
--enable-libgcj-sublibs --disable-java-awt --disable-symvers
--with-ecj-jar=/usr/share/java/ecj.jar --with-gnu-ld --with-gnu-as
--with-cloog-include=/usr/include/cloog-isl --without-libiconv-prefix
--without-libintl-prefix --with-system-zlib --libexecdir=/usr/lib
Thread model: posix
gcc version 4.8.3 (GCC)

2014-07-01 12:59 GMT+02:00 dominiq at lps dot ens.fr
<gcc-bugzilla@gcc.gnu.org>:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628
>
> --- Comment #11 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
> Created attachment 33038
>   --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=33038&action=edit
> gzipped output of the executable on x86_64-apple-darwin13 r212179
>
> --
> You are receiving this mail because:
> You reported the bug.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #13 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Well I have been playing around a little.  I wanted to see if the problem was
in pack, so I did this:

    read( 10 ) matrix
    write(*,*) nx, ny, size(matrix)
    test = pack( matrix, matrix /= 1 )
    write (*,*) test(1:5227)

The array 'test' is declared and allocated exactly as 'matrix'.

What I found is that if I set the range of test as shown, the write will give
all the values up to that size. Greater then 5227 and things quit working and
the write gives nothing.

I still have no idea what's wrong. But it acts like either the allocation is
not working correctly or a pointer is getting clobbered.  If I was going to
bet, it would be memory allocation problem.

I will be doing some more tinkering as I have time.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #14 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Here is a reduced test case.  It has nothing to do with pack or allocate.

program readprint
    implicit none

    integer, dimension(319389) :: matrix
    write(*,*) 'begin'
    write(*,*) matrix
    write(*,*) 'done'
end program readprint

This program writes 'begin' and then returns to the command prompt.

It will write out matrix up to 5227 elements.

I am wondering if the mingw build is broken.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[arjen.markus895 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #15 from Arjen Markus <arjen.markus895 at gmail dot com> ---
Hi Jerry,

I am glad you have been able to reproduce the problem and even reduce
the program to a bare minimum.

What a curious number to fail upon. What I can do is try and find a
different version of gfortran under MinGW and try again.

Is it somehow connected to the size of the matrix or is 5227 some
funny magical number?

Regards,

Arjen

2014-07-02 3:55 GMT+02:00 jvdelisle at gcc dot gnu.org
<gcc-bugzilla@gcc.gnu.org>:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628
>
> --- Comment #14 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
> Here is a reduced test case.  It has nothing to do with pack or allocate.
>
> program readprint
>     implicit none
>
>     integer, dimension(319389) :: matrix
>     write(*,*) 'begin'
>     write(*,*) matrix
>     write(*,*) 'done'
> end program readprint
>
> This program writes 'begin' and then returns to the command prompt.
>
> It will write out matrix up to 5227 elements.
>
> I am wondering if the mingw build is broken.
>
> --
> You are receiving this mail because:
> You reported the bug.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #16 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
There is nothing special about the 5227, more to do with array size.

I did go to the official mingw site and used thier "get" utility and installled
the latest of everything and get the exact same result with the 4.8 version of
gfortran.  We will need a MinGW person to investigate further.

[Bug fortran/61628] A program that reads from a file with stream access and uses pack() suddenly stops

[arjen.markus895 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #17 from Arjen Markus <arjen.markus895 at gmail dot com> ---
Hi Jerry,

thank you very much for looking into this problem. I stopped reducing
the test case because I thought it had to do with pack creating a
temporary array, but obviously the problem has nothing to do with
this.

Regards,

Arjen

2014-07-02 17:03 GMT+02:00 jvdelisle at gcc dot gnu.org
<gcc-bugzilla@gcc.gnu.org>:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628
>
> --- Comment #16 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
> There is nothing special about the 5227, more to do with array size.
>
> I did go to the official mingw site and used thier "get" utility and installled
> the latest of everything and get the exact same result with the 4.8 version of
> gfortran.  We will need a MinGW person to investigate further.
>
> --
> You are receiving this mail because:
> You reported the bug.

[Bug fortran/61628] [MinGW)Write of medium sized or larger matrix fails without error message.

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW
            Summary|A program that reads from a |[MinGW)Write of medium
                   |file with stream access and |sized or larger matrix
                   |uses pack() suddenly stops  |fails without error
                   |                            |message.

--- Comment #18 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Confirmed on 32 bit MinGW.

Revised description.

[Bug fortran/61628] [MinGW)Write of medium sized or larger matrix fails without error message.

[burnus at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

Tobias Burnus <burnus at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |burnus at gcc dot gnu.org

--- Comment #20 from Tobias Burnus <burnus at gcc dot gnu.org> ---
That's a very odd problem. The error message tells us that the size of an array
element (here: 4 byte) does not fit into the SIZE part of the array descriptor.
Obviously, the number "4" (bytes) does fit there.

The questions are:
a) Why does it have the wrong value under Win64 - it should take 4 not some
other value.
b) Why does this message doesn't pop up under Win32 (or Linux)?
c) Why does it fail under Win32?

One thing which looks suspicious is the code for gfc_max_array_element_size; it
should calculate sizeof(ptrdiff_t) - bits used for other purposes. For some
reasons it uses "long" which has no relation to ptrdiff_t and is 32 bytes under
Windows (both 32 and 64 bit Windows). That should be fixed by the following
patch (untested).

However, it still does not explain why it fails with an error message under
Win64 - and only under Win64 and not under Win32. Still, I subspect that the
sizeof(long) == 32 is behind the problems we are seening. Presumably, Cygwin
uses sizeof(long) == 64, which would explain why it only fails with Windows.
(It doesn't explain the other issues.)


[Side remark: I am looking for the new array descriptor to get rid of this
mess. That one has a simple "size_t elem_len", an "int rank" and another
element for the data type.]

--- a/gcc/fortran/trans-types.c
+++ b/gcc/fortran/trans-types.c
@@ -956,7 +956,7 @@ gfc_init_types (void)
   n = TYPE_PRECISION (gfc_array_index_type) - GFC_DTYPE_SIZE_SHIFT;
   gfc_max_array_element_size
-    = wide_int_to_tree (long_unsigned_type_node,
+    = wide_int_to_tree (size_type_node,
                        wi::mask (n, UNSIGNED,
-                                 TYPE_PRECISION (long_unsigned_type_node)));
+                                 TYPE_PRECISION (size_type_node)));

   boolean_type_node = gfc_get_logical_type (gfc_default_logical_kind);

[Bug fortran/61628] [MinGW)Write of medium sized or larger matrix fails without error message.

[burnus at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #21 from Tobias Burnus <burnus at gcc dot gnu.org> ---
Author: burnus
Date: Sat Jul 12 19:05:52 2014
New Revision: 212485

URL: https://gcc.gnu.org/viewcvs?rev=212485&root=gcc&view=rev
Log:
2014-07-12  Tobias Burnus  <burnus@net-b.de>

        PR fortran/61628
        * trans-types.c (gfc_init_types): Fix data-type bug
        with gfc_max_array_element_size.


Modified:
    trunk/gcc/fortran/ChangeLog
    trunk/gcc/fortran/trans-types.c

[Bug fortran/61628] [MinGW)Write of medium sized or larger matrix fails without error message.

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61628

--- Comment #22 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Arjen, any further results or information on this bug?