[Bug fortran/60956] New: error reading (and writing) large text files in gfortran

[Bug fortran/60956] New: error reading (and writing) large text files in gfortran

[awrichar at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

            Bug ID: 60956
           Summary: error reading (and writing) large text files in
                    gfortran
           Product: gcc
           Version: 4.7.3
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: fortran
          Assignee: unassigned at gcc dot gnu.org
          Reporter: awrichar at gmail dot com

Created attachment 32678
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=32678&action=edit
demonstrates gfortran failed read/write of large text files

I'm getting some buggy behavior when working with very large data files
(complex, n=120^4) in gfortran 4.7.3.  I get "End of record" errors when
reading, and recently, "Cannot allocate memory" when writing.

Originally, I was able to successfuly write the large files, but when I try to
read them, gfortran issues the error:

>> test_complex_sort.f90 (unit = 100, file = 'sigma_hat_1.txt')
>> Fortran runtime error: End of file

If I explicitly loop through each element of the array during the READ instead,
gfortran has no issues. (Also if I use the Portland Group compiler and assign
the array all at once, there are no issues.)  If I reduce the variable size to
(120^4)/12, it runs, but (120^4)/10 fails.


I've written a test case (see attached), but this fails to WRITE the large
file, saying:

>> At line 19 of file test_complex_io.f90 (unit = 100, file = 'dummy.txt')
>> Fortran runtime error: Cannot allocate memory

The line in question (19) is simply
>> WRITE(100, *) u
and runs fine for cases (n.LT.10**8).  (All cases run fine when compiling with
the Portland Group Fortran compiler.)

Lastly, if I generate the 4GB text file using PG Fortran, then comment out
lines 18-20 in the attached file and complie with gfortran, I get a seg-fault
during the READ for the n=10^8 case.

These errors occurs on a computer with dual Xeon E-2690v2 processors and 256GB
(not a typo) of RAM, under Ubuntu 13.04 with gfortran 4.7.3.  (NOTE: When using
the PG Fortran compiler, the attached code uses ~3.2GB of memory, and writes a
4GB text file for the largest test case.)

Has anyone else experienced similar issues?  Is this reproducible for anyone
else, or in newer gfortran versions?  I haven't found any recent bugs reports
that seem similar, but I easily could have overlooked something.


--Andy

[Bug fortran/60956] error reading (and writing) large text files in gfortran

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

--- Comment #3 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
I took a closer look at this one. The error I am getting is running out of
allocatable space in realloc used in fbuf.c (fbuf_alloc).  We are allocating
one byte at a time, and just keep trying to douple the buffer when its not big
enough.

I think we need to check for some max buffer size and when we approach it,
flush the fbuf and continue.

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

Thomas Koenig <tkoenig at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |5.0
            Summary|error reading (and writing) |[4.8/4.9/5 Regression]
                   |large text files in         |error reading (and writing)
                   |gfortran                    |large text files in
                   |                            |gfortran

--- Comment #4 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
Based on Dominique's analysis, this is a regression.

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[burnus at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

Tobias Burnus <burnus at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |burnus at gcc dot gnu.org

--- Comment #5 from Tobias Burnus <burnus at gcc dot gnu.org> ---
(In reply to Jerry DeLisle from comment #3)
> I took a closer look at this one. The error I am getting is running out of
> allocatable space in realloc used in fbuf.c (fbuf_alloc).  We are allocating
> one byte at a time, and just keep trying to douple the buffer when its not
> big enough.
> 
> I think we need to check for some max buffer size and when we approach it,
> flush the fbuf and continue.

I think we cannot completely get rid of buffering a complete record - if we do
formatted I/O (contrary to list-directed I/O and unformatted). The problem is
that the format can contain T, TL and TR.

However, the test case only contains list-directed I/O, which doesn't require
to cache the whole record.  [For formatted, one could check the format
statement - but there could be some performance trade off.]

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

--- Comment #6 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
For formatted, we already are parsing along the format string and we know when
its list directed, so I think we could either call fbuf_alloc with a flag or
call a different version of it that would auto flush or something similar. 
Maybe not buffer and let the OS do its thing.

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

--- Comment #7 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
I have finished reviewing the code.  We do have specific functions called for
list directed reads and writes where we know we will have no back and forth
tabbing issues, so we can use fixed length buffering for these cases.

write.c (list_formatted_write_scalar).  I think set the buffering style in
there by setting a function pointer or similar to switch the behavior. Very low
overhead in doing so.  (similar in the READ case)

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

--- Comment #8 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Patch

https://gcc.gnu.org/ml/gcc-patches/2015-02/msg00449.html

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

--- Comment #9 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Author: jvdelisle
Date: Sat Feb  7 15:13:15 2015
New Revision: 220505

URL: https://gcc.gnu.org/viewcvs?rev=220505&root=gcc&view=rev
Log:
2015-02-07  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

    PR libgfortran/60956
    * io/fbuf.c (fbuf_flush_list): New function that only flushes
    if current fbuf position exceeds a limit.
    * io/fbuf.h: Declare the new function.
    * io/io.h (enum unit_mode): Add two new modes.
    * io/list_read.c (list_formatted_read_scalar): Call new function.
    * io/write.c: Include fbuf.h. (list_formatted_write_scalar):
    Call new function.

Modified:
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/io/fbuf.c
    trunk/libgfortran/io/fbuf.h
    trunk/libgfortran/io/io.h
    trunk/libgfortran/io/list_read.c
    trunk/libgfortran/io/write.c

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P3                          |P4
                 CC|                            |jakub at gcc dot gnu.org

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

--- Comment #10 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Author: jvdelisle
Date: Mon Mar 23 00:16:01 2015
New Revision: 221578

URL: https://gcc.gnu.org/viewcvs?rev=221578&root=gcc&view=rev
Log:
2015-03-22  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

    PR libgfortran/60956
    Backport from mainline
    * io/fbuf.c (fbuf_flush_list): New function that only flushes
    if current fbuf position exceeds a limit.
    * io/fbuf.h: Declare the new function.
    * io/io.h (enum unit_mode): Add two new modes.
    * io/list_read.c (list_formatted_read_scalar): Call new function.
    * io/write.c: Include fbuf.h. (list_formatted_write_scalar):
    Call new function.

Modified:
    branches/gcc-4_9-branch/libgfortran/ChangeLog
    branches/gcc-4_9-branch/libgfortran/io/fbuf.c
    branches/gcc-4_9-branch/libgfortran/io/fbuf.h
    branches/gcc-4_9-branch/libgfortran/io/io.h
    branches/gcc-4_9-branch/libgfortran/io/list_read.c
    branches/gcc-4_9-branch/libgfortran/io/write.c

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

--- Comment #11 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Author: jvdelisle
Date: Mon Mar 23 01:32:15 2015
New Revision: 221584

URL: https://gcc.gnu.org/viewcvs?rev=221584&root=gcc&view=rev
Log:
2015-03-22  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

    PR libgfortran/60956
    Backport from mainline
    * io/fbuf.c (fbuf_flush_list): New function that only flushes
    if current fbuf position exceeds a limit.
    * io/fbuf.h: Declare the new function.
    * io/io.h (enum unit_mode): Add two new modes.
    * io/list_read.c (list_formatted_read_scalar): Call new function.
    * io/write.c: Include fbuf.h. (list_formatted_write_scalar):
    Call new function.

Modified:
    branches/gcc-4_8-branch/libgfortran/ChangeLog
    branches/gcc-4_8-branch/libgfortran/io/fbuf.c
    branches/gcc-4_8-branch/libgfortran/io/fbuf.h
    branches/gcc-4_8-branch/libgfortran/io/io.h
    branches/gcc-4_8-branch/libgfortran/io/list_read.c
    branches/gcc-4_8-branch/libgfortran/io/write.c

[Bug fortran/60956] [4.8/4.9/5 Regression] error reading (and writing) large text files in gfortran

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60956

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #12 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Backported and closing.