[Bug libfortran/50016] New: The fortran program 's io is very slow

[Bug libfortran/50016] New: The fortran program 's io is very slow

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

           Summary: The fortran program 's io is very slow
           Product: gcc
           Version: 4.6.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libfortran
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: xunxun1982@gmail.com
              Host: *-*-mingw*
            Target: *-*-mingw*
             Build: *-*-mingw*


I found that if I write some binary file operation in gfortran4.6.1, the io
becomes very slow, but gfortran4.5.* is very fast.

[Bug libfortran/50016] The fortran program 's io is very slow

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #1 from PcX <xunxun1982 at gmail dot com> 2011-08-07 17:14:10 UTC ---
Created attachment 24942
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=24942
test program

[Bug libfortran/50016] The fortran program 's io is very slow

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

PcX <xunxun1982 at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |xunxun1982 at gmail dot com

--- Comment #2 from PcX <xunxun1982 at gmail dot com> 2011-08-07 17:17:21 UTC ---
gfortran -O3 writebin.f90

gfortran-4.5 only cost 0.725s

gfortran-4.6 cost very long time.

[Bug libfortran/50016] The fortran program 's io is very slow

[jvdelisle at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jvdelisle at gcc dot
                   |                            |gnu.org

--- Comment #3 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> 2011-08-07 23:06:37 UTC ---
I can not reproduce this on my system.  I did notice a warning about line
truncation on the long line in the example.  You may want to use line
continuations for that one like this:

      write(13,rec=jj*i+j)nlgs(1,j,i),dlgs(200,j,i),nlgs(2,j,i),dlgs(199,j,i),&
      & nlgs(3,j,i),dlgs(198,j,i),nlgs(4,j,i),dlgs(197,j,i),nlgs(5:8,j,i),&
      & dlgs(154:190,j,i)

Sometimes if you get truncated in just the right place, the program will still
compile and run, but not the way you think it is.

[Bug libfortran/50016] The fortran program 's io is very slow

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #4 from PcX <xunxun1982 at gmail dot com> 2011-08-07 23:22:55 UTC ---
(In reply to comment #3)
> I can not reproduce this on my system.  I did notice a warning about line
> truncation on the long line in the example.  You may want to use line
> continuations for that one like this:
> 
>       write(13,rec=jj*i+j)nlgs(1,j,i),dlgs(200,j,i),nlgs(2,j,i),dlgs(199,j,i),&
>       & nlgs(3,j,i),dlgs(198,j,i),nlgs(4,j,i),dlgs(197,j,i),nlgs(5:8,j,i),&
>       & dlgs(154:190,j,i)
> 
> Sometimes if you get truncated in just the right place, the program will still
> compile and run, but not the way you think it is.

Yea, I modify it as your follows, but it will also take a very long time (so I
break it when it runs more than 1 minute).
At the same time, gcc 4.5.3 will only take some seconds.

btw, I use Win7 64bit, the CRT is MinGW64 project's CRT.
But my gcc4.5.3 also use MinGW64 CRT, so I think that's the gcc4.6's reason,
not CRT's.

[Bug libfortran/50016] The fortran program 's io is very slow

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #5 from PcX <xunxun1982 at gmail dot com> 2011-08-07 23:24:46 UTC ---
Created attachment 24946
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=24946
fixed test

[Bug libfortran/50016] The fortran program 's io is very slow

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

Janne Blomqvist <jb at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jb at gcc dot gnu.org

--- Comment #6 from Janne Blomqvist <jb at gcc dot gnu.org> 2011-08-08 11:51:34 UTC ---
Maybe this is MingW specific?

I haven't tested this,  but here goes my guess:

Since in the write loop we're jumping back and forth across the file, the
buffer must be frequently flushed; in libgfortran/io/unix.c (buf_flush) we have
in the trunk and 4.6 branch:

#ifdef _WIN32
  _commit (s->fd);
#endif

If this is some kind of win32 equivalent to fsync(), i.e. commit stuff to disk,
then it's no surprise it's slow?

This code was added to buf_flush() as a response to
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44698 .

[Bug libfortran/50016] The fortran program 's io is very slow

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #7 from PcX <xunxun1982 at gmail dot com> 2011-08-08 12:05:47 UTC ---
(In reply to comment #6)
> Maybe this is MingW specific?
> 
> I haven't tested this,  but here goes my guess:
> 
> Since in the write loop we're jumping back and forth across the file, the
> buffer must be frequently flushed; in libgfortran/io/unix.c (buf_flush) we have
> in the trunk and 4.6 branch:
> 
> #ifdef _WIN32
>   _commit (s->fd);
> #endif
> 
> If this is some kind of win32 equivalent to fsync(), i.e. commit stuff to disk,
> then it's no surprise it's slow?
> 
> This code was added to buf_flush() as a response to
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44698 .

That sounds reasonable.
So this bug is fixed in the trunk, but haven't been backported to 4.6?

[Bug libfortran/50016] The fortran program 's io is very slow

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #8 from PcX <xunxun1982 at gmail dot com> 2011-08-08 12:35:12 UTC ---
(In reply to comment #7)
> (In reply to comment #6)
> > Maybe this is MingW specific?
> > 
> > I haven't tested this,  but here goes my guess:
> > 
> > Since in the write loop we're jumping back and forth across the file, the
> > buffer must be frequently flushed; in libgfortran/io/unix.c (buf_flush) we have
> > in the trunk and 4.6 branch:
> > 
> > #ifdef _WIN32
> >   _commit (s->fd);
> > #endif
> > 
> > If this is some kind of win32 equivalent to fsync(), i.e. commit stuff to disk,
> > then it's no surprise it's slow?
> > 
> > This code was added to buf_flush() as a response to
> > http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44698 .
> 
> That sounds reasonable.
> So this bug is fixed in the trunk, but haven't been backported to 4.6?

Sorry, I'm wrong.
I think I can try 4.7 to test this.

[Bug libfortran/50016] The fortran program 's io is very slow

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #9 from PcX <xunxun1982 at gmail dot com> 2011-08-08 12:38:11 UTC ---
CC list add Kai.

[Bug libfortran/50016] The fortran program 's io is very slow

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

PcX <xunxun1982 at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
      Known to fail|                            |4.7.0

--- Comment #10 from PcX <xunxun1982 at gmail dot com> 2011-08-08 13:01:58 UTC ---
I test it using gcc4.7.0 on Win7, and it also takes a very long time.

ps: that works fine on Linux.

[Bug libfortran/50016] The fortran program 's io is very slow

[cgl_lgs at 163 dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

好小爱新 <cgl_lgs at 163 dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |cgl_lgs at 163 dot com

--- Comment #11 from 好小爱新 <cgl_lgs at 163 dot com> 2011-08-09 07:46:28 UTC ---
(In reply to comment #3)
> I can not reproduce this on my system.  I did notice a warning about line
> truncation on the long line in the example.  You may want to use line
> continuations for that one like this:
> 
>       write(13,rec=jj*i+j)nlgs(1,j,i),dlgs(200,j,i),nlgs(2,j,i),dlgs(199,j,i),&
>       & nlgs(3,j,i),dlgs(198,j,i),nlgs(4,j,i),dlgs(197,j,i),nlgs(5:8,j,i),&
>       & dlgs(154:190,j,i)
> 
> Sometimes if you get truncated in just the right place, the program will still
> compile and run, but not the way you think it is.

you can compile it using "-ffree-line-length-none":)

[Bug libfortran/50016] The fortran program 's io is very slow

[jvdelisle at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #12 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> 2011-08-12 16:17:21 UTC ---
I would like to see if Kai has any thought on this.  Unfortunately I am not
very familiar with the windows OS.

[Bug libfortran/50016] [4.6/4.7 Regression] Drastic I/O performance regression on Windows

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

Tobias Burnus <burnus at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |missed-optimization
           Priority|P3                          |P4
                 CC|                            |burnus at gcc dot gnu.org
   Target Milestone|---                         |4.6.2
            Summary|The fortran program 's io   |[4.6/4.7 Regression]
                   |is very slow                |Drastic I/O performance
                   |                            |regression on Windows

--- Comment #13 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-10-14 10:28:20 UTC ---
See also discussion started at
http://gcc.gnu.org/ml/fortran/2011-10/msg00079.html

[Bug libfortran/50016] [4.6/4.7 Regression] Drastic I/O performance regression on Windows

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #14 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-10-18 12:58:46 UTC ---
Author: burnus
Date: Tue Oct 18 12:58:42 2011
New Revision: 180138

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=180138
Log:
2011-10-18  Tobias Burnus  <burnus@net-b.de>
            Janne Blomqvist  <jb@gcc.gnu.org>

        PR fortran/50016
        * io/file_pos.c (st_flush): Call _commit on MinGW(-w64).
        * io/intrinsics.c (flush_i4, flush_i8): Ditto.
        * io/unix.c (flush_all_units_1, flush_all_units): Ditto.
        (buf_flush): Remove _commit call.
        * io/inquire.c (inquire_via_unit): Flush internal buffer
        and call file_length instead of invoking stat via file_size.


Modified:
    branches/gcc-4_6-branch/libgfortran/ChangeLog
    branches/gcc-4_6-branch/libgfortran/io/file_pos.c
    branches/gcc-4_6-branch/libgfortran/io/inquire.c
    branches/gcc-4_6-branch/libgfortran/io/intrinsics.c
    branches/gcc-4_6-branch/libgfortran/io/unix.c

[Bug libfortran/50016] [4.7 Regression] Drastic I/O performance regression on Windows

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

Tobias Burnus <burnus at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2011-10-18
   Target Milestone|4.6.2                       |4.7.0
            Summary|[4.6/4.7 Regression]        |[4.7 Regression] Drastic
                   |Drastic I/O performance     |I/O performance regression
                   |regression on Windows       |on Windows
     Ever Confirmed|0                           |1

--- Comment #15 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-10-18 13:08:22 UTC ---
[4.6] FIXED (comment 14) or at least mitigated the issue on the 4.6 branch - in
time for the 4.6.2 release. See also
http://gcc.gnu.org/ml/fortran/2011-10/msg00132.html

 * * *

For the 4.7 trunk, we are still discussion, when to call _commit.
MinGW/MinGW-w64 users of GCC 4.7 could consider using that patch in the
meanwhile.

Start of the discussion:
  http://gcc.gnu.org/ml/fortran/2011-10/threads.html#00079

Patch version 1: Keep _commit only for FLUSH() subroutine/statement
  http://gcc.gnu.org/ml/fortran/2011-10/msg00132.html

Patch version 2: Remove _commit completely, use internal file-size when
inquiring open files. (The latter is an independent separate issue, but was the
reason for adding _commit at the first place; cf. PR 44698.)
  http://gcc.gnu.org/ml/fortran/2011-10/msg00094.html

 * * *

Xunxun: Thanks for the report and sorry for making the Windows version that
slow and for the two-months delay in fixing it.

[Bug libfortran/50016] [4.7 Regression] Drastic I/O performance regression on Windows

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #16 from xunxun <xunxun1982 at gmail dot com> 2011-10-18 13:20:27 UTC ---
Good job. I will extract the patch to 4.6.1 release.
( I don't use gcc4.6 latest branch temporarily because PR 50664 )
Thanks.

[Bug libfortran/50016] [4.7 Regression] Drastic I/O performance regression on Windows

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #17 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-10-19 17:27:08 UTC ---
Author: burnus
Date: Wed Oct 19 17:27:03 2011
New Revision: 180199

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=180199
Log:
2011-10-19  Janne Blomqvist  <jb@gcc.gnu.org>
            Kai Tietz  <ktietz@redhat.com>
            Tobias Burnus  <burnus@net-b.de>

        PR fortran/50016
        * io/unix.h (flush_sync): Add new libgfortran-internal
        * prototype.
        * io/unix.c (flush_sync): New function, which calls sflush and
        on MinGW(-w64) also _commit.
        (flush_all_units, flush_all_units_1): Replace sflush/_commit by
        flush_sync.
        * io/file_pos.c (st_flush): Replace sflush/_commit by
        * flush_sync.
        * io/intrinsics.c (flush_i4, flush_i8): Ditto.


Modified:
    branches/gcc-4_6-branch/libgfortran/ChangeLog
    branches/gcc-4_6-branch/libgfortran/io/file_pos.c
    branches/gcc-4_6-branch/libgfortran/io/intrinsics.c
    branches/gcc-4_6-branch/libgfortran/io/unix.c
    branches/gcc-4_6-branch/libgfortran/io/unix.h

[Bug libfortran/50016] [4.7 Regression] Drastic I/O performance regression on Windows

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #18 from xunxun <xunxun1982 at gmail dot com> 2011-10-20 06:49:19 UTC ---
(In reply to comment #17)
> Author: burnus
> Date: Wed Oct 19 17:27:03 2011
> New Revision: 180199
> 
> URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=180199
> Log:
> 2011-10-19  Janne Blomqvist  <jb@gcc.gnu.org>
>             Kai Tietz  <ktietz@redhat.com>
>             Tobias Burnus  <burnus@net-b.de>
> 
>         PR fortran/50016
>         * io/unix.h (flush_sync): Add new libgfortran-internal
>         * prototype.
>         * io/unix.c (flush_sync): New function, which calls sflush and
>         on MinGW(-w64) also _commit.
>         (flush_all_units, flush_all_units_1): Replace sflush/_commit by
>         flush_sync.
>         * io/file_pos.c (st_flush): Replace sflush/_commit by
>         * flush_sync.
>         * io/intrinsics.c (flush_i4, flush_i8): Ditto.
> 
> 
> Modified:
>     branches/gcc-4_6-branch/libgfortran/ChangeLog
>     branches/gcc-4_6-branch/libgfortran/io/file_pos.c
>     branches/gcc-4_6-branch/libgfortran/io/intrinsics.c
>     branches/gcc-4_6-branch/libgfortran/io/unix.c
>     branches/gcc-4_6-branch/libgfortran/io/unix.h

Will the rev 180199 merge into gcc4.6.2 release?
Because the first 4.6.2 RC doesn't contain the modify, gcc-4.6.2-RC-20111019
libgfortran build on mingw/mingw64 can cause:

  libtool: compile:  i686-w64-mingw32-gcc
-L/mingw/i686-w64-mingw32/lib -L/mingw/mingw/lib -isystem
/mingw/i686-w64-mingw32/include -isystem /mingw/mingw/include
-DHAVE_CONFIG_H -I. -I../.././libgfortran
-iquote../.././libgfortran/io -I../.././libgfortran/../gcc
-I../.././libgfortran/../gcc/config
-I../.././libgfortran/../libquadmath -I../../host-i686-w64-mingw32/gcc
-D_GNU_SOURCE -std=gnu99 -Wall -Wstrict-prototypes
-Wmissing-prototypes -Wold-style-definition -Wextra -Wwrite-strings
-fcx-fortran-rules -ffunction-sections -fdata-sections -g -pipe -O2
-I/mingw/include -MT file_pos.lo -MD -MP -MF .deps/file_pos.Tpo -c
../.././libgfortran/io/file_pos.c -o file_pos.o
../.././libgfortran/io/file_pos.c: In function 'st_flush':
../.././libgfortran/io/file_pos.c:457:20: error: 'stream' has no
member named 'fd'

I would hope the next 4.6.2 RC or release fix the problem.
Thanks.

[Bug libfortran/50016] [4.7 Regression] Drastic I/O performance regression on Windows

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

--- Comment #19 from Janne Blomqvist <jb at gcc dot gnu.org> 2011-11-09 15:46:20 UTC ---
Author: jb
Date: Wed Nov  9 15:46:15 2011
New Revision: 181207

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=181207
Log:
PR 50016 Slow I/O on MingW due to _commit

frontend ChangeLog:

2011-11-09  Janne Blomqvist  <jb@gcc.gnu.org>

    PR libfortran/50016
    * gfortran.texi (Data consistency and durability): New section.


testsuite ChangeLog:

2011-11-09  Janne Blomqvist  <jb@gcc.gnu.org>

    PR libfortran/50016
    * gfortran.dg/inquire_size.f90: Don't flush the unit.

libgfortran ChangeLog:

2011-11-09  Janne Blomqvist  <jb@gcc.gnu.org>

    PR libfortran/50016
    * io/inquire.c (inquire_via_unit): Flush the unit and use ssize.
    * io/unix.c (buf_flush): Don't call _commit.


Modified:
    trunk/gcc/fortran/ChangeLog
    trunk/gcc/fortran/gfortran.texi
    trunk/gcc/testsuite/ChangeLog
    trunk/gcc/testsuite/gfortran.dg/inquire_size.f90
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/io/inquire.c
    trunk/libgfortran/io/unix.c

[Bug libfortran/50016] [4.7 Regression] Drastic I/O performance regression on Windows

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016

Janne Blomqvist <jb at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #20 from Janne Blomqvist <jb at gcc dot gnu.org> 2011-11-18 11:38:02 UTC ---
Fixed, closing.