[Bug libfortran/16339] New: Unformatted i/o on large arrays inefficient

[Bug libfortran/16339] New: Unformatted i/o on large arrays inefficient

[rth at gcc dot gnu dot org]

Looking at a test case provided by Bud Davis, I see that we're writing out
(and reading back) each element of the array individually, which then gets
blocked and written by the library in page-sized chunks:

write(3, "\0\0\0\0\0\0\0\0\1\0\0\0\2\0\0\0\3\0\0\0\4\0\0\0\5\0\0"..., 8192) = 81
92
write(3, "\377\7\0\0\0\10\0\0\1\10\0\0\2\10\0\0\3\10\0\0\4\10\0\0"..., 8192) = 8
192
...
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x2000001a000
munmap(0x2000001a000, 16384)            = 0
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0x4000) = 0x2000001a000
munmap(0x2000001a000, 16384)            = 0

Surely we should be writing out (and reading back) the entire array (or at
least columns) at a time.

I will also note in passing that mapping and unmapping memory is a relatively
expensive operation.  You only want to do it if you (1) actually require the
sharing semantics or (2) can map enough to make the overhead pay off.  The 16k
sliding window seen here does not meet either requirement.  In the case of these
unformatted reads, the most efficient thing is a read directly into the array
and not fiddling with mmap at all.

-- 
           Summary: Unformatted i/o on large arrays inefficient
           Product: gcc
           Version: 3.5.0
            Status: UNCONFIRMED
          Severity: enhancement
          Priority: P2
         Component: libfortran
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: rth at gcc dot gnu dot org
                CC: gcc-bugs at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16339

[Bug libfortran/16339] Unformatted i/o on large arrays inefficient

[rth at gcc dot gnu dot org]

------- Additional Comments From rth at gcc dot gnu dot org  2004-07-03 01:43 -------
Created an attachment (id=6678)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=6678&action=view)
test case


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16339

[Bug libfortran/16339] Unformatted i/o on large arrays inefficient

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2004-07-03 01:52 -------
Confirmed, it looks like we only nmap in a page at a time.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever Confirmed|                            |1
   Last reconfirmed|0000-00-00 00:00:00         |2004-07-03 01:52:02
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16339

[Bug libfortran/16339] Unformatted i/o on large arrays inefficient

[Thomas dot Koenig at online dot de]

------- Additional Comments From Thomas dot Koenig at online dot de  2005-03-04 10:47 -------
This is really _very_ inefficient, by a factor of 20.

Some test numbers:

$ g77 write-record.f
$ time ./a.out

real    0m1.819s
user    0m1.774s
sys     0m0.044s
$ gfortran write-record.f
$ time ./a.out

real    0m43.723s
user    0m9.003s
sys     0m34.571s
$ cat write-record.f
      program main
      integer n
      parameter (n=10000000)
      real a(n)
      write (10) (a(i),i=1,n)
      end
$ gfortran -v
Using built-in specs.
Target: ia64-unknown-linux-gnu
Configured with: ../gcc-4.1-20050227/configure --prefix=/home/zfkts
--enable-languages=c,f95
Thread model: posix
gcc version 4.1.0 20050227 (experimental)

By comparison:
$ ifort write-record.f
$ time ./a.out

real    0m0.117s
user    0m0.001s
sys     0m0.116s


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16339

[Bug libfortran/16339] Unformatted i/o on large arrays inefficient

[jblomqvi at cc dot hut dot fi]

------- Additional Comments From jblomqvi at cc dot hut dot fi  2005-06-04 17:50 -------
It seems that with current mainline, while gfortran still loses to g77 and
ifort, the difference isn't that large. Executing via strace shows that writes
are nowadays done in 8k blocks, which probably explains the change. Here are the
total times (removing the output file between each invocation):

ifort 8.0: 0,109

g77 3.3: 1,349

gfortran 4.1 20050604: 1,458

Blocksizes in bytes are:

ifort 8.0: 262144

g77 3.3: 4096

gfortran 4.1 20050604: 8192 

Gfortran is the only one which uses mmap. Also, we can see that ifort uses a
_huge_ blocksize, which probably explains why it's over 10 times faster. 

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16339

[Bug libfortran/16339] Unformatted i/o on large arrays inefficient

[jblomqvi at cc dot hut dot fi]

------- Additional Comments From jblomqvi at cc dot hut dot fi  2005-06-05 21:19 -------
Some further thoughts on this issue:

http://gcc.gnu.org/ml/fortran/2005-06/msg00084.html

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16339

[Bug libfortran/16339] Unformatted i/o on large arrays inefficient

[fxcoudert at gcc dot gnu dot org]

------- Comment #8 from fxcoudert at gcc dot gnu dot org  2005-12-09 09:24 -------
*** Bug 21820 has been marked as a duplicate of this bug. ***


-- 

fxcoudert at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kargl at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16339

[Bug libfortran/16339] Unformatted i/o on large arrays inefficient

[fxcoudert at gcc dot gnu dot org]

------- Comment #7 from fxcoudert at gcc dot gnu dot org  2005-11-04 19:50 -------
With array size=20000000, timings are:

g77: 2.30 s
gfortran-4.0 (without commited patch): 17.0
gfortran-mainline: 2.66 s
intel: 3.15 s

I think we can close this PR.


-- 

fxcoudert at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fxcoudert at gcc dot gnu dot
                   |                            |org
             Status|NEW                         |RESOLVED
      Known to fail|                            |4.0.2
      Known to work|                            |4.1.0
         Resolution|                            |FIXED
   Target Milestone|---                         |4.1.0


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16339

[Bug libfortran/16339] Unformatted i/o on large arrays inefficient

[cvs-commit at gcc dot gnu dot org]

------- Comment #6 from cvs-commit at gcc dot gnu dot org  2005-10-07 20:02 -------
Subject: Bug 16339

CVSROOT:        /cvs/gcc
Module name:    gcc
Changes by:     tkoenig@gcc.gnu.org     2005-10-07 20:02:28

Modified files:
        libgfortran    : ChangeLog 
        libgfortran/io : io.h unix.c transfer.c 

Log message:
        2005-10-07  Janne Blomqvist <jblomqvi@cc.hut.fi>

        PR fortran/16339
        PR fortran/23363
        * io/io.h: Add read and write members to stream, define access
        macros.
        * io/transfer.c (read_block_direct): New function.
        (write_block_direct): New function.
        (unformatted_read): Change to use read_block_direct.
        (unformatted_write): Change to use write_block_direct.
        * io/unix.c: Remove mmap includes and defines.
        (writen): Remove.
        (readn): Remove.
        (reset_stream): New function.
        (do_read): New function.
        (do_write): New function.
        (fd_flush): Change to use do_write() instead of writen().
        (fd_alloc_r_at): Change to use do_read().
        (fd_seek): Change return type to try, as the prototype. Add check
        to avoid syscall overhead if possible.
        (fd_read): New function.
        (fd_write): New function.
        (fd_open): Set pointers for new functions.
        (mem_read): New function.
        (mem_write): New function.
        (open_internal): Set pointers for new functions.
        (is_seekable): Clean up comment.

Patches:
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libgfortran/ChangeLog.diff?cvsroot=gcc&r1=1.319&r2=1.320
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libgfortran/io/io.h.diff?cvsroot=gcc&r1=1.32&r2=1.33
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libgfortran/io/unix.c.diff?cvsroot=gcc&r1=1.42&r2=1.43
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libgfortran/io/transfer.c.diff?cvsroot=gcc&r1=1.62&r2=1.63


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16339