[Bug fortran/40773] New: gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[Bug fortran/40773] New: gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[dh458 at oakapple dot net]

On linux x86,  with -m32 or -m64,
gfortran 4.4.0 exhausts over 2GB of swap space compiling the attached source
file
at -O3.    In contrast 4.1.2 compiles unexceptionally.      This might be
expected behavior of 4.4.0 due to some new laborious optimization, which
would terminate eventually with enough RAM/swap, or it might just
be spinning its wheels pointlessly.    

To reproduce from attached files, substantially condensed from SPEC MPI 2007
127.wrf2:

/bin/rm -f *.o *.mod
gfortran -c bigmod.f90
gfortran -c shift_domain_em.f90 -O3
f951: out of memory allocating 239668920 bytes after a total of 3520262144
bytes


Compile times:

        4.1.2       4.4.0
-O2:     
real    1m19.827s   real    2m58.945s
user    1m14.541s   user    2m44.450s
sys     0m1.668s    sys     0m3.140s

-O3:
real    1m30.645s   real   10m5.166s
user    1m25.613s   user   8m39.388s
sys     0m1.492s    sys     0m7.632s





Verbose expansion of failing compile:

/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/local/bin/gfortran
--verbose -c shift_domain_em.f90 -O3
Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with:
/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/gcc-4.4.0/configure
--prefix=/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/local
--exec-prefix=/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/local
--enable-languages=c++,fortran
--with-gmp-include=/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/local/include
--with-mpfr-include=/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/local/include
--with-gmp-lib=/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/local/lib64
--with-mpfr-lib=/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/local/lib64
Thread model: posix
gcc version 4.4.0 (GCC)
COLLECT_GCC_OPTIONS='-v' '-c' '-O3' '-mtune=generic'
/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/local/libexec/gcc/x86_64-unknown-linux-gnu/4.4.0/f951
shift_domain_em.f90 -quiet -dumpbase shift_domain_em.f90 -mtune=generic
-auxbase shift_domain_em -O3 -version -fintrinsic-modules-path
/net/hs-usca-01.sfbay/export/home1/12/dgh/gnu-linux/local/lib/gcc/x86_64-unknown-linux-gnu/4.4.0/finclude
-o /tmp/ccBfbVaD.s
GNU Fortran (GCC) version 4.4.0 (x86_64-unknown-linux-gnu)
        compiled by GNU C version 4.4.0, GMP version 4.3.1, MPFR version 2.4.1.
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072

f951: out of memory allocating 239668920 bytes after a total of 3520262144
bytes



Verbose expansion of correspond successful compile:

/usr/bin/gfortran --verbose -c shift_domain_em.f90 -O3
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-libgcj-multifile
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --enable-plugin
--with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
--host=x86_64-redhat-linux
Thread model: posix
gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)
 /usr/libexec/gcc/x86_64-redhat-linux/4.1.2/f951 shift_domain_em.f90 -quiet
-dumpbase shift_domain_em.f90 -mtune=generic -auxbase shift_domain_em -O3
-version -I /usr/lib/gcc/x86_64-redhat-linux/4.1.2/finclude -o /tmp/cc0MGeZy.s
GNU F95 version 4.1.2 20070626 (Red Hat 4.1.2-14) (x86_64-redhat-linux)
        compiled by GNU C version 4.1.2 20070626 (Red Hat 4.1.2-14).
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
 as -V -Qy -o shift_domain_em.o /tmp/cc0MGeZy.s
GNU assembler version 2.17.50.0.6-5.el5 (x86_64-redhat-linux) using BFD version
2.17.50.0.6-5.el5 20061020


-- 
           Summary: gfortran 4.4.0 -O3 exhausts memory in 127.wrf2
           Product: gcc
           Version: 4.4.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: fortran
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: dh458 at oakapple dot net


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

[Bug fortran/40773] gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[dh458 at oakapple dot net]

------- Comment #1 from dh458 at oakapple dot net  2009-07-16 16:16 -------
Created an attachment (id=18207)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=18207&action=view)
modules definition file, bzip2'd

cat bigmod.f90.bz2 | bunzip2 > bigmod.f90

compressed to avoid bugzilla size limitations


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

[Bug fortran/40773] gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[dh458 at oakapple dot net]

------- Comment #2 from dh458 at oakapple dot net  2009-07-16 16:17 -------
Created an attachment (id=18208)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=18208&action=view)
failing source file that uses modules in bigmod.f90


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

[Bug fortran/40773] gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[kargl at gcc dot gnu dot org]

------- Comment #3 from kargl at gcc dot gnu dot org  2009-07-16 17:22 -------
Does it compile with either -O or -O2?  gfortran 4.4.0 may be trying
to do more inlining, which can consume more memory for temporary
arrays.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

[Bug fortran/40773] gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[dh458 at oakapple dot net]

------- Comment #4 from dh458 at oakapple dot net  2009-07-16 18:24 -------
Compiles OK at -O2, but more slowly than with 4.1.2:

        4.1.2       4.4.0
-O2:     
real    1m19.827s   real    2m58.945s
-O3:
real    1m30.645s   real   10m5.166s

Even if -O3 ultimately generates correct code in some suitably large
configuration,
it may be that some optimizer
algorithms could stand tuning to avoid time-consuming
but ultimately unprofitable blind alleys.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

[Bug fortran/40773] gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[kargl at gcc dot gnu dot org]

------- Comment #5 from kargl at gcc dot gnu dot org  2009-07-16 19:55 -------
(In reply to comment #4)
> Compiles OK at -O2, but more slowly than with 4.1.2:
> 
>         4.1.2       4.4.0
> -O2:     
> real    1m19.827s   real    2m58.945s
> -O3:
> real    1m30.645s   real   10m5.166s
> 
> Even if -O3 ultimately generates correct code in some suitably large
> configuration,
> it may be that some optimizer
> algorithms could stand tuning to avoid time-consuming
> but ultimately unprofitable blind alleys.
> 

Hmmm, something is definitely admiss here.  After 22 minutes of
compiling with -O3 on my system, f951 'appears' stuck with only
450 MB of used memory.  I attached ktrace to the process and 
found 

 50649 f951     CALL 
mmap(0,0x300000,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,0xffffffff,0)
 50649 f951     RET   mmap 500170752/0x21dd00000
 50649 f951     CALL  madvise(0x22b179000,0x83000,MADV_FREE)
 50649 f951     RET   madvise 0
 50649 f951     CALL  madvise(0x22b002000,0x79000,MADV_FREE)
 50649 f951     RET   madvise 0
 50649 f951     CALL  madvise(0x21e0f8000,0x8000,MADV_FREE)
 50649 f951     RET   madvise 0
 50649 f951     CALL  madvise(0x21e002000,0xde000,MADV_FREE)
 50649 f951     RET   madvise 0
 50649 f951     CALL  munmap(0x21dd00000,0x300000)
 50649 f951     RET   munmap 0

repeated over and over and over again.  It appears on my
system the f951 is looping over some garbage collection, but
never reaching a termination point. :(


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

[Bug fortran/40773] gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[kargl at gcc dot gnu dot org]

------- Comment #6 from kargl at gcc dot gnu dot org  2009-07-16 20:29 -------
(In reply to comment #5)
> (In reply to comment #4)
> > Compiles OK at -O2, but more slowly than with 4.1.2:
> > 
> >         4.1.2       4.4.0
> > -O2:     
> > real    1m19.827s   real    2m58.945s
> > -O3:
> > real    1m30.645s   real   10m5.166s
> > 
> > Even if -O3 ultimately generates correct code in some suitably large
> > configuration,
> > it may be that some optimizer
> > algorithms could stand tuning to avoid time-consuming
> > but ultimately unprofitable blind alleys.
> > 
> 
> Hmmm, something is definitely admiss here.  After 22 minutes of
> compiling with -O3 on my system, f951 'appears' stuck with only
> 450 MB of used memory.  I attached ktrace to the process and 
> found 

After 26 minutes, I suddenly see in top(1)

 50879 sgk           1 -20    0  7346M  6724M swread  1  26:21  1.51% f951

so -O3 is causing a massive increase in the amount of memory required.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

[Bug fortran/40773] gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[kargl at gcc dot gnu dot org]

------- Comment #7 from kargl at gcc dot gnu dot org  2009-07-16 22:47 -------
Further investigation suggests that -O3 is unusable with Fortran code
when the size of the code in a single file or module becomes very large.
Specifically, gcc.info states:

  '-O3' turns on all optimizations specified by
     '-O2' and also turns on the '-finline-functions',
     '-funswitch-loops', '-fpredictive-commoning',
     '-fgcse-after-reload' and '-ftree-vectorize' options.

On my system with OPT="-O2 -pipe -march=native", I get

% time gfc4x -c $OPT shift_domain_em.f90
      292.91 real       270.35 user        20.92 sys
% time gfc4x -c $OPT -finline-functions shift_domain_em.f90
      297.52 real       275.26 user        21.39 sys
% time gfc4x -c $OPT -fpredictive-commoning shift_domain_em.f90
      294.75 real       271.22 user        21.21 sys
% time gfc4x -c $OPT -fgcse-after-reload shift_domain_em.f90
      295.13 real       270.59 user        21.51 sys
% time gfc4x -c $OPT -funswitch-loops shift_domain_em.f90
      630.73 real       601.11 user        24.99 sys
% time gfc4x -c $OPT -ftree-vectorize shift_domain_em.f90
      880.61 real       610.32 user        56.16 sys

It is also noteworthy that the first 5 command lines require
about 2 GB of memory to compile the code.  The last 
command line required more than 6 GB of memory, and
on my system this leads to swapping.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

[Bug tree-optimization/40773] gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[kargl at gcc dot gnu dot org]

------- Comment #8 from kargl at gcc dot gnu dot org  2009-07-17 00:28 -------
This appears to be more than a fortran issue.  I've changed
the component to tree-optimization out lack of a better choice.


-- 

kargl at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|fortran                     |tree-optimization


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

[Bug tree-optimization/40773] gfortran 4.4.0 -O3 exhausts memory in 127.wrf2

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40773

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED

--- Comment #9 from Richard Biener <rguenth at gcc dot gnu.org> ---
We do track WRF in SPEC 2006 and 2017 and fixed quite some memory/compile-time
hog issues with it.  Let's close this report about an ancient version and
assume the issue was fixed.