[Bug c++/53226] New: memory consumption for heavy template instantiations increased massively

[Bug c++/53226] New: memory consumption for heavy template instantiations increased massively

[mario-baumann at web dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

             Bug #: 53226
           Summary: memory consumption for heavy template instantiations
                    increased massively
    Classification: Unclassified
           Product: gcc
           Version: 4.8.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: mario-baumann@web.de


Hi All,

i noticed that memory consumption for heavy template instantiations increased
massively between gcc revision 187034 and 187074. If a compile my current
project (unfortunately closed source) i got e.g.

> top
top - 08:58:08 up 1 day, 30 min, 11 users,  load average: 28.83, 19.13, 13.36
Mem:  24607424k total, 24464568k used,   142856k free,      752k buffers
Swap: 52436120k total,  6650108k used, 45786012k free,  4174408k cached
  PID PR  NI  VIRT  RES  SHR S %CPU %MEM   TIME+   COMMAND
14825 20   0 24.5g  18g  496 R 89.1 79.0   3:48.75 cc1plus

Mario.

---

> uname -a
Linux ahsoka.intec.dom 2.6.32-220.13.1.el6.x86_64 #1 SMP Thu Mar 29 11:46:40
EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

> rpm -qa "glibc*" | grep -e 'glibc-[0-9]' | sort -u
glibc-2.12-1.47.el6_2.9.i686
glibc-2.12-1.47.el6_2.9.x86_64

> g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/app2/gcc/4.8.0-20120503-svn187074/x86_64/libexec/gcc/x86_64-unknown-linux-gnu/4.8.0/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ./configure --prefix=/app2/gcc/4.8.0-20120503-svn187074/x86_64
--enable-languages=c,c++,fortran --disable-nls
--with-gmp=/app2/gcc/4.8.0-20120503-svn187074/x86_64/aux
--with-mpfr=/app2/gcc/4.8.0-20120503-svn187074/x86_64/aux
--with-mpc=/app2/gcc/4.8.0-20120503-svn187074/x86_64/aux
--with-ppl=/app2/gcc/4.8.0-20120503-svn187074/x86_64/aux
--with-cloog=/app2/gcc/4.8.0-20120503-svn187074/x86_64/aux
Thread model: posix
gcc version 4.8.0 20120503 (experimental) (GCC) 

> ld -v
GNU ld (GNU Binutils) 2.22.52.20120503

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[manu at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

Manuel López-Ibáñez <manu at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2012-05-04
                 CC|                            |manu at gcc dot gnu.org
     Ever Confirmed|0                           |1

--- Comment #1 from Manuel López-Ibáñez <manu at gcc dot gnu.org> 2012-05-04 07:22:33 UTC ---
http://gcc.gnu.org/bugs/minimize.html

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #2 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-04 09:01:32 UTC ---
Can you bi-sect to an exact revision causing the regression?

What compile options do you use?

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[mario-baumann at web dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #3 from Mario Baumann <mario-baumann at web dot de> 2012-05-04 09:33:07 UTC ---
Hi Richard,

compile options are: -m32 -msse -msse2 -mfpmath=sse -fPIC -pipe -Wall -Wunused
-Werror -O2 -g

bi-sect: in progress ...

Mario.

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[mario-baumann at web dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #4 from Mario Baumann <mario-baumann at web dot de> 2012-05-04 19:51:24 UTC ---
Correction: -m32 works fine.

1st bi-sect: gcc revision 187054 shows the same problem if compiling with -m64.

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[mario-baumann at web dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #5 from Mario Baumann <mario-baumann at web dot de> 2012-05-08 06:48:38 UTC ---
I've finished the bi-section -> revision 187053 caused the problem.

I'm trying to produce a testcase ...

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[mario-baumann at web dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #6 from Mario Baumann <mario-baumann at web dot de> 2012-05-08 07:54:22 UTC ---
Created attachment 27341
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=27341
c++ source

Added testcase (needs include files from boost ->
http://sourceforge.net/projects/boost/files/boost/1.49.0/ -> just extract any
tarball).

> /app2/gcc/4.8.0-20120507-svn187053/x86_64/bin/g++ -v
Using built-in specs.
COLLECT_GCC=/app2/gcc/4.8.0-20120507-svn187053/x86_64/bin/g++
COLLECT_LTO_WRAPPER=/app2/gcc/4.8.0-20120507-svn187053/x86_64/libexec/gcc/x86_64-unknown-linux-gnu/4.8.0/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ./configure --prefix=/app2/gcc/4.8.0-20120507-svn187053/x86_64
--enable-languages=c,c++,fortran --disable-nls
--with-gmp=/app2/gcc/4.8.0-20120507-svn187053/x86_64/aux
--with-mpfr=/app2/gcc/4.8.0-20120507-svn187053/x86_64/aux
--with-mpc=/app2/gcc/4.8.0-20120507-svn187053/x86_64/aux
--with-ppl=/app2/gcc/4.8.0-20120507-svn187053/x86_64/aux
--with-cloog=/app2/gcc/4.8.0-20120507-svn187053/x86_64/aux
Thread model: posix
gcc version 4.8.0 20120502 (experimental) (GCC) 

bad: g++ -Iboost_1_49_0/ -c foo.cpp -O2 -m64
bad: g++ -Iboost_1_49_0/ -c foo.cpp -O1 -m64
 ok: g++ -Iboost_1_49_0/ -c foo.cpp -O0 -m64
 ok: g++ -Iboost_1_49_0/ -c foo.cpp -O2 -m32

Mario.

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

Richard Guenther <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |UNCONFIRMED
                 CC|                            |matz at gcc dot gnu.org
     Ever Confirmed|1                           |0

--- Comment #7 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-08 10:09:49 UTC ---
Oh well.  Micha - can you investigate?  Maybe we fail to properly garbage
collect some stuff now?

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[matz at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

Michael Matz <matz at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
   Last reconfirmed|2012-05-04 00:00:00         |2012-05-08
         AssignedTo|unassigned at gcc dot       |matz at gcc dot gnu.org
                   |gnu.org                     |
     Ever Confirmed|0                           |1

--- Comment #8 from Michael Matz <matz at gcc dot gnu.org> 2012-05-08 10:46:38 UTC ---
Yep, mine.

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jakub at gcc dot gnu.org

--- Comment #9 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-05-08 12:03:55 UTC ---
Ugh, remove_prop_source_from_use can remove stmts while prev iterator points to
them.  This has been buggy before, but didn't result in endless loop.
I guess instead of using prev/prev_initialized, the loop could gimple_set_uid
(stmt, 0) the stmts it is about to process and gimple_set_uid (stmt, 1) stmts
that don't need to be processed again, then gsi = prev; could be just replaced
by doing gsi_prev (&gsi); enough times to reach start of bb or a stmt with uid
1.

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #10 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-08 12:09:05 UTC ---
(In reply to comment #9)
> Ugh, remove_prop_source_from_use can remove stmts while prev iterator points to
> them.  This has been buggy before, but didn't result in endless loop.
> I guess instead of using prev/prev_initialized, the loop could gimple_set_uid
> (stmt, 0) the stmts it is about to process and gimple_set_uid (stmt, 1) stmts
> that don't need to be processed again, then gsi = prev; could be just replaced
> by doing gsi_prev (&gsi); enough times to reach start of bb or a stmt with uid
> 1.

The other way is to try to get away without immediately removing propagated
source stmts - the obvious downside is that transforms relying on
single-use definitions will be confused by those dead uses (but the scheme
how it works now is fragile anyway because we are not sure we visit the stmts
in the correct order).

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[matz at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #11 from Michael Matz <matz at gcc dot gnu.org> 2012-05-08 12:12:20 UTC ---
(In reply to comment #9)
> I guess instead of using prev/prev_initialized, the loop could gimple_set_uid
> (stmt, 0) the stmts it is about to process and gimple_set_uid (stmt, 1) stmts
> that don't need to be processed again, then gsi = prev; could be just replaced
> by doing gsi_prev (&gsi); enough times to reach start of bb or a stmt with uid
> 1.

Yeah, something along these lines.  If you get to that, please do,
I can't care for this today.

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #12 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-08 12:12:42 UTC ---
One way would be to have a queue of propagation sources to be removed and
delay the removal until after we have processed the function (the basic
block?).
Then have a worklist of stmts to re-visit which we initialize from the uses
of the stmts we remove.

Or something like that ... or simply do not care (though we have a few
testcases that depend on this I think)

[Bug c++/53226] memory consumption for heavy template instantiations increased massively

[matz at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #13 from Michael Matz <matz at gcc dot gnu.org> 2012-05-08 13:19:42 UTC ---
(In reply to comment #10)
> The other way is to try to get away without immediately removing propagated
> source stmts - the obvious downside is that transforms relying on
> single-use definitions will be confused by those dead uses

Just for completeness: If we would simply NOP out the to-be-removed statements
that would be taken care of.  Additionally iterator copies to those
non-removed stmts wouldn't break.

[Bug tree-optimization/53226] [4.8 Regression] Endless loop in forwprop

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|c++                         |tree-optimization
   Target Milestone|---                         |4.8.0
            Summary|memory consumption for      |[4.8 Regression] Endless
                   |heavy template              |loop in forwprop
                   |instantiations increased    |
                   |massively                   |

--- Comment #14 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-05-08 16:20:50 UTC ---
Reduced testcase, not template heavy...:

/* PR tree-optimization/53226 */

void
foo (unsigned long *x, char y, char z)
{
  int i;
  for (i = y; i < z; ++i)
    {
      unsigned long a = ((unsigned char) i) & 63UL;
      unsigned long b = 1ULL << a;
      *x |= b;
    }
}

[Bug tree-optimization/53226] [4.8 Regression] Endless loop in forwprop

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

--- Comment #15 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-05-09 12:19:41 UTC ---
Author: jakub
Date: Wed May  9 12:19:34 2012
New Revision: 187328

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=187328
Log:
    PR tree-optimization/53226
    * tree-ssa-forwprop.c (ssa_forward_propagate_and_combine): Remove
    prev and prev_initialized vars, gimple_set_plf (stmt, GF_PLF_1, false)
    before processing it and gimple_set_plf (stmt, GF_PLF_1, true) if it
    doesn't need to be revisited, look for earliest stmt with
    !gimple_plf (stmt, GF_PLF_1) if something changed.

    * gcc.c-torture/compile/pr53226.c: New test.

Added:
    trunk/gcc/testsuite/gcc.c-torture/compile/pr53226.c
Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/testsuite/ChangeLog
    trunk/gcc/tree-ssa-forwprop.c

[Bug tree-optimization/53226] [4.8 Regression] Endless loop in forwprop

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53226

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #16 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-05-09 12:31:47 UTC ---
Fixed.