[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[rguenth at gcc dot gnu dot org]

------- Comment #24 from rguenth at gcc dot gnu dot org  2005-10-03 17:54 -------
It somehow works (partially), but there's a lot of fallout.  Ugh.  I don't like
it very much.  Preliminary patch:

http://gcc.gnu.org/ml/gcc-patches/2005-10/msg00091.html


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[rth at gcc dot gnu dot org]

------- Comment #25 from rth at gcc dot gnu dot org  2005-10-11 19:24 -------
I don't think we can reasonably attack this for 4.1.  This is something
that should be done during a stage 1.


-- 

rth at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.1.0                       |---


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[ian at airs dot com]

------- Comment #26 from ian at airs dot com  2005-10-11 19:30 -------
Regression bugs should have target milestones.


-- 

ian at airs dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |4.2.0


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[niemayer at isg dot de]

------- Comment #27 from niemayer at isg dot de  2006-12-13 11:37 -------
I would like to mention that this problem seems to have worsened a lot for the
current snapshots of gcc-4.2 (currently testing with 4.2.0 20061205
(prerelease)) when compiling with at least -O1 - maybe due to the static
constant elimination?.

I tried to compile a Unicode normalization test C++ source that took gcc about
300MB of RAM before to compile with -O1 - now with gcc 4.2 I cannot compile
this source anymore on a machine with 1 GB of physical + 1 GB of virtual RAM
before the kernel OOM killer is killing cc1plus.

If somebody would like the source of my test-case, I can supply it.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[mmitchel at gcc dot gnu dot org]

------- Comment #28 from mmitchel at gcc dot gnu dot org  2007-05-14 22:25 -------
Will not be fixed in 4.2.0; retargeting at 4.2.1.


-- 

mmitchel at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.2.0                       |4.2.1


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[niemayer at isg dot de]

------- Comment #29 from niemayer at isg dot de  2007-05-15 16:54 -------
That's sad - while memory gets cheaper, it has still not become cheap enough to
cope with that huge increase in memory usage imposed by gcc 4.2. Seems I have
to stick with 4.1 until that problem is fixed...


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[pluto at agmk dot net]

------- Comment #30 from pluto at agmk dot net  2007-05-15 17:04 -------
looks like related to PR30052.


-- 

pluto at agmk dot net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pluto at agmk dot net


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[mmitchel at gcc dot gnu dot org]

-- 

mmitchel at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.2.1                       |4.2.2


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[mmitchel at gcc dot gnu dot org]

------- Comment #31 from mmitchel at gcc dot gnu dot org  2007-10-09 19:20 -------
Change target milestone to 4.2.3, as 4.2.2 has been released.


-- 

mmitchel at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.2.2                       |4.2.3


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[rguenth at gcc dot gnu dot org]

------- Comment #32 from rguenth at gcc dot gnu dot org  2008-01-05 14:07 -------
The difference between using gcc and g++ for the testcase seems to be gone on
the trunk, where gcc peaks at 480MB and g++ at 530MB.  For 4.1 g++ used 780MB.


-- 

rguenth at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
      Known to fail|3.4.3 4.0.0                 |3.3.3 3.4.3 4.0.0 4.3.0
   Last reconfirmed|2006-09-03 21:41:14         |2008-01-05 14:07:50
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[jsm28 at gcc dot gnu dot org]

------- Comment #33 from jsm28 at gcc dot gnu dot org  2008-01-17 15:42 -------
This memory use regression has been present since at least 3.3; at least part
of it may be an unavoidable consequence of supporting C99 overriding in
designated initializers; a proper fix would likely involve major changes to the
datastructures for initializers (as RTH notes in comment#25, it's not suitable
for a stage 3 fix); the priority seems to have been P2 from the start rather
than having been set by an RM.  In view of these (but especially the likely
unsuitability of a fix for stage 3), downgrading to P4 (the same as the
corresponding C++ bug, bug 14179).


-- 

jsm28 at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P2                          |P4


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[niemayer at isg dot de]

------- Comment #34 from niemayer at isg dot de  2008-01-17 17:02 -------
Can you suggest any kind of work-around? Any alternative to represent constant
arrays in C/C++?

The problem with leaving this bug open indefinitely is that there are existing
programs (as the Unicode-test-case I mentioned above) which will simply not
compile on any reasonably equipped machine anymore.

I wouldn't mind to change the source code to represent the constant arrays in a
different way, but I have not found a method yet (other than using platform
dependend methods like generating assembler source).


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[ian at airs dot com]

------- Comment #35 from ian at airs dot com  2008-01-18 06:37 -------
The bug should certainly be fixed.  But it's unfortunately a lot of work for a
small payoff--most people are not in your situation.  I think Joseph is correct
in lowering the priority.  It's pointless for us to describe this bug as
release-blocking, when it clearly is not.

The core problem is C99 designated initializers.  Those require us to read the
entire array into memory before we emit any of it.  Otherwise we could generate
the wrong code, and there is no way to recover.

So the only plausible fix is to optimize the memory representation used for
large array initializers.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.0/4.1/4.2/4.3 regression] Uses lots of memory when compiling large initialized arrays

[jsm28 at gcc dot gnu dot org]

------- Comment #36 from jsm28 at gcc dot gnu dot org  2008-02-01 16:52 -------
4.2.3 is being released now, changing milestones of open bugs to 4.2.4.


-- 

jsm28 at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.2.3                       |4.2.4


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.1/4.2/4.3/4.4 regression] Uses lots of memory when compiling large initialized arrays

[jsm28 at gcc dot gnu dot org]

------- Comment #37 from jsm28 at gcc dot gnu dot org  2008-05-19 20:22 -------
4.2.4 is being released, changing milestones to 4.2.5.


-- 

jsm28 at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.2.4                       |4.2.5


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.2/4.3/4.4 regression] Uses lots of memory when compiling large initialized arrays

[jsm28 at gcc dot gnu dot org]

------- Comment #38 from jsm28 at gcc dot gnu dot org  2008-07-04 22:44 -------
Closing 4.1 branch.


-- 

jsm28 at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[4.1/4.2/4.3/4.4 regression]|[4.2/4.3/4.4 regression]
                   |Uses lots of memory when    |Uses lots of memory when
                   |compiling large initialized |compiling large initialized
                   |arrays                      |arrays


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.2/4.3/4.4 regression] Uses lots of memory when compiling large initialized arrays

[rguenth at gcc dot gnu dot org]

------- Comment #39 from rguenth at gcc dot gnu dot org  2009-02-10 10:12 -------
*** Bug 39142 has been marked as a duplicate of this bug. ***


-- 

rguenth at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |macdonellba+gcc at gmail dot
                   |                            |com


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.2/4.3/4.4 regression] Uses lots of memory when compiling large initialized arrays

[hubicka at gcc dot gnu dot org]

------- Comment #40 from hubicka at gcc dot gnu dot org  2009-02-21 12:40 -------
I happen to have compiler with statistics around:
We still need about 400MB, mostly integer constants:
c-decl.c:473 (bind)                                  125040: 0.0%          0:
0.0%          0: 0.0%          0: 0.0%       2605
tree.c:5905 (build_function_type)                     13000: 0.0%          0:
0.0%     113400: 0.1%       5056: 0.0%        632
stringpool.c:73 (alloc_node)                           6032: 0.0%          0:
0.0%     174096: 0.1%      13856: 0.0%       1732
langhooks.c:543 (add_builtin_function_common)             0: 0.0%          0:
0.0%     442224: 0.2%      59760: 0.2%       1494
c-typeck.c:6472 (output_init_element)                     0: 0.0%  
47910400:100.0%   45541112:23.7%   26342936:66.6%         19
convert.c:752 (convert_to_integer)                117415728:44.6%          0:
0.0%          0: 0.0%   13046192:33.0%    1630774
ggc-common.c:187 (ggc_calloc)                      67094608:25.5%          0:
0.0%   67162736:34.9%       1088: 0.0%         58
tree.c:1004 (build_int_cst_wide)                   78264768:29.8%          0:
0.0%   78266496:40.7%          0: 0.0%    3261068
Total                                             262986355         47910416   
    192171521         39527780          4905807
source location                                     Garbage            Freed   
         Leak         Overhead            Times


It seems that we produce awful amount of garbage during the initializer
construction.  Perhaps by forcing ggc_collect there we can get down to 200MB
that we need to reprezent it at the end?

Honza


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.2/4.3/4.4 regression] Uses lots of memory when compiling large initialized arrays

[rguenther at suse dot de]

------- Comment #41 from rguenther at suse dot de  2009-02-21 12:50 -------
Subject: Re:  [4.2/4.3/4.4 regression] Uses lots of memory when
 compiling large initialized arrays

On Sat, 21 Feb 2009, hubicka at gcc dot gnu dot org wrote:

> ------- Comment #40 from hubicka at gcc dot gnu dot org  2009-02-21 12:40 -------
> I happen to have compiler with statistics around:
> We still need about 400MB, mostly integer constants:
> c-decl.c:473 (bind)                                  125040: 0.0%          0:
> 0.0%          0: 0.0%          0: 0.0%       2605
> tree.c:5905 (build_function_type)                     13000: 0.0%          0:
> 0.0%     113400: 0.1%       5056: 0.0%        632
> stringpool.c:73 (alloc_node)                           6032: 0.0%          0:
> 0.0%     174096: 0.1%      13856: 0.0%       1732
> langhooks.c:543 (add_builtin_function_common)             0: 0.0%          0:
> 0.0%     442224: 0.2%      59760: 0.2%       1494
> c-typeck.c:6472 (output_init_element)                     0: 0.0%  
> 47910400:100.0%   45541112:23.7%   26342936:66.6%         19
> convert.c:752 (convert_to_integer)                117415728:44.6%          0:
> 0.0%          0: 0.0%   13046192:33.0%    1630774
> ggc-common.c:187 (ggc_calloc)                      67094608:25.5%          0:
> 0.0%   67162736:34.9%       1088: 0.0%         58
> tree.c:1004 (build_int_cst_wide)                   78264768:29.8%          0:
> 0.0%   78266496:40.7%          0: 0.0%    3261068
> Total                                             262986355         47910416   
>     192171521         39527780          4905807
> source location                                     Garbage            Freed   
>          Leak         Overhead            Times
> 
> 
> It seems that we produce awful amount of garbage during the initializer
> construction.  Perhaps by forcing ggc_collect there we can get down to 200MB
> that we need to reprezent it at the end?

We need the integer csts in the constructor lists.  I have a patch
somewhere (or is it even attached?) that tries to do index compression
and not use the integer csts for counting.  Didn't work out too much
though.

Richard.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.2/4.3/4.4 regression] Uses lots of memory when compiling large initialized arrays

[hubicka at gcc dot gnu dot org]

------- Comment #42 from hubicka at gcc dot gnu dot org  2009-02-22 11:21 -------
Actual representation of constructor don't seem to be major problem here.

We seem to build _a lot_ (117MB) of CONVERT exprs just to call fold on it and
convert integer to proper type, so counting in INTEGER_CSTs should be just
slightly less than half of memory needed.  This seems quite silly.

The patch to not use HOST_WIDE_INT or similar for counting should save another
70MB of garbage (and speed up compilation), so perhaps you could dig it out?
:))

Following patch:
Index: convert.c
===================================================================
--- convert.c   (revision 144352)
+++ convert.c   (working copy)
@@ -749,6 +749,11 @@ convert_to_integer (tree type, tree expr
          break;
        }

+      /* When parsing long initializers, we might end up with a lot of casts.
+         Shortcut this.  */
+      if (TREE_CODE (expr) == INTEGER_CST)
+       return fold_unary (CONVERT_EXPR, type, expr);
+
       return build1 (CONVERT_EXPR, type, expr);

     case REAL_TYPE:

Cuts gabrage production in half:
c-typeck.c:6472 (output_init_element)                     0: 0.0%  
47910400:100.0%   45541112:23.7%   26342936:99.5%         19
ggc-common.c:187 (ggc_calloc)                      67094608:46.1%          0:
0.0%   67162736:34.9%       1088: 0.0%         58
tree.c:1004 (build_int_cst_wide)                   78264768:53.8%          0:
0.0%   78266496:40.7%          0: 0.0%    3261068
Total                                             145570627         47910416   
    192171521         26481588          3275033
source location                                     Garbage            Freed   
         Leak         Overhead            Times

I will give the patch testing, but I am not too hopeful it will just work. ;)

Honza


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.2/4.3/4.4 regression] Uses lots of memory when compiling large initialized arrays

[rguenther at suse dot de]

------- Comment #43 from rguenther at suse dot de  2009-02-22 19:03 -------
Subject: Re:  [4.2/4.3/4.4 regression] Uses lots of memory when
 compiling large initialized arrays

On Sun, 22 Feb 2009, hubicka at gcc dot gnu dot org wrote:

> Actual representation of constructor don't seem to be major problem here.
> 
> We seem to build _a lot_ (117MB) of CONVERT exprs just to call fold on it and
> convert integer to proper type, so counting in INTEGER_CSTs should be just
> slightly less than half of memory needed.  This seems quite silly.
> 
> The patch to not use HOST_WIDE_INT or similar for counting should save another
> 70MB of garbage (and speed up compilation), so perhaps you could dig it out?
> :))
> 
> Following patch:
> Index: convert.c
> ===================================================================
> --- convert.c   (revision 144352)
> +++ convert.c   (working copy)
> @@ -749,6 +749,11 @@ convert_to_integer (tree type, tree expr
>           break;
>         }
> 
> +      /* When parsing long initializers, we might end up with a lot of casts.
> +         Shortcut this.  */
> +      if (TREE_CODE (expr) == INTEGER_CST)
> +       return fold_unary (CONVERT_EXPR, type, expr);

fold_convert ().  But maybe not valid to do here for C std reasons, who 
knows.

> +
>        return build1 (CONVERT_EXPR, type, expr);

And probably just generally using fold_convert () would be ok as well.
Maybe they are there to make sure to build rvalues.

>      case REAL_TYPE:
> 
> Cuts gabrage production in half:
> c-typeck.c:6472 (output_init_element)                     0: 0.0%  
> 47910400:100.0%   45541112:23.7%   26342936:99.5%         19
> ggc-common.c:187 (ggc_calloc)                      67094608:46.1%          0:
> 0.0%   67162736:34.9%       1088: 0.0%         58
> tree.c:1004 (build_int_cst_wide)                   78264768:53.8%          0:
> 0.0%   78266496:40.7%          0: 0.0%    3261068
> Total                                             145570627         47910416   
>     192171521         26481588          3275033
> source location                                     Garbage            Freed   
>          Leak         Overhead            Times
> 


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.2/4.3/4.4 regression] Uses lots of memory when compiling large initialized arrays

[hubicka at gcc dot gnu dot org]

------- Comment #44 from hubicka at gcc dot gnu dot org  2009-02-23 13:41 -------
Hi,
I believe that using fold_convert instead of fold_build1 means that we would
bypass folding done in fold_unary that handles stuff like two conversions in a
row while fold_convert is primarily about returning constant when result is
constant.

Since I want to avoid wrapping fold calls all frontends except for C++
consistently put around convert_to_* calls, I want to do this kind of folding.

I believe only reason to avoid folding is C++ template stuff.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.2/4.3/4.4 regression] Uses lots of memory when compiling large initialized arrays

[hubicka at gcc dot gnu dot org]

------- Comment #45 from hubicka at gcc dot gnu dot org  2009-02-23 16:46 -------
Subject: Bug 12245

Author: hubicka
Date: Mon Feb 23 16:46:32 2009
New Revision: 144384

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=144384
Log:
        PR c/12245
        * ggc.h (htab_create_ggc): Use ggc_free to free hashtable when
resizing.

Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/ggc.h


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.3/4.4/4.5 regression] Uses lots of memory when compiling large initialized arrays

[jsm28 at gcc dot gnu dot org]

------- Comment #46 from jsm28 at gcc dot gnu dot org  2009-03-31 16:13 -------
Closing 4.2 branch.


-- 

jsm28 at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[4.2/4.3/4.4/4.5 regression]|[4.3/4.4/4.5 regression]
                   |Uses lots of memory when    |Uses lots of memory when
                   |compiling large initialized |compiling large initialized
                   |arrays                      |arrays
   Target Milestone|4.2.5                       |4.3.4


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.3/4.4/4.5 regression] Uses lots of memory when compiling large initialized arrays

[rguenth at gcc dot gnu dot org]

------- Comment #47 from rguenth at gcc dot gnu dot org  2009-08-04 12:25 -------
GCC 4.3.4 is being released, adjusting target milestone.


-- 

rguenth at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.3.4                       |4.3.5


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [4.3/4.4/4.5/4.6 regression] Uses lots of memory when compiling large initialized arrays

[rguenth at gcc dot gnu dot org]

------- Comment #48 from rguenth at gcc dot gnu dot org  2010-05-22 18:09 -------
GCC 4.3.5 is being released, adjusting target milestone.


-- 

rguenth at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.3.5                       |4.3.6


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245