[Bug c/12245] New: Crashes when compiling large initialized arrays

[Bug c/12245] New: Crashes when compiling large initialized arrays

[gccBug dot 9 dot OkianWarrior at SpamGourmet dot com]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

           Summary: Crashes when compiling large initialized arrays
           Product: gcc
           Version: 3.3.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: c
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: gccBug dot 9 dot OkianWarrior at SpamGourmet dot com
                CC: gcc-bugs at gcc dot gnu dot org
 GCC build triplet: 3.3.1
  GCC host triplet: Linux 2.4.19-16mdk (Mandrake)
GCC target triplet: Linux 2.4.19-16mdk (Mandrake)

This is a simple program with large initialized static arrays (20 arrays of
200,000 elements each, plus a single array with 500,000 elements.

Compiling crashes the compiler after about an hour (900 MHZ system).
Reproduce by typing "Make" (Makefile included).
Actual results are listed below.
Compiling with -Wall shows no warnings.

I really hate to do this to you, but I believe that the problem is related to
initializing very large arrays. Consequently, I cannot generate a small program
which illustrates the problem (and it takes an hour for each experiment, which
doesn't help either). My minimal solution is a C source file of 152 lines
(inconsequential), and two include files of 200,000 lines and 500,000 lines
each. The include files contain initialization data for 21 arrays, and are
nothing more than a list of numbers and commas.

The total source and intermediate files are so BIG that I have not included them
here. I know I'm not supposed to send archives and I'm *really* not supposed to
ask you to download the test cases from the net, but in this instance I think
it's appropriate.

You can get the complete test set (1 source, 2 includes, Makefile, and saved
intermediate file) at www.OkianWarrior.com/gccBug.tar.gz

/home/kibaro/tmp: make
gcc -v -save-temps -o CSolv CSolv.c
Reading specs from /usr/local/lib/gcc-lib/i686-pc-linux-gnu/3.3.1/specs
Configured with: ./configure
Thread model: posix
gcc version 3.3.1
 /usr/local/lib/gcc-lib/i686-pc-linux-gnu/3.3.1/cc1 -E -quiet -v -D__GNUC__=3
-D__GNUC_MINOR__=3 -D__GNUC_PATCHLEVEL__=1 CSolv.c CSolv.i
ignoring nonexistent directory "NONE/include"
ignoring nonexistent directory "/usr/local/i686-pc-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /usr/local/lib/gcc-lib/i686-pc-linux-gnu/3.3.1/include
 /usr/include
End of search list.
 /usr/local/lib/gcc-lib/i686-pc-linux-gnu/3.3.1/cc1 -fpreprocessed CSolv.i
-quiet -dumpbase CSolv.c -auxbase CSolv -version -o CSolv.s
GNU C version 3.3.1 (i686-pc-linux-gnu)
        compiled by GNU C version 3.2 (Mandrake Linux 9.0 3.2-1mdk).
GGC heuristics: --param ggc-min-expand=47 --param ggc-min-heapsize=32119
gcc: Internal error: Killed (program cc1)
Please submit a full bug report.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
make: *** [CSolv] Error 1

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245


pinskia at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING


------- Additional Comments From pinskia at gcc dot gnu dot org  2003-09-11 15:53 -------
Please attach the preprocessed files to this bug.

[Bug c/12245] Crashes when compiling large initialized arrays

[barrabas at barrabas dot mv dot com]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245



------- Additional Comments From barrabas at barrabas dot mv dot com  2003-09-11 19:26 -------
Subject: Re:  Crashes when compiling large initialized arrays (gccBug: message 3 of 9)

> Please attach the preprocessed files to this bug.

	The file is too big to send to Bugzilla. As mentioned in the bug posting, the 
file is available for http download here:

www.OkianWarrior.com/gccBug.tar.gz

	I've read the guidelines for posting bugs. I know I'm not supposed to post 
archives, or links to archives, or multiple file examples. I know all this. I 
believe that this is an exception for reasons stated in the bug description, 
and I ask that you bear with me.

																R. Barrabas

==================================================

My younger brother asked me what happens after we die. I told
  him we get buried under a bunch of dirt and worms eat our bodies.
  I guess I should have told him the truth-that most of us go to hell
  and burn eternally - but I didn't want to upset him.

[Bug c/12245] Crashes when compiling large initialized arrays

[barrabas at barrabas dot mv dot com]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245



------- Additional Comments From barrabas at barrabas dot mv dot com  2003-09-11 19:28 -------
Subject: Re:  Crashes when compiling large initialized arrays (gccBug: message 3 of 9)

> ------- Additional Comments From pinskia at gcc dot gnu dot org  2003-09-11 15:53 -------
> Please attach the preprocessed files to this bug.

	I would guess that the bug is caused by:

	1) The compiler allocates lots of storage for intermediate results.
	2) Virtual memory gets used up, and the next allocations fails.
	3) The allocation is not checked, leading to eventual failure.

																R. Barrabas

==================================================

Dictatorship (n): a form of government under which everything
                  which is not prohibited is compulsory.

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245



------- Additional Comments From pinskia at gcc dot gnu dot org  2003-09-17 04:07 -------
How much memory do you have?

[Bug c/12245] Crashes when compiling large initialized arrays

[barrabas at barrabas dot mv dot com]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245



------- Additional Comments From barrabas at barrabas dot mv dot com  2003-09-17 04:33 -------
Subject: Re:  Crashes when compiling large initialized arrays (gccBug: message 7 of 9)

> How much memory do you have?

256MB of ram + 256MB of swap.


==================================================

Those who live by the sword get shot by those who don't.

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245



------- Additional Comments From pinskia at gcc dot gnu dot org  2003-09-17 05:21 -------
Must be a memory intensive as I cannot reproduce on a system with 1GB of memory.

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245


pinskia at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |INVALID


------- Additional Comments From pinskia at gcc dot gnu dot org  2003-10-01 03:33 -------
Not GCC problem that the OS returns a non-zero pointer when memory is full.

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245


pinskia at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |UNCONFIRMED
           Keywords|                            |memory-hog
         Resolution|INVALID                     |


------- Additional Comments From pinskia at gcc dot gnu dot org  2003-10-01 03:35 -------
Though on the other hand GCC should not be such a hog of memory.

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245


pinskia at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING


------- Additional Comments From pinskia at gcc dot gnu dot org  2003-10-05 06:32 -------
Note that one day the web server will be down and "we" (meaning GCC developers) 
cannot access the testcase so we will ask you for the testcase to be attached, can you 
just attach the preprocessed source.

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2003-11-22 20:39 -------
The testcase takes about 445M on i686-pc-linux-gnu and more than 500M on powerpc-
apple-darwin7.0.0.  Will attach testcase.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW
     Ever Confirmed|                            |1
   Last reconfirmed|0000-00-00 00:00:00         |2003-11-22 20:39:52
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2004-01-19 12:57 -------
Still is a problem on the mainline, targeting 3.5.0 for now.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |3.5.0


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|3.5.0                       |---


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] Crashes when compiling large initialized arrays

[falk at debian dot org]

------- Additional Comments From falk at debian dot org  2004-09-12 15:36 -------
The URL of the test case doesn't seem to work anymore. Does anybody still have
the test case?


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pinskia at gcc dot gnu dot
                   |                            |org
             Status|NEW                         |WAITING


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] Crashes when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2004-12-02 15:42 -------
We take with the attached testcase about 300M with the C front-end but a huge amount more for the 
C++ front-end, why?

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW
   Last reconfirmed|2004-05-01 02:21:15         |2004-12-02 15:42:48
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.4.3/4.0.0 regression] Uses lots of memory when compiling large initialized arrays

[ian at wasabisystems dot com]

------- Additional Comments From ian at wasabisystems dot com  2004-12-02 15:53 -------
I attached a simple test case.  This is based on real existing code, although I
changed all the values to hide potentially proprietary information.  When I
compile this file without optimization, it uses some 200M, and garbage collects
while compiling this file.  The compilation takes 1 minutes, 45 seconds.  (This
is much better than gcc 3.4.3, actually, which used all available memory,
garbage collected twice, and wound up swapping for 10 minutes or so before
completing).

When compiling with 2.95.3, the compiler uses 20M and completes in 37 seconds.

The compiler used to work fine when processing very large initializers.  As it
read the initializer, gcc would output the initializer to the assembler file
directly.  This capability was removed here:
    http://gcc.gnu.org/ml/gcc-patches/2000-10/msg00933.html
The followups to that message mention this type of problem.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
      Known to fail|                            |3.4.3 4.0.0
      Known to work|                            |2.95.3
            Summary|Crashes when compiling large|[3.4.3/4.0.0 regression]
                   |initialized arrays          |Uses lots of memory when
                   |                            |compiling large initialized
                   |                            |arrays


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.3/3.4/4.0 regression] Uses lots of memory when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[3.4.3/4.0.0 regression]    |[3.3/3.4/4.0 regression]
                   |Uses lots of memory when    |Uses lots of memory when
                   |compiling large initialized |compiling large initialized
                   |arrays                      |arrays
   Target Milestone|---                         |3.4.4


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.3/3.4/4.0 regression] Uses lots of memory when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2004-12-02 16:10 -------
PR 14179 is for the C++ problem.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
  BugsThisDependsOn|                            |14179


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.3/3.4/4.0 regression] Uses lots of memory when compiling large initialized arrays

[rth at gcc dot gnu dot org]

------- Additional Comments From rth at gcc dot gnu dot org  2005-01-06 01:22 -------
No chance this is getting done for 4.0.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|3.4.4                       |4.1.0


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2005-07-22 18:36 -------
Does anyone have the current numbers for this bug?
I know for C, the memory usage has gone down but I don't know by how much.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2005-07-25 01:27 -------
c-typeck.c:5987 (output_init_element)                     0: 0.0%   23955160:100.0%   22770552:20.9%   
13171408:99.1%         19
convert.c:671 (convert_to_integer)                 52184768:37.8%          0: 0.0%          0: 0.0%          0: 0.0%    
1630774
ggc-common.c:193 (ggc_calloc)                      33547596:24.3%          0: 0.0%   33577352:30.7%        
544: 0.0%         45
tree.c:828 (build_int_cst_wide)                    52176864:37.8%          0: 0.0%   52177536:47.8%          0: 
0.0%    3261075


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2005-07-25 01:30 -------
There must be a better way to add on to celt in output_init_element.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[rguenth at gcc dot gnu dot org]

------- Additional Comments From rguenth at gcc dot gnu dot org  2005-09-12 08:55 -------
Max memory usage on (checking-disabled) mainline is now 253149kB (on a machine
with 1GB of RAM) for C and 403669kB for C++ (!)

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[rguenth at gcc dot gnu dot org]

------- Additional Comments From rguenth at gcc dot gnu dot org  2005-09-12 10:03 -------
One problem is that we use integer tree nodes for counting from zero to N, which
is just stupid and wastes RAM (because we do not collect during building the
initializer).  Of course we also store that "index" in the initializer element
list.

This whole mess asks for a (less general) rewrite.  Minimal-invasive surgery
is impossible.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245

[Bug c/12245] [3.4/4.0/4.1 regression] Uses lots of memory when compiling large initialized arrays

[giovannibajo at libero dot it]

------- Additional Comments From giovannibajo at libero dot it  2005-09-12 10:08 -------
The problem is that the gimplifier always want the index field of the 
constructor element to be filled. If you fix that in the obvious way (so 
that "no index" means "previous index + 1"), it should be quite easy to fix, 
for C++. In C, I have no clue how this interacts with designated initializers 
though.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12245