[Bug c/44563] New: GCC uses a lot of RAM when compiling a large numbers of functions

[Bug c/44563] New: GCC uses a lot of RAM when compiling a large numbers of functions

[jvoss at altsci dot com]

When compiling a 5 MB C file with 65337 functions, gcc takes over 700 MB of ram
compiling the file to assembly. Files with fewer functions (32769 and 16385)
take exponentially less time. A workaround is to split the file into multiple c
files and compile into intermediate files seperately.

tinycc aka tcc takes 0.5 seconds to compile the same file.

I'd be happy to provide more information and justifications for such a large
number of functions, but I assume that the bug can be fixed without it.


-- 
           Summary: GCC uses a lot of RAM when compiling a large numbers of
                    functions
           Product: gcc
           Version: 4.3.4
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: jvoss at altsci dot com
  GCC host triplet: x86_64-pc-linux-gnu
GCC target triplet: x86_64-pc-linux-gnu


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44563

[Bug tree-optimization/44563] GCC uses a lot of RAM when compiling a large numbers of functions

[jvoss at altsci dot com]

------- Comment #1 from jvoss at altsci dot com  2010-06-17 04:14 -------
Created an attachment (id=20931)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=20931&action=view)
A c file that generates a c file that is the test case.

gcc -o gcc_64kgen1 gcc_64kgen1.c -Wall
./gcc_64kgen1 > gcc_64k3.c
gcc -o gcc_64k3 gcc_64k3.c -Wall
# seven or so minutes later, it finishes.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44563

[Bug tree-optimization/44563] GCC uses a lot of RAM when compiling a large numbers of functions

[rguenth at gcc dot gnu dot org]

------- Comment #2 from rguenth at gcc dot gnu dot org  2010-06-17 10:36 -------
The issue is not so much the number of functions but the number of calls
in main (and thus its size).

With GCC 4.3.4 I see

 parser                :   6.87 (12%) usr   0.79 (14%) sys   8.14 (13%) wall 
266316 kB (18%) ggc
 expand                :  14.02 (24%) usr   1.38 (25%) sys  15.50 (24%) wall 
619261 kB (41%) ggc
 global alloc          :  10.65 (18%) usr   0.27 ( 5%) sys  10.79 (17%) wall  
81920 kB ( 5%) ggc
 final                 :   6.16 (11%) usr   0.17 ( 3%) sys   6.21 (10%) wall  
32704 kB ( 2%) ggc
 TOTAL                 :  57.78             5.60            63.76           
1509835 kB

peaking at 1.3GB virtual memory usage.

With GCC 4.5 I see

 parser                :   1.81 ( 3%) usr   0.66 ( 5%) sys   2.42 ( 3%) wall 
196460 kB (13%) ggc
 expand                :  15.24 (25%) usr   3.45 (28%) sys  18.95 (26%) wall 
506302 kB (34%) ggc
 integrated RA         :  11.75 (20%) usr   0.27 ( 2%) sys  12.09 (17%) wall  
89600 kB ( 6%) ggc
 reload                :   6.07 (10%) usr   0.15 ( 1%) sys   5.98 ( 8%) wall  
54527 kB ( 4%) ggc
 thread pro- & epilogue:   2.97 ( 5%) usr   0.08 ( 1%) sys   3.22 ( 4%) wall 
131586 kB ( 9%) ggc
 final                 :   4.73 ( 8%) usr   0.15 ( 1%) sys   5.17 ( 7%) wall  
36288 kB ( 2%) ggc
 TOTAL                 :  59.83            12.45            72.64           
1504261 kB

peaking at 1.7GB virtual memory usage.

Turning on basic optimization (-O1) inlines all your functions making
main() smaller, reducing peak memory usage to around 1GB but using
a lot of compile-time (I stopped it after 10min).

You basically have a gigantic basic-block in main() and GCC isn't
tuned to handle that case well.

Removing the main() function unfortunately doesn't make GCC behave much
saner here with regarding to memory usage.  Removing all functions
but main reproduces the above figures with slightly reduced memory usage
and increased compile-time because the functions are not known to be const.


-- 

rguenth at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever Confirmed|0                           |1
           Keywords|                            |compile-time-hog, memory-hog
   Last reconfirmed|0000-00-00 00:00:00         |2010-06-17 10:36:48
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44563

[Bug tree-optimization/44563] GCC uses a lot of RAM when compiling a large numbers of functions

[rguenth at gcc dot gnu dot org]

------- Comment #3 from rguenth at gcc dot gnu dot org  2010-06-17 10:44 -------
Growth in time and memory is linear in the number of functions for me (GCC
4.5).

rguenther@murzim:/tmp> ~/bin/maxmem2.sh /usr/bin/time gcc-4.5 -S gcc_16kgen1.c
-o /dev/null
14.60user 0.61system 0:15.42elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
80inputs+0outputs (0major+129132minor)pagefaults 0swaps
total: 537864 kB
rguenther@murzim:/tmp> ~/bin/maxmem2.sh /usr/bin/time gcc-4.5 -S gcc_32kgen1.c
-o /dev/null
29.81user 1.43system 0:31.45elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+222866minor)pagefaults 0swaps
total: 905120 kB
rguenther@murzim:/tmp> ~/bin/maxmem2.sh /usr/bin/time gcc-4.5 -S gcc64k3.c -o
/dev/null
60.65user 2.72system 1:03.92elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+442682minor)pagefaults 0swaps
total: 1766812 kB

anyway - Honza, this is probably another nice target for profiling ...

Good old GCC 3.3 uses less memory but takes more compile-time btw.

I suppose this is another case where we build a load of INTEGER_CSTs
that remain live (like in the huge array-initializer case).  Which doesn't
explain the compile-time issue of course.


-- 

rguenth at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hubicka at gcc dot gnu dot
                   |                            |org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44563

[Bug tree-optimization/44563] GCC uses a lot of RAM when compiling a large numbers of functions

[jakub at gcc dot gnu dot org]

------- Comment #4 from jakub at gcc dot gnu dot org  2010-06-17 12:28 -------
Created an attachment (id=20932)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=20932&action=view)
callgrind.out.bz2

Seems it is mainly df and RA that eats the time for -O0 -g0.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44563