Can GCC use more than 4G while compiling ?

Can GCC use more than 4G while compiling ?

[Tomas Kalibera]

Hello,

GCC runs out of memory while compiling my very large generated source 
file. Although I have x86_64 with more than 4G of RAM, it seems that GCC 
only uses something less than 4G. Is there a way to allow GCC to use 
more ? Something like ggc-heap-size...

I'm using GCC 4.2.3, compiled from sources, gcc binary is a 64-bit 
executable.

Thanks,
Tomas

Re: Can GCC use more than 4G while compiling ?

[Ian Lance Taylor]

Tomas Kalibera <kalibera@cs.purdue.edu> writes:

> GCC runs out of memory while compiling my very large generated source
> file. Although I have x86_64 with more than 4G of RAM, it seems that
> GCC only uses something less than 4G. Is there a way to allow GCC to
> use more ? Something like ggc-heap-size...
> 
> I'm using GCC 4.2.3, compiled from sources, gcc binary is a 64-bit
> executable.

What is the exact error message?  Are all the compiler binaries 64-bit
executables?

gcc should not run out of memory until malloc or mmap fail to allocate
any more memory.  There is no maximum heap size.  gcc's garbage
collector doesn't run at arbitrary times; in particular it doesn't run
when there is an allocation failure, nor when the heap reaches a
certain size.

Ian

Re: Can GCC use more than 4G while compiling ?

[Tomas Kalibera]

Ian Lance Taylor wrote:
> Tomas Kalibera <kalibera@cs.purdue.edu> writes:
>
>   
>> GCC runs out of memory while compiling my very large generated source
>> file. Although I have x86_64 with more than 4G of RAM, it seems that
>> GCC only uses something less than 4G. Is there a way to allow GCC to
>> use more ? Something like ggc-heap-size...
>>
>> I'm using GCC 4.2.3, compiled from sources, gcc binary is a 64-bit
>> executable.
>>     
>
> What is the exact error message?  Are all the compiler binaries 64-bit
> executables?
>   
- "virtual memory exhausted: Cannot allocate memory"
- yes, I re-checked all binaries are 64-bit

I'm now trying my luck with

"--param ggc-min-expand=0 --param ggc-min-heapsize=4096"

Compilation is now by orders of magnitude slower, it's been already 
compiling the single file for 6 hours, I don't know how far it got.

The machine has 6G of RAM + 16G of swap and I can malloc more than 4G in 
a test application.
> gcc should not run out of memory until malloc or mmap fail to allocate
> any more memory.  There is no maximum heap size.  gcc's garbage
> collector doesn't run at arbitrary times; in particular it doesn't run
> when there is an allocation failure, nor when the heap reaches a
> certain size.
>   
OK, and there is not any special option needed when compiling the gcc 
compiler ?
I compiled with

Target: x86_64-pc-linux-gnu
Configured with: ./configure 
 --prefix=my_directory
 --host=x86_64-pc-linux-gnu 
 --build=x86_64-pc-linux-gnu 
 --disable-altivec 
 --enable-nls 
 --with-system-zlib 
 --enable-multilib 
 --enable-languages=c,c++ 
 --enable-shared 
 --enable-threads=posix 
 --enable-__cxa_atexit 
 --enable-clocale=gnu 
 --enable-mpfr
Thread model: posix
gcc version 4.2.3


Thanks, Tomas
> Ian
>   

Re: Can GCC use more than 4G while compiling ?

[Ian Lance Taylor]

Tomas Kalibera <kalibera@cs.purdue.edu> writes:

> > What is the exact error message?  Are all the compiler binaries 64-bit
> > executables?
> >
> - "virtual memory exhausted: Cannot allocate memory"

OK, that means that a call to mmap using MAP_ANONYMOUS returned
MAP_FAILED with errno set to ENOMEM.  I see from the mmap man page
that that can happen when the process's has reached the maximum number
of mappings.  It seems at least possible that you are running out of
maps before you are running out of memory.  On a GNU/Linux system you
can see the maximum map count with "/sbin/sysctl vm.max_map_count".
You can change that value using /sbin/sysctl -w.


> I'm now trying my luck with
> 
> "--param ggc-min-expand=0 --param ggc-min-heapsize=4096"
> 
> Compilation is now by orders of magnitude slower, it's been already
> compiling the single file for 6 hours, I don't know how far it got.

Those options are just going to make the compiler run much slower,
they are unlikely to reduce the maximum amount of allocated memory.


> OK, and there is not any special option needed when compiling the gcc
> compiler ?

No.

Ian

Re: Can GCC use more than 4G while compiling ?

[Tomas Kalibera]

Ian Lance Taylor wrote:
> Tomas Kalibera <kalibera@cs.purdue.edu> writes:
>
>   
>>> What is the exact error message?  Are all the compiler binaries 64-bit
>>> executables?
>>>
>>>       
>> - "virtual memory exhausted: Cannot allocate memory"
>>     
>
> OK, that means that a call to mmap using MAP_ANONYMOUS returned
> MAP_FAILED with errno set to ENOMEM.  I see from the mmap man page
> that that can happen when the process's has reached the maximum number
> of mappings.  It seems at least possible that you are running out of
> maps before you are running out of memory.  On a GNU/Linux system you
> can see the maximum map count with "/sbin/sysctl vm.max_map_count".
> You can change that value using /sbin/sysctl -w.
>   

Thanks! This helped.

I assume that it might also help to override GGC_QUIRE_SIZE (now it's 
set in ggc-page.c to 256, when mmap is used). But I did not check.

Tomas