FRE may run out of memory

FRE may run out of memory

[dxq]

hi all,

We found that gcc would run out of memory on Windows when compiling a *big*
function (100000 lines).

More investigation shows that gcc crashes at the function *compute_avail*,
in tree-fre pass.  *compute_avail* collects information from basic blocks,
so memory is allocated to record informantion.
However, if there are huge number of basic blocks,  the memory would be
exhausted and gcc would crash down, especially for Windows PC, only 2G or 4G
memory generally. It's ok On linux, and *compute_avail* allocates *2.4G*
memory. I guess some optimization passes in gcc like FRE didn't consider the
extreme
case. 

When disable tree-fre pass, gcc crashes at IRA pass.  I will do more
investigation about that.

Any suggestions?

Thanks!

danxiaoqiang



--
View this message in context: http://gcc.1065356.n5.nabble.com/FRE-may-run-out-of-memory-tp1009578.html
Sent from the gcc - patches mailing list archive at Nabble.com.

Re: FRE may run out of memory

[Richard Biener]

On Sat, Feb 8, 2014 at 8:29 AM, dxq <ziyan01@163.com> wrote:
> hi all,
>
> We found that gcc would run out of memory on Windows when compiling a *big*
> function (100000 lines).
>
> More investigation shows that gcc crashes at the function *compute_avail*,
> in tree-fre pass.  *compute_avail* collects information from basic blocks,
> so memory is allocated to record informantion.
> However, if there are huge number of basic blocks,  the memory would be
> exhausted and gcc would crash down, especially for Windows PC, only 2G or 4G
> memory generally. It's ok On linux, and *compute_avail* allocates *2.4G*
> memory. I guess some optimization passes in gcc like FRE didn't consider the
> extreme
> case.

This was fixed for GCC 4.8, FRE no longer uses compute_avail (but PRE
still does).
Basically GCC 4.8 should (at -O1) compile most extreme cases just fine.

Richard.

> When disable tree-fre pass, gcc crashes at IRA pass.  I will do more
> investigation about that.
>
> Any suggestions?
>
> Thanks!
>
> danxiaoqiang
>
>
>
> --
> View this message in context: http://gcc.1065356.n5.nabble.com/FRE-may-run-out-of-memory-tp1009578.html
> Sent from the gcc - patches mailing list archive at Nabble.com.

Re: FRE may run out of memory

[dxq]

Richard Biener-2 wrote
> On Sat, Feb 8, 2014 at 8:29 AM, dxq <

> ziyan01@

> > wrote:
>> hi all,
>>
>> We found that gcc would run out of memory on Windows when compiling a
>> *big*
>> function (100000 lines).
>>
>> More investigation shows that gcc crashes at the function
>> *compute_avail*,
>> in tree-fre pass.  *compute_avail* collects information from basic
>> blocks,
>> so memory is allocated to record informantion.
>> However, if there are huge number of basic blocks,  the memory would be
>> exhausted and gcc would crash down, especially for Windows PC, only 2G or
>> 4G
>> memory generally. It's ok On linux, and *compute_avail* allocates *2.4G*
>> memory. I guess some optimization passes in gcc like FRE didn't consider
>> the
>> extreme
>> case.
> 
> This was fixed for GCC 4.8, FRE no longer uses compute_avail (but PRE
> still does).
> Basically GCC 4.8 should (at -O1) compile most extreme cases just fine.
> 
> Richard.

hi, Richard,

More  investigation shows that 
1, loop related passes take more compiling time and memory, especially
pass_rtl_move_loop_invariants, lim, 
  and at least lim on tree will impact a lot to the following passes. 
2, ira will take more than 20g memory in function *create_loop_tree_nodes*,
because ira chooses 'mixed' 
  or 'all' region when optimize level. 
3, sms pass always creats ddgs for all loops in compiled function, then does
sms optimization for all loops, 
  and finally frees ddgs. If there are huge number of loops, sms may crash
when creating ddgs because of 
  running out of memory.

The passes above , should someone confirm about memory pressure problem?

Thanks for your reply!

danxiaoqiang



--
View this message in context: http://gcc.1065356.n5.nabble.com/FRE-may-run-out-of-memory-tp1009578p1011035.html
Sent from the gcc - patches mailing list archive at Nabble.com.

Re: FRE may run out of memory

[Richard Biener]

On Fri, Feb 14, 2014 at 3:50 AM, dxq <ziyan01@163.com> wrote:
> Richard Biener-2 wrote
>> On Sat, Feb 8, 2014 at 8:29 AM, dxq &lt;
>
>> ziyan01@
>
>> &gt; wrote:
>>> hi all,
>>>
>>> We found that gcc would run out of memory on Windows when compiling a
>>> *big*
>>> function (100000 lines).
>>>
>>> More investigation shows that gcc crashes at the function
>>> *compute_avail*,
>>> in tree-fre pass.  *compute_avail* collects information from basic
>>> blocks,
>>> so memory is allocated to record informantion.
>>> However, if there are huge number of basic blocks,  the memory would be
>>> exhausted and gcc would crash down, especially for Windows PC, only 2G or
>>> 4G
>>> memory generally. It's ok On linux, and *compute_avail* allocates *2.4G*
>>> memory. I guess some optimization passes in gcc like FRE didn't consider
>>> the
>>> extreme
>>> case.
>>
>> This was fixed for GCC 4.8, FRE no longer uses compute_avail (but PRE
>> still does).
>> Basically GCC 4.8 should (at -O1) compile most extreme cases just fine.
>>
>> Richard.
>
> hi, Richard,
>
> More  investigation shows that
> 1, loop related passes take more compiling time and memory, especially
> pass_rtl_move_loop_invariants, lim,
>   and at least lim on tree will impact a lot to the following passes.
> 2, ira will take more than 20g memory in function *create_loop_tree_nodes*,
> because ira chooses 'mixed'
>   or 'all' region when optimize level.
> 3, sms pass always creats ddgs for all loops in compiled function, then does
> sms optimization for all loops,
>   and finally frees ddgs. If there are huge number of loops, sms may crash
> when creating ddgs because of
>   running out of memory.
>
> The passes above , should someone confirm about memory pressure problem?

What compiler version did you check?  I think that 4.8 has improvements
for 1. and 2. (SMS is unmaintained).  Note that we only spent time to
make -O1 behave sanely with extremely large functions.

Finally I'd suggest you open a bugreport and attach a testcase to it
that exposes the issues you list.

Richard.

> Thanks for your reply!
>
> danxiaoqiang
>
>
>
> --
> View this message in context: http://gcc.1065356.n5.nabble.com/FRE-may-run-out-of-memory-tp1009578p1011035.html
> Sent from the gcc - patches mailing list archive at Nabble.com.

Re: FRE may run out of memory

[dxq]

Richard Biener-2 wrote
> On Fri, Feb 14, 2014 at 3:50 AM, dxq <

> ziyan01@

> > wrote:
> 
> What compiler version did you check?  I think that 4.8 has improvements
> for 1. and 2. (SMS is unmaintained).  Note that we only spent time to
> make -O1 behave sanely with extremely large functions.
> 
> Finally I'd suggest you open a bugreport and attach a testcase to it
> that exposes the issues you list.
> 
> Richard.

hi Richard,

We are working on gcc-4.7.1.  seems that these issues have been fixed in
gcc-4.8.3.

BTW, we notice that Obstack is used in LIM for memory management, which is
really a nice way, 
and also, we used Ostack to solve the GGC problem in our SMS-UNROLL
framework, 
we put all backup into the  Obstack, so it's safe now. 

http://gcc.1065356.n5.nabble.com/A-GGC-related-question-td988400.html
 
Thanks!

danxiaoqiang



--
View this message in context: http://gcc.1065356.n5.nabble.com/FRE-may-run-out-of-memory-tp1009578p1012489.html
Sent from the gcc - patches mailing list archive at Nabble.com.