Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Dominique Dhumieres]

On powerpc-apple-darwin8, I killed jc1 after it took over 37:29.81 at:

...
echo ../../../../gcc-4.3-work/libjava/classpath/lib/gnu/javax/swing/text/html/parser/HTML_401F*.class> gnu/javax/swing/text/html/parser/HTML_401F.list
/bin/sh ./libtool --tag=GCJ --mode=compile /opt/gcc/darwin_buildw/gcc/gcj -B/opt/gcc/darwin_buildw/powerpc-apple-darwin8/ppc64/libjava/ -B/opt/gcc/darwin_buildw/gcc/ -fclasspath= -fbootclasspath=../../../../gcc-4.3-work/libjava/classpath/lib --encoding=UTF-8 -Wno-deprecated -fbootstrap-classes -g -O2  -m64 -c -o gnu/javax/swing/text/html/parser/HTML_401F.lo -fsource-filename=/opt/gcc/darwin_buildw/powerpc-apple-darwin8/ppc64/libjava/classpath/lib/classes -MT gnu/javax/swing/text/html/parser/HTML_401F.lo -MD -MP -MF gnu/javax/swing/text/html/parser/HTML_401F.deps @gnu/javax/swing/text/html/parser/HTML_401F.list
libtool: compile:  /opt/gcc/darwin_buildw/gcc/gcj -B/opt/gcc/darwin_buildw/powerpc-apple-darwin8/ppc64/libjava/ -B/opt/gcc/darwin_buildw/gcc/ -fclasspath= -fbootclasspath=../../../../gcc-4.3-work/libjava/classpath/lib --encoding=UTF-8 -Wno-deprecated -fbootstrap-classes -g -O2 -m64 -c -fsource-filename=/opt/gcc/darwin_buildw/powerpc-apple-darwin8/ppc64/libjava/classpath/lib/classes -MT gnu/javax/swing/text/html/parser/HTML_401F.lo -MD -MP -MF gnu/javax/swing/text/html/parser/HTML_401F.deps @gnu/javax/swing/text/html/parser/HTML_401F.list  -fno-common -o gnu/javax/swing/text/html/parser/.libs/HTML_401F.o

Was I pessimistic or is there a bug?

TIA

Dominique

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[David Daney]

Dominique Dhumieres wrote:
> On powerpc-apple-darwin8, I killed jc1 after it took over 37:29.81 at:
>
> ...
> echo
../../../../gcc-4.3-work/libjava/classpath/lib/gnu/javax/swing/text/html/parser/HTML_401F*.class>
gnu/javax/swing/text/html/parser/HTML_401F.list
> /bin/sh ./libtool --tag=GCJ --mode=compile
/opt/gcc/darwin_buildw/gcc/gcj
-B/opt/gcc/darwin_buildw/powerpc-apple-darwin8/ppc64/libjava/
-B/opt/gcc/darwin_buildw/gcc/ -fclasspath=
-fbootclasspath=../../../../gcc-4.3-work/libjava/classpath/lib
--encoding=UTF-8 -Wno-deprecated -fbootstrap-classes -g -O2  -m64 -c -o
gnu/javax/swing/text/html/parser/HTML_401F.lo
-fsource-filename=/opt/gcc/darwin_buildw/powerpc-apple-darwin8/ppc64/libjava/classpath/lib/classes
-MT gnu/javax/swing/text/html/parser/HTML_401F.lo -MD -MP -MF
gnu/javax/swing/text/html/parser/HTML_401F.deps
@gnu/javax/swing/text/html/parser/HTML_401F.list
> libtool: compile:  /opt/gcc/darwin_buildw/gcc/gcj
-B/opt/gcc/darwin_buildw/powerpc-apple-darwin8/ppc64/libjava/
-B/opt/gcc/darwin_buildw/gcc/ -fclasspath=
-fbootclasspath=../../../../gcc-4.3-work/libjava/classpath/lib
--encoding=UTF-8 -Wno-deprecated -fbootstrap-classes -g -O2 -m64 -c
-fsource-filename=/opt/gcc/darwin_buildw/powerpc-apple-darwin8/ppc64/libjava/classpath/lib/classes
-MT gnu/javax/swing/text/html/parser/HTML_401F.lo -MD -MP -MF
gnu/javax/swing/text/html/parser/HTML_401F.deps
@gnu/javax/swing/text/html/parser/HTML_401F.list  -fno-common -o
gnu/javax/swing/text/html/parser/.libs/HTML_401F.o
>
> Was I pessimistic or is there a bug?
>

You need at least 256MB of memory to compile HTML_401F.  A lot of time
is also useful.  If the system is not thrashing, I would give it a
couple of hours before calling it broken.

David Daney

> TIA
>
> Dominique

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Dominique Dhumieres]

David,

> ... I would give it a couple of hours before calling it broken.

You are right, a small "couple" of hours is need for the three stages: slightly
less than two hours on my machine (1.8Ghz G5). I never noticed that this part
was so long and I was too eager to do something else with the CPU. Sorry for the
noise.

Thanks for the answer.

Dominique

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Gerald Pfeifer]

On Sun, 28 Oct 2007, David Daney wrote:
> You need at least 256MB of memory to compile HTML_401F.  A lot of time
> is also useful.  If the system is not thrashing, I would give it a
> couple of hours before calling it broken.

Have we reduced memory consumption recently?  On i386-freebsd the number
I identified earlier this year was 700MB, 512MB definitely _not_ being 
sufficient.  I'd be very interested in your measurements, perhaps we can
reduce the limit somewhat!

Gerald

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[David Daney]

Gerald Pfeifer wrote:
> On Sun, 28 Oct 2007, David Daney wrote:
>   
>> You need at least 256MB of memory to compile HTML_401F.  A lot of time
>> is also useful.  If the system is not thrashing, I would give it a
>> couple of hours before calling it broken.
>>     
>
> Have we reduced memory consumption recently? 

That would be nice, but I don't think so.

>  On i386-freebsd the number
> I identified earlier this year was 700MB, 512MB definitely _not_ being 
> sufficient.  I'd be very interested in your measurements, perhaps we can
> reduce the limit somewhat!
>   

I regularly bootstrap c,c++,java on a mips-linux-gnu system with 256MB
of RAM (and 2000MB swap).

I also bootstrap c,c++ on mipsel-linux-gnu with a 128MB (also with large
swap) system.

Some files in libjava (interpreter.cc and HTML_401F.java) use a
proverbial ton of memory,but don't thrash so badly that they cannot
finish in a couple of hours with 256MB.

David Daney

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Eric Botcazou]

> I never noticed that this part was so long and I was too eager to do
> something else with the CPU. Sorry for the noise.

Don't be sorry, I can reproduce a compilation time surge for libjava on my 
machine (AMD64, 2.4 GHz, 1 GB).  In particular, HTML_401F.o now takes 40 min 
to compile for each version of the library.

The surge comes from the fix for PR tree-optimization/33870 (rev 129675).  
With it, 'make all-target-libjava' takes 261 min user time.  If I revert it,
do a 'make quickstrap' and run the command again, it only takes 109 min.

-- 
Eric Botcazou

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Richard Guenther]

On 10/28/07, Eric Botcazou <ebotcazou@libertysurf.fr> wrote:
> > I never noticed that this part was so long and I was too eager to do
> > something else with the CPU. Sorry for the noise.
>
> Don't be sorry, I can reproduce a compilation time surge for libjava on my
> machine (AMD64, 2.4 GHz, 1 GB).  In particular, HTML_401F.o now takes 40 min
> to compile for each version of the library.
>
> The surge comes from the fix for PR tree-optimization/33870 (rev 129675).
> With it, 'make all-target-libjava' takes 261 min user time.  If I revert it,
> do a 'make quickstrap' and run the command again, it only takes 109 min.

I'm working on it.

Richard.

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Eric Botcazou]

> I'm working on it.

Thanks.  However, don't we simply void the benefit of memory partitioning by 
recursing on the MPTs?

-- 
Eric Botcazou

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Richard Guenther]

On 10/28/07, Eric Botcazou <ebotcazou@libertysurf.fr> wrote:
> > I'm working on it.
>
> Thanks.  However, don't we simply void the benefit of memory partitioning by
> recursing on the MPTs?

Yes.  At least what compile-time is concerned (we still have less VOPs).  The
patch I just committed avoids some/most parts of the recursion.

The other option we have to fix the miscompile(s) is to put all SFTs of a
SFT parent var into the same partition always.  Which would essentially
either put all SFTs of a variable into a single MPT or none of the SFTs of
a variable into any MPT.

Let's see what the results are on the now "optimal" first strategy.

Richard.

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Gerald Pfeifer]

On Sun, 28 Oct 2007, Richard Guenther wrote:
>> Don't be sorry, I can reproduce a compilation time surge for libjava on 
>> my machine (AMD64, 2.4 GHz, 1 GB).  In particular, HTML_401F.o now 
>> takes 40 min to compile for each version of the library.
>>
>> The surge comes from the fix for PR tree-optimization/33870 (rev 129675).
> I'm working on it.

This is pretty tricky a case -- HTML_401F.o already is the single reason 
why GCC needs close to 700 MB of virtual memory to bootstrap.  It would
be really good to see this reduced.  Tom gave it a try by splitting the
input early this year, but this mostly seems like an issue of the middle 
end.

Gerald

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Gerald Pfeifer]

On Sun, 28 Oct 2007, David Daney wrote:
>>  On i386-freebsd the number I identified earlier this year was 700MB, 
>> 512MB definitely _not_ being sufficient.  I'd be very interested in 
>> your measurements, perhaps we can reduce the limit somewhat!
> I regularly bootstrap c,c++,java on a mips-linux-gnu system with 256MB
> of RAM (and 2000MB swap).

Here we go!  This means your system actually features 2256MB of memory, 
not just 256MB. ;-)  Sorry for not making it more clear that I was really
referring to overall memory, not just main memory.  According to my tests 
our peak in virtual memory use is a bit below 700MB while building libgcj.

Gerald

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Dominique Dhumieres]

There is something I don't understand: why the problem shows only (mainly) in jc1?
If a similar increase had happened in gfortran, I (and others) would have seen it.

Dominique

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[David Daney]

Gerald Pfeifer wrote:
> On Sun, 28 Oct 2007, David Daney wrote:
>   
>>>  On i386-freebsd the number I identified earlier this year was 700MB, 
>>> 512MB definitely _not_ being sufficient.  I'd be very interested in 
>>> your measurements, perhaps we can reduce the limit somewhat!
>>>       
>> I regularly bootstrap c,c++,java on a mips-linux-gnu system with 256MB
>> of RAM (and 2000MB swap).
>>     
>
> Here we go!  This means your system actually features 2256MB of memory, 
> not just 256MB. ;-)  Sorry for not making it more clear that I was really
> referring to overall memory, not just main memory.  According to my tests 
> our peak in virtual memory use is a bit below 700MB while building libgcj.
>
>   
I don't dispute that this is true.  However since most modern systems 
are able to swap to disk, it is often more useful to know the minimum 
amount of RAM required as it can be much more difficult to add RAM to a 
system than increase the size of its swap device.

David Daney

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Eric Botcazou]

> Let's see what the results are on the now "optimal" first strategy.

Far better, but I've still a 51 min gap on a full checking bootstrap.

-- 
Eric Botcazou

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Richard Guenther]

On 10/29/07, Eric Botcazou <ebotcazou@libertysurf.fr> wrote:
> > Let's see what the results are on the now "optimal" first strategy.
>
> Far better, but I've still a 51 min gap on a full checking bootstrap.

More tweaks to come.

Richard.

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Richard Guenther]

On 10/29/07, Richard Guenther <richard.guenther@gmail.com> wrote:
> On 10/29/07, Eric Botcazou <ebotcazou@libertysurf.fr> wrote:
> > > Let's see what the results are on the now "optimal" first strategy.
> >
> > Far better, but I've still a 51 min gap on a full checking bootstrap.
>
> More tweaks to come.

I have applied the "final" patch, if you want to re-test.  Though I expect
Diego to come up with a different fix for the original problem.

Richard.

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Eric Botcazou]

> I have applied the "final" patch, if you want to re-test.

Slightly better, but not much (7 min for a full-checking bootstrap).

-- 
Eric Botcazou

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Dominique Dhumieres]

A full bootstrap took 12h 38' on my machine (1.8Ghz G5) minus probably ~1h diverted for
other tasks. Although I did not measured accurately this time before, it could be my
fastest one.  Most of the improvement from my original post comes from
gnu/javax/swing/text/html/parser/HTML_401F.deps, for which the compile time went
from over 100 minutes to below 10 (twice due to multilib). For all the other pieces of
code the saving (if any) was clearly well below a factor 2. Does anyone understand
what is so special to jc1 and HTML_401F in order to explain this order of magnitude?

Dominique

Re: Infinite loop in compiling javax/swing/text/html/parser/HTML_401F.list

[Eric Botcazou]

> Most of the improvement from my original post comes from
> gnu/javax/swing/text/html/parser/HTML_401F.deps, for which the compile time
> went from over 100 minutes to below 10 (twice due to multilib). For all the
> other pieces of code the saving (if any) was clearly well below a factor 2. 
> Does anyone understand what is so special to jc1 and HTML_401F in order to	
> explain this order of magnitude?

Very likely because HTML_401F relies heavily on memory partitioning, which is 
kind of a fuse that triggers when the number of virtual operands goes to the 
roof.  The fix for PR tree-optimization/33870 basically short-circuited it.

-- 
Eric Botcazou