Should we remove java from the default bootstrap languages?

Should we remove java from the default bootstrap languages?

[Diego Novillo]

I posted this question to the SC panel at the GCC Summit today.  I 
wanted to consider the possibility of making java a non-default language.

Would the Java maintainers agree to this?

The rationale is mostly that Java takes a very long time to build, and 
it does not seem to be used widely enough (maybe I'm wrong the latter 
point).


Thanks.  Diego.

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Diego Novillo wrote:
> 
> I posted this question to the SC panel at the GCC Summit today.  I
> wanted to consider the possibility of making java a non-default language.
> 
> Would the Java maintainers agree to this?
> 
> The rationale is mostly that Java takes a very long time to build, and
> it does not seem to be used widely enough (maybe I'm wrong the latter
> point).

Heh.  How widely is enough?

If this were to happen it would break repeatedly.  It's used as part of
every Fedora and Debian system, even in the presence of OpenJDK, as OpenJDK
only runs on x86_)64 and 386 systems, although there's a (very slow) version
that runs on PPC.  gcj is essential for bootstrapping OpenJDK on new systems.
gcj is the only free java on some systems, particularly ARM and embedded MIPS.

I'm seriously considering porting the openJDK library to gcj.

I wonder if some compromise less than disabling it as a default everywhere
is possible.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Tom Tromey]

>>>>> "Andrew" == Andrew Haley <aph@redhat.com> writes:

Diego> I posted this question to the SC panel at the GCC Summit today.  I
Diego> wanted to consider the possibility of making java a non-default language.

Andrew> If this were to happen it would break repeatedly.

Yeah, our experience back when libgcj was not in the tree was not very
good.  It broke all the time.

But, I am actually ok with having it be disabled by default, provided
that regressions affect gcj are treated seriously: fixed in a timely
way by the person causing the regression, or, if not, letting gcj
maintainers start the patch-reversion clock.

If we make this change I'll set up an auto-tester on the compile farm
that builds gcj along with everything else.  I think this would
provide a pretty reasonable compromise.  Ideally we could find a PPC
box somewhere to do this as well -- anyone have some cycles to spare?

What do you think of this?

Another idea is to build jc1 but not libgcj.  That would prevent a
certain (more minor) class of breakage.

What is behind my thinking here is just the fact that gcj development
has dropped off quite a bit.

Tom

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Tom Tromey wrote:
>>>>>> "Andrew" == Andrew Haley <aph@redhat.com> writes:
> 
> Diego> I posted this question to the SC panel at the GCC Summit today.  I
> Diego> wanted to consider the possibility of making java a non-default language.
> 
> Andrew> If this were to happen it would break repeatedly.
> 
> Yeah, our experience back when libgcj was not in the tree was not very
> good.  It broke all the time.
> 
> But, I am actually ok with having it be disabled by default, provided
> that regressions affect gcj are treated seriously: fixed in a timely
> way by the person causing the regression, or, if not, letting gcj
> maintainers start the patch-reversion clock.
> 
> If we make this change I'll set up an auto-tester on the compile farm
> that builds gcj along with everything else.  I think this would
> provide a pretty reasonable compromise.  Ideally we could find a PPC
> box somewhere to do this as well -- anyone have some cycles to spare?
> 
> What do you think of this?
> 
> Another idea is to build jc1 but not libgcj.  That would prevent a
> certain (more minor) class of breakage.

My suggestion is that we build jc1 but not libgcj by default.
HOWEVER, we build libgcj on the autobuilders and make very sure that
if anyone breaks the libgcj build they have to fix their breakage,
even tho it's not part of the default build.  That will prevent most
of the bitrot while we figure out how to go forward.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Ian Lance Taylor]

Andrew Haley <aph@redhat.com> writes:

[ Java ]

> I wonder if some compromise less than disabling it as a default everywhere
> is possible.

Is it possible to only build and test a subset of libjava by default,
and still get useful coverage of Java?  The issue I see is simply that
building libjava is half of the time required for a bootstrap.

Ian

Re: Should we remove java from the default bootstrap languages?

[Tom Tromey]

>>>>> "Ian" == Ian Lance Taylor <iant@google.com> writes:

Ian> Is it possible to only build and test a subset of libjava by default,
Ian> and still get useful coverage of Java?  The issue I see is simply that
Ian> building libjava is half of the time required for a bootstrap.

We could look into this.  The minimum subset is probably several
hundred classes.  For instance, Class refers to URL, which will
probably pull in most of java.net.

Maybe David Daney already knows what is minimal...?

Tom

Re: Should we remove java from the default bootstrap languages?

[Janis Johnson]

On Fri, 2008-06-20 at 10:41 -0600, Tom Tromey wrote:
> >>>>> "Andrew" == Andrew Haley <aph@redhat.com> writes:
> 
> Diego> I posted this question to the SC panel at the GCC Summit today.  I
> Diego> wanted to consider the possibility of making java a non-default language.
> 
> Andrew> If this were to happen it would break repeatedly.
> 
> Yeah, our experience back when libgcj was not in the tree was not very
> good.  It broke all the time.
> 
> But, I am actually ok with having it be disabled by default, provided
> that regressions affect gcj are treated seriously: fixed in a timely
> way by the person causing the regression, or, if not, letting gcj
> maintainers start the patch-reversion clock.
> 
> If we make this change I'll set up an auto-tester on the compile farm
> that builds gcj along with everything else.  I think this would
> provide a pretty reasonable compromise.  Ideally we could find a PPC
> box somewhere to do this as well -- anyone have some cycles to spare?

I'll continue to include java in my nightly builds on
powerpc64-unknown-linux-gnu, for which I test with both -m32 and -m64.
My test results for trunk are sent to gcc-testresults daily.  I also
do regular (not daily) builds for release branches as well as trunk,
but mailing to the outside world hasn't been working lately on the
machine where those builds are done; I'll get that fixed.

Janis

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Janis Johnson wrote:
> On Fri, 2008-06-20 at 10:41 -0600, Tom Tromey wrote:
>>>>>>> "Andrew" == Andrew Haley <aph@redhat.com> writes:
>> Diego> I posted this question to the SC panel at the GCC Summit today.  I
>> Diego> wanted to consider the possibility of making java a non-default language.
>>
>> Andrew> If this were to happen it would break repeatedly.
>>
>> Yeah, our experience back when libgcj was not in the tree was not very
>> good.  It broke all the time.
>>
>> But, I am actually ok with having it be disabled by default, provided
>> that regressions affect gcj are treated seriously: fixed in a timely
>> way by the person causing the regression, or, if not, letting gcj
>> maintainers start the patch-reversion clock.
>>
>> If we make this change I'll set up an auto-tester on the compile farm
>> that builds gcj along with everything else.  I think this would
>> provide a pretty reasonable compromise.  Ideally we could find a PPC
>> box somewhere to do this as well -- anyone have some cycles to spare?
> 
> I'll continue to include java in my nightly builds on
> powerpc64-unknown-linux-gnu, for which I test with both -m32 and -m64.
> My test results for trunk are sent to gcc-testresults daily.  I also
> do regular (not daily) builds for release branches as well as trunk,
> but mailing to the outside world hasn't been working lately on the
> machine where those builds are done; I'll get that fixed.

That would be really helpful.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Tom Tromey]

Andrew> My suggestion is that we build jc1 but not libgcj by default.
Andrew> HOWEVER, we build libgcj on the autobuilders and make very sure that
Andrew> if anyone breaks the libgcj build they have to fix their breakage,
Andrew> even tho it's not part of the default build.  That will prevent most
Andrew> of the bitrot while we figure out how to go forward.

Good idea.

Maybe instead of removing libgcj from the default builds, we can just
say that maintainers can --disable-libjava for regression testing
purposes.  This would make testers continue to test libgcj by default.

Tom

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Tom Tromey wrote:
>>>>>> "Ian" == Ian Lance Taylor <iant@google.com> writes:
> 
> Ian> Is it possible to only build and test a subset of libjava by default,
> Ian> and still get useful coverage of Java?  The issue I see is simply that
> Ian> building libjava is half of the time required for a bootstrap.
> 
> We could look into this.  The minimum subset is probably several
> hundred classes.  For instance, Class refers to URL, which will
> probably pull in most of java.net.
> 
> Maybe David Daney already knows what is minimal...?

Yeah.  Let's not disable java as default until we've at least
investigated this.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Diego Novillo]

On 6/19/08 11:06 AM, Janis Johnson wrote:

> I'll continue to include java in my nightly builds on
> powerpc64-unknown-linux-gnu, for which I test with both -m32 and -m64.

Likewise.  I will keep including java in my ppc64, i686 and x86_64 daily 
testers.  I'm only trying to address the every day experience for 
developers.

I also agree that java breakages should be treated the same way as any 
other bootstrap breaking patch.


Diego.

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Diego Novillo wrote:
> On 6/19/08 11:06 AM, Janis Johnson wrote:
> 
>> I'll continue to include java in my nightly builds on
>> powerpc64-unknown-linux-gnu, for which I test with both -m32 and -m64.
> 
> Likewise.  I will keep including java in my ppc64, i686 and x86_64 daily
> testers.  I'm only trying to address the every day experience for
> developers.
> 
> I also agree that java breakages should be treated the same way as any
> other bootstrap breaking patch.

Well, it won't be quite the same, since developers won't
be expected to know they've broken Java.  We need to make
sure this "you broke it, you fix it" policy is written in
stone somewhere.

Ok, we have a plan.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Florian Weimer]

* Tom Tromey:

> We could look into this.  The minimum subset is probably several
> hundred classes.  For instance, Class refers to URL, which will
> probably pull in most of java.net.

Can't you fallback to the interpreter for the URL class?

Re: Should we remove java from the default bootstrap languages?

[Tom Tromey]

>>>>> "Florian" == Florian Weimer <fw@deneb.enyo.de> writes:

>> We could look into this.  The minimum subset is probably several
>> hundred classes.  For instance, Class refers to URL, which will
>> probably pull in most of java.net.

Florian> Can't you fallback to the interpreter for the URL class?

We prefer to build most of the core without -findirect-dispatch,
because it yields better performance.  This means that all the direct
dependencies must be compiled and linked in.

Of course we could try to change this.  But, then we'd need a special
mode for this kind of build (I think) and we wouldn't really be
testing things the same way that users build them.

Tom

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Tom Tromey wrote:
>>>>>> "Florian" == Florian Weimer <fw@deneb.enyo.de> writes:
> 
>>> We could look into this.  The minimum subset is probably several
>>> hundred classes.  For instance, Class refers to URL, which will
>>> probably pull in most of java.net.
> 
> Florian> Can't you fallback to the interpreter for the URL class?
> 
> We prefer to build most of the core without -findirect-dispatch,
> because it yields better performance.  This means that all the direct
> dependencies must be compiled and linked in.
> 
> Of course we could try to change this.

No.  It would cause all manner of problems at startup; not worth
it.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Kaveh R. GHAZI]

On Fri, 20 Jun 2008, Tom Tromey wrote:

> Andrew> My suggestion is that we build jc1 but not libgcj by default.
> Andrew> HOWEVER, we build libgcj on the autobuilders and make very sure that
> Andrew> if anyone breaks the libgcj build they have to fix their breakage,
> Andrew> even tho it's not part of the default build.  That will prevent most
> Andrew> of the bitrot while we figure out how to go forward.
>
> Good idea.
>
> Maybe instead of removing libgcj from the default builds, we can just
> say that maintainers can --disable-libjava for regression testing
> purposes.  This would make testers continue to test libgcj by default.
> Tom

Ugh, I think this is a terrible idea.  It took me all of zero days to find
an example of libjava breaking when someone didn't test it:

http://gcc.gnu.org/ml/gcc-patches/2008-06/msg01351.html

If we make not testing java an actual policy, you can expect much more
breakage.  Things that aren't tested suffer bitrot, plain and simple.

That aside, our current policy already allows e.g. not testing java if
your change is to a part of the compiler that can't possible affect it.
E.g. changing the fortran or ada frontends doesn't affect java.  But IMHO
we shouldn't relax the testing rules for the overlapping parts if we want
to keep our bits all working nicely.

		--Kaveh
--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Should we remove java from the default bootstrap languages?

[Diego Novillo]

On Fri, Jun 20, 2008 at 16:56, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:

> That aside, our current policy already allows e.g. not testing java if
> your change is to a part of the compiler that can't possible affect it.

I didn't make it completely clear, but my suggestion was mostly to
help us middle/back-end hackers.


Diego.

Re: Should we remove java from the default bootstrap languages?

[Kaveh R. GHAZI]

On Fri, 20 Jun 2008, Diego Novillo wrote:

> On Fri, Jun 20, 2008 at 16:56, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
>
> > That aside, our current policy already allows e.g. not testing java if
> > your change is to a part of the compiler that can't possible affect it.
>
> I didn't make it completely clear, but my suggestion was mostly to
> help us middle/back-end hackers.
> Diego.

Yeah, that's what worries me, all roads lead through the middle-end.  :-)

		--Kaveh
--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Should we remove java from the default bootstrap languages?

[Joe Buck]

On Fri, Jun 20, 2008 at 05:16:41PM -0400, Kaveh R. GHAZI wrote:
> On Fri, 20 Jun 2008, Diego Novillo wrote:
> 
> > On Fri, Jun 20, 2008 at 16:56, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
> >
> > > That aside, our current policy already allows e.g. not testing java if
> > > your change is to a part of the compiler that can't possible affect it.
> >
> > I didn't make it completely clear, but my suggestion was mostly to
> > help us middle/back-end hackers.
> > Diego.
> 
> Yeah, that's what worries me, all roads lead through the middle-end.  :-)

But if I understood the proposal correctly, auto-testers (as well as Java
developers) would continue to test Java on a daily basis, and anyone
submitting a patch that caused breakage would be responsible for fixing
the damage or reverting.  If problems are always detected within one day,
the opportunities for rot are limited.

Beyond that I wouldn't venture an opinion on whether it's the right thing.

Re: Should we remove java from the default bootstrap languages?

[Steven Bosscher]

On Fri, Jun 20, 2008 at 11:16 PM, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
> On Fri, 20 Jun 2008, Diego Novillo wrote:
>
>> On Fri, Jun 20, 2008 at 16:56, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
>>
>> > That aside, our current policy already allows e.g. not testing java if
>> > your change is to a part of the compiler that can't possible affect it.
>>
>> I didn't make it completely clear, but my suggestion was mostly to
>> help us middle/back-end hackers.
>> Diego.
>
> Yeah, that's what worries me, all roads lead through the middle-end.  :-)

What is far more worrying to me, actually, is that libjava grows
bigger and bigger and bigger with every release, so that testing it
costs developers who care zilch about java (i.e. most people) get
penalized more and more with increased bootstrap and test times.

In my latest timings, building and testing java takes close to two
thirds of the total bootstrap+test time with all default languages
enabled. That's a lot for a practically unused library and front end.
It is the limiting time factor for me, at least, when doing gcc
development.

Things would be so much better for me if we'd only test a subset of
libjava by default... *sigh*

Gr.
Steven

Re: Should we remove java from the default bootstrap languages?

[David Miller]

From: "Steven Bosscher" <stevenb.gcc@gmail.com>
Date: Sat, 21 Jun 2008 00:09:26 +0200

> What is far more worrying to me, actually, is that libjava grows
> bigger and bigger and bigger with every release, so that testing it
> costs developers who care zilch about java (i.e. most people) get
> penalized more and more with increased bootstrap and test times.

I agree and will admit that this is the one thing that has curtailed
severely my contributions to GCC in the past 4 to 5 years.

I used to be able to bootstrap gcc fully in minutes on average
hardware 6 or so years ago.  Those days are long gone.  On my largest
64 cpu and 128 cpu boxes it takes forever these days.

The libjava build is notoriously not helped by parallelization because
certain compiles are extremely expensive, which effectively
single-threads the build.

Re: Should we remove java from the default bootstrap languages?

[Kaveh R. Ghazi]

From: "Joe Buck" <Joe.Buck@synopsys.COM>

> On Fri, Jun 20, 2008 at 05:16:41PM -0400, Kaveh R. GHAZI wrote:
>> On Fri, 20 Jun 2008, Diego Novillo wrote:
>>
>> > On Fri, Jun 20, 2008 at 16:56, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> 
>> > wrote:
>> >
>> > > That aside, our current policy already allows e.g. not testing java 
>> > > if
>> > > your change is to a part of the compiler that can't possible affect 
>> > > it.
>> >
>> > I didn't make it completely clear, but my suggestion was mostly to
>> > help us middle/back-end hackers.
>> > Diego.
>>
>> Yeah, that's what worries me, all roads lead through the middle-end.  :-)
>
> But if I understood the proposal correctly, auto-testers (as well as Java
> developers) would continue to test Java on a daily basis, and anyone
> submitting a patch that caused breakage would be responsible for fixing
> the damage or reverting.  If problems are always detected within one day,
> the opportunities for rot are limited.

Fundamentally, our philosophy has been to catch errors *before* they get 
into the repository.  Sure one day of breaking the trunk isn't so bad, but 
when it breaks it affects hundreds of developers and it adds up.  Everyone 
separately either stops and waits, or tracks down which patch it was and 
reverts it so they can continue working.  Either way lots of time is wasted, 
and this serves as a counter argument against the time spent testing patches 
with java enabled.  I suspect some people run their tests overnight, in that 
case it doesn't matter if the regtest takes 2 hours or 5.  Either way the 
results will be waiting for you in the morning.

I don't like the idea of taking java out, but if we do I suggest we swap in 
objc++.  That would only add 42 seconds to the bootstrap and test process. 
:-)

http://gcc.gnu.org/ml/gcc-patches/2008-03/msg01763.html

--Kaveh

Re: Should we remove java from the default bootstrap languages?

[Richard Kenner]

> Fundamentally, our philosophy has been to catch errors *before* they get 
> into the repository.  Sure one day of breaking the trunk isn't so bad, but 
> when it breaks it affects hundreds of developers and it adds up.  Everyone 
> separately either stops and waits, or tracks down which patch it was and 
> reverts it so they can continue working.  

An interesting question that I see as relevant here and for which I have no
data is: what percentage of the time does a patch cause an error *only*
in libjava?   I think you have to weigh the cost of the build of that
library against the number of bugs that it finds.

Re: Should we remove java from the default bootstrap languages?

[Steven Bosscher]

On Sat, Jun 21, 2008 at 12:41 AM, Kaveh R. Ghazi <ghazi@caip.rutgers.edu> wrote:
> Fundamentally, our philosophy has been to catch errors *before* they get
> into the repository.  Sure one day of breaking the trunk isn't so bad, but
> when it breaks it affects hundreds of developers and it adds up.

But, for languages that are not enabled by default, no-one is directly
affected except the people who might have caused the breakage, and the
people who are working on the broken part of the compiler. In the case
of java, there are not hundreds but only a handful of people working
on it.



> I don't like the idea of taking java out, but if we do I suggest we swap in
> objc++.  That would only add 42 seconds to the bootstrap and test process.
> :-)

Exchange one unused language for another, great idea :-)  I'd rather
see objc++ go away completely, since Apple is clearly not keeping its
promise to maintain it, and AFAIK no-one else uses this language.

Gr.
Steven

Re: Should we remove java from the default bootstrap languages?

[Ralf Wildenhues]

* David Miller wrote on Sat, Jun 21, 2008 at 12:26:03AM CEST:
> 
> I used to be able to bootstrap gcc fully in minutes on average
> hardware 6 or so years ago.  Those days are long gone.  On my largest
> 64 cpu and 128 cpu boxes it takes forever these days.
> 
> The libjava build is notoriously not helped by parallelization because
> certain compiles are extremely expensive, which effectively
> single-threads the build.

Would that be compiles of object files that end up in libgcj (as opposed
to the link, or stuff that depends on libgcj)?  If yes, the lack of
parallelism should be fixable.

IIRC, then objects in libjava were built from lists of source files as a
means to avoid per-object overhead of libtool and some other stuff, and
to produce a bit better code[1].  Now, at least libtool compile mode
overhead should be a fair bit lower than back then (upstream is a bit
better, if that turns out to be significant, GCC could sync again).

I don't know whether the added debug info overhead, or slightly(?) worse
code generation, would be a show-stopper.  There should be no need to go
back to creating one object per source file, a reasonably fine-grained
split should be sufficient.  I don't have such a big machine to test on,
though, last time I looked, on 8-way things looked not too bad.

Have I overlooked something?  Has anybody ever measured the code
generation improvements that [1] brought?

Cheers,
Ralf

[1] <http://gcc.gnu.org/ml/java-patches/2005-q2/msg00429.html>

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Diego Novillo wrote:
> On Fri, Jun 20, 2008 at 16:56, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
> 
>> That aside, our current policy already allows e.g. not testing java if
>> your change is to a part of the compiler that can't possible affect it.
> 
> I didn't make it completely clear, but my suggestion was mostly to
> help us middle/back-end hackers.

Kaveh does have a point, Diego.  The libjava build regularly finds middle-end
problems that are not revealed by bootstrap testing.  So, even though you
may not think that the libjava build is useful to middle/back-end hackers,
it probably is.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Steven Bosscher wrote:
> On Sat, Jun 21, 2008 at 12:41 AM, Kaveh R. Ghazi <ghazi@caip.rutgers.edu> wrote:
>> Fundamentally, our philosophy has been to catch errors *before* they get
>> into the repository.  Sure one day of breaking the trunk isn't so bad, but
>> when it breaks it affects hundreds of developers and it adds up.
> 
> But, for languages that are not enabled by default, no-one is directly
> affected except the people who might have caused the breakage, and the
> people who are working on the broken part of the compiler.

They are directly affected by these errors, but they don't know: the middle-end
bugs revealed by the libjava build and testing are real bugs in other languages
too, they're just not detected by bootstrap & test.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Diego Novillo]

On Sat, Jun 21, 2008 at 05:21, Andrew Haley <aph@redhat.com> wrote:

> Kaveh does have a point, Diego.  The libjava build regularly finds middle-end
> problems that are not revealed by bootstrap testing.

So does Ada.  This is why I have offered keep building it on my nightly testers.

IME, bugs found during libjava have been also triggered during
libstdc++ and/or C.  Though several folks at the summit mentioned that
they had found bugs triggered only by libjava.

My point remains that libjava has become a serious problem in the
development cycle of GCC.  It takes roughly 3 hours on modern hardware
to finish a GCC bootstrap (with -j2).  A significant chunk of which is
taken by libjava.  If we could at least reduce the overhead by not
building all of it by default, it would be a huge win.


Diego.

Re: Should we remove java from the default bootstrap languages?

[Richard Kenner]

> IME, bugs found during libjava have been also triggered during
> libstdc++ and/or C.  Though several folks at the summit mentioned that
> they had found bugs triggered only by libjava.

To me, as I said, this is the key issue: how often do we have such bugs?

Re: Should we remove java from the default bootstrap languages?

[NightStrike]

On 6/21/08, Diego Novillo <dnovillo@google.com> wrote:
> My point remains that libjava has become a serious problem in the
> development cycle of GCC.  It takes roughly 3 hours on modern hardware
> to finish a GCC bootstrap (with -j2).  A significant chunk of which is
> taken by libjava.  If we could at least reduce the overhead by not
> building all of it by default, it would be a huge win.

What do you define as "modern hardware"?

Re: Should we remove java from the default bootstrap languages?

[Diego Novillo]

On Sat, Jun 21, 2008 at 11:27, NightStrike <nightstrike@gmail.com> wrote:

> What do you define as "modern hardware"?

Dell Precision 390 Core2 6600 @2.40Ghz.  4Gb RAM.  Fedora 8.


Diego.

Re: Should we remove java from the default bootstrap languages?

[H.J. Lu]

It takes about 50 minutes to bootstrap gcc with -j4 on a Core 2 Quad 2.66GHz
with default language, both 32bit and 64bit enabled. If I use
--enable-checking=assert,
it takes 25 minutes. Given the price of quad core today, there is no
reason no to
use quad core for gcc build.


H.J.
--
On Sat, Jun 21, 2008 at 8:30 AM, Diego Novillo <dnovillo@google.com> wrote:
> On Sat, Jun 21, 2008 at 11:27, NightStrike <nightstrike@gmail.com> wrote:
>
>> What do you define as "modern hardware"?
>
> Dell Precision 390 Core2 6600 @2.40Ghz.  4Gb RAM.  Fedora 8.
>
>
> Diego.
>



-- 
H.J.

Re: Should we remove java from the default bootstrap languages?

[Diego Novillo]

On Sat, Jun 21, 2008 at 11:39, H.J. Lu <hjl.tools@gmail.com> wrote:
> It takes about 50 minutes to bootstrap gcc with -j4 on a Core 2 Quad 2.66GHz
> with default language, both 32bit and 64bit enabled. If I use
> --enable-checking=assert,
> it takes 25 minutes. Given the price of quad core today, there is no
> reason no to
> use quad core for gcc build.

Irrelevant.


Diego.

Re: Should we remove java from the default bootstrap languages?

[Richard Guenther]

On Sat, Jun 21, 2008 at 12:50 AM, Richard Kenner
<kenner@vlsi1.ultra.nyu.edu> wrote:
>> Fundamentally, our philosophy has been to catch errors *before* they get
>> into the repository.  Sure one day of breaking the trunk isn't so bad, but
>> when it breaks it affects hundreds of developers and it adds up.  Everyone
>> separately either stops and waits, or tracks down which patch it was and
>> reverts it so they can continue working.
>
> An interesting question that I see as relevant here and for which I have no
> data is: what percentage of the time does a patch cause an error *only*
> in libjava?   I think you have to weigh the cost of the build of that
> library against the number of bugs that it finds.

Happened to me multiple times.

The thing to tackle is to make libjava build more parallel.

Richard.

Re: Should we remove java from the default bootstrap languages?

[David Miller]

From: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
Date: Sat, 21 Jun 2008 10:58:55 +0200

> Would that be compiles of object files that end up in libgcj (as opposed
> to the link, or stuff that depends on libgcj)?  If yes, the lack of
> parallelism should be fixable.

It's the compilation of the object files, not the linking.

Re: Should we remove java from the default bootstrap languages?

[NightStrike]

On 6/21/08, Diego Novillo <dnovillo@google.com> wrote:
> On Sat, Jun 21, 2008 at 11:39, H.J. Lu <hjl.tools@gmail.com> wrote:
> > It takes about 50 minutes to bootstrap gcc with -j4 on a Core 2 Quad 2.66GHz
> > with default language, both 32bit and 64bit enabled. If I use
> > --enable-checking=assert,
> > it takes 25 minutes. Given the price of quad core today, there is no
> > reason no to
> > use quad core for gcc build.
>
> Irrelevant.

How is that irrelevant?  If the argument is that libjava takes too
long to build on modern hardware, and someone else has a different
view of what is modern hardware where the original argument is
invalid... what makes your view "correct" and HJ's view "incorrect"?

For comparison, I build with -j12 on an 8 core machine in 4 minutes.

Re: Should we remove java from the default bootstrap languages?

[Laurent GUERBY]

On Sat, 2008-06-21 at 08:10 -0400, Diego Novillo wrote:
> On Sat, Jun 21, 2008 at 05:21, Andrew Haley <aph@redhat.com> wrote:
> 
> > Kaveh does have a point, Diego.  The libjava build regularly finds middle-end
> > problems that are not revealed by bootstrap testing.
> 
> So does Ada.  This is why I have offered keep building it on my nightly testers.
> 
> IME, bugs found during libjava have been also triggered during
> libstdc++ and/or C.  Though several folks at the summit mentioned that
> they had found bugs triggered only by libjava.
> 
> My point remains that libjava has become a serious problem in the
> development cycle of GCC.  It takes roughly 3 hours on modern hardware
> to finish a GCC bootstrap (with -j2).  A significant chunk of which is
> taken by libjava.  If we could at least reduce the overhead by not
> building all of it by default, it would be a huge win.

Some data: with --enable-languages=all,ada --disable-multilib on 4.3.1
release on a 4 Opteron 2.0 GHz cores machine from the compile farm, I
got for make bootstrap:

-j1 2h18
-j2 1h14
-j4 0h43
-j6 0h42

And for make -k check:

-j1 2h18
-j2 1h11
-j4 0h55
-j6 0h44

Without Ada (=all) bootstrap gets down to 1h54 at -j1 so Ada is 0h24, or
17% of =all, without Ada and Java (=c,c++,fortran,objc) it gets down to
1h17 at -j1 so java is 0h37 or 27% of =all.

On trunk at -j1 with the same configure my autotester for all,ada get
3h22 boostrap (+1h04, +46%) and 2h52 check (+0h34, +25%). It shows that
additional checking on trunk is not cheap (assuming we didn't
get much slower since 4.3).

The compile farm has three 8 cores machines but I didn't try recently
-j8 on them but it should be in the 20-25 minute range with all,ada:

http://gcc.gnu.org/wiki/CompileFarm

Laurent

Re: Should we remove java from the default bootstrap languages?

[Arnaud Charlet]

> Without Ada (=all) bootstrap gets down to 1h54 at -j1 so Ada is 0h24, or
> 17% of =all, without Ada and Java (=c,c++,fortran,objc) it gets down to
> 1h17 at -j1 so java is 0h37 or 27% of =all.

Note that I recently made the ada build parallel on trunk, so these figures
should be very different on trunk, since now building e.g. libada is
much faster with -jxx and will give a significant speed up.

Arno

Re: Should we remove java from the default bootstrap languages?

[Eric Botcazou]

> How is that irrelevant?  If the argument is that libjava takes too
> long to build on modern hardware, and someone else has a different
> view of what is modern hardware where the original argument is
> invalid... what makes your view "correct" and HJ's view "incorrect"?

Quad cores are available for exactly 1 architecture (x86) at affordable 
prices, GCC is supposed to support more than just x86.

-- 
Eric Botcazou

Re: Should we remove java from the default bootstrap languages?

[Laurent GUERBY]

On Sat, 2008-06-21 at 22:05 +0200, Eric Botcazou wrote:
> > How is that irrelevant?  If the argument is that libjava takes too
> > long to build on modern hardware, and someone else has a different
> > view of what is modern hardware where the original argument is
> > invalid... what makes your view "correct" and HJ's view "incorrect"?
> 
> Quad cores are available for exactly 1 architecture (x86) at affordable 
> prices, GCC is supposed to support more than just x86.

I'm curious at how many GCC developpers use non x86/_64 as their
main development machine (and how many non x86/_64 core they use).

Laurent

RE: Should we remove java from the default bootstrap languages?

[Dave Korn]

NightStrike wrote on 21 June 2008 17:08:

> On 6/21/08, Diego Novillo <dnovillo> wrote:
>> On Sat, Jun 21, 2008 at 11:39, H.J. Lu <hjl.tools> wrote:
>>> It takes about 50 minutes to bootstrap gcc with -j4 on a Core 2 Quad
>>> 2.66GHz with default language, both 32bit and 64bit enabled. If I use
>>> --enable-checking=assert, it takes 25 minutes. Given the price of quad
>>> core today, there is no 
>>> reason no to
>>> use quad core for gcc build.
>> 
>> Irrelevant.
> 
> How is that irrelevant?  If the argument is that libjava takes too
> long to build on modern hardware, and someone else has a different
> view of what is modern hardware where the original argument is
> invalid... 

  And how exactly does this help non-x86 platform users?  Or people without
the latest hardware?

  I don't think it aids the cause of software freedom to set a high barrier
to entry.  Nor do I think it helps recruit volunteer workers for GCC.  And I
really don't think it's right to just flippantly tell people they have to go
out and spend big money on the latest hardware in order to be able to
participate.  Volunteers and voluntary orgs often don't have budgets or
funding.


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: Should we remove java from the default bootstrap languages?

[Richard Guenther]

On Sat, Jun 21, 2008 at 10:21 PM, Dave Korn <dave.korn@artimi.com> wrote:
> NightStrike wrote on 21 June 2008 17:08:
>
>> On 6/21/08, Diego Novillo <dnovillo> wrote:
>>> On Sat, Jun 21, 2008 at 11:39, H.J. Lu <hjl.tools> wrote:
>>>> It takes about 50 minutes to bootstrap gcc with -j4 on a Core 2 Quad
>>>> 2.66GHz with default language, both 32bit and 64bit enabled. If I use
>>>> --enable-checking=assert, it takes 25 minutes. Given the price of quad
>>>> core today, there is no
>>>> reason no to
>>>> use quad core for gcc build.
>>>
>>> Irrelevant.
>>
>> How is that irrelevant?  If the argument is that libjava takes too
>> long to build on modern hardware, and someone else has a different
>> view of what is modern hardware where the original argument is
>> invalid...
>
>  And how exactly does this help non-x86 platform users?  Or people without
> the latest hardware?
>
>  I don't think it aids the cause of software freedom to set a high barrier
> to entry.  Nor do I think it helps recruit volunteer workers for GCC.  And I
> really don't think it's right to just flippantly tell people they have to go
> out and spend big money on the latest hardware in order to be able to
> participate.  Volunteers and voluntary orgs often don't have budgets or
> funding.

There is the compile-farm that is available to volunteers.

Richard.

Re: Should we remove java from the default bootstrap languages?

[Gerald Pfeifer]

On Sat, 21 Jun 2008, Laurent GUERBY wrote:
> I'm curious at how many GCC developpers use non x86/_64 as their
> main development machine (and how many non x86/_64 core they use).

It's not just backend changes that require testing on non-x86 
architectures.  And it is very worthwhile to get testing for
other changes on such architectures as well.

Gerald

Re: Should we remove java from the default bootstrap languages?

[Laurent GUERBY]

On Sat, 2008-06-21 at 21:03 +0200, Arnaud Charlet wrote:
> > Without Ada (=all) bootstrap gets down to 1h54 at -j1 so Ada is 0h24, or
> > 17% of =all, without Ada and Java (=c,c++,fortran,objc) it gets down to
> > 1h17 at -j1 so java is 0h37 or 27% of =all.
> 
> Note that I recently made the ada build parallel on trunk, so these figures
> should be very different on trunk, since now building e.g. libada is
> much faster with -jxx and will give a significant speed up.

More data on make bootstrap times:

baseline: gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
trunk revision 137004 Sat Jun 21 19:20:06 UTC 2008

-j4 =c
4671.57user 231.40system 22:37.84elapsed 361%CPU

-j4 =c,c++
5432.07user 293.57system 26:43.52elapsed 357%CPU

-j4 =c,ada
6819.21user 304.12system 32:08.14elapsed 369%CPU

-j4 =c,java (,c++ implied)
8909.32user 471.50system 45:33.95elapsed 343%CPU

So about estimated 1h59 for =c,ada at -j1 vs 1h22 for =c
and around 90% of expected speedup with or without Ada (according to
user+system vs elapsed). Ada cost of 0h37, +45% vs =c.

On the two 8 barcelona 2.2 GHz cores machine donated by AMD
to the GCC compile farm I measured:

-j8 =c
4115.01user 177.29system 12:50.72elapsed 556%CPU 

-j8 =c,fortran
4379.00user 205.48system 12:34.10elapsed 607%CPU 
(see below)

-j8 =c,c++
4632.48user 213.45system 14:04.17elapsed 574%CPU

-j8 =c,ada
5805.12user 227.32system 16:57.56elapsed 592%CPU 

-j8 =c,java (,c++ implied)
7845.79user 348.71system 23:45.03elapsed 575%CPU 

I noticed insn-attrtab compilation is about 5mn20s and probably
explain about all of the not so perfect speedup: when I look at top it
takes more than 1 minute per stage finishing alone. I've seen it up to 3
minutes alone. There's a comment in the Makefile about trying to
compile it earlier but it doesn't seem to work all the time
(likely reason behind c,fortran faster than =c)

Is there a plan to cut this one?

Laurent

Re: Should we remove java from the default bootstrap languages?

[Richard Kenner]

> > An interesting question that I see as relevant here and for which I have no
> > data is: what percentage of the time does a patch cause an error *only*
> > in libjava?   I think you have to weigh the cost of the build of that
> > library against the number of bugs that it finds.
> 
> Happened to me multiple times.

Is there a way to characterize the kinds of changes that are more likely
to make this happen?  We see this with Ada, for example: many middle-end
developers by now know which changes are most likely to potentially affect
Ada and do an Ada build too for those cases.  Would this sort of approach
work for Java as well?

Re: Should we remove java from the default bootstrap languages?

[David Miller]

From: Laurent GUERBY <laurent@guerby.net>
Date: Sat, 21 Jun 2008 22:12:24 +0200

> I'm curious at how many GCC developpers use non x86/_64 as their
> main development machine (and how many non x86/_64 core they use).

I definitely am one.

Or, maybe you are asking the wrong question, which likely
should be how many maintainers of other cpus backends are
in this situation :-)

Re: Should we remove java from the default bootstrap languages?

[Laurent GUERBY]

On Sun, 2008-06-22 at 00:04 -0700, David Miller wrote:
> From: Laurent GUERBY <laurent@guerby.net>
> Date: Sat, 21 Jun 2008 22:12:24 +0200
> 
> > I'm curious at how many GCC developpers use non x86/_64 as their
> > main development machine (and how many non x86/_64 core they use).
> 
> I definitely am one.

How many core does your main development machine have?

> Or, maybe you are asking the wrong question, which likely
> should be how many maintainers of other cpus backends are
> in this situation :-)

:)

Laurent

Re: Should we remove java from the default bootstrap languages?

[David Miller]

From: Laurent GUERBY <laurent@guerby.net>
Date: Sun, 22 Jun 2008 09:21:32 +0200

> On Sun, 2008-06-22 at 00:04 -0700, David Miller wrote:
> > From: Laurent GUERBY <laurent@guerby.net>
> > Date: Sat, 21 Jun 2008 22:12:24 +0200
> > 
> > > I'm curious at how many GCC developpers use non x86/_64 as their
> > > main development machine (and how many non x86/_64 core they use).
> > 
> > I definitely am one.
> 
> How many core does your main development machine have?

8 cores and 8 cpu threads per core on one, 64 cpus total.
16 cores and 8 cpu threads per core on another, 128 cpus total.

It still takes an hour or so to bootstrap on these machines because
the individual cpu threads are slow and individual expensive
compilations become the bottleneck, especially in the libjava build.

And the way the testsuite works it can't even come close to using
even a significant fraction of those cpus.

Re: Should we remove java from the default bootstrap languages?

[Laurent GUERBY]

On Sun, 2008-06-22 at 00:45 -0700, David Miller wrote:
> > How many core does your main development machine have?
> 
> 8 cores and 8 cpu threads per core on one, 64 cpus total.
> 16 cores and 8 cpu threads per core on another, 128 cpus total.
> 
> It still takes an hour or so to bootstrap on these machines because
> the individual cpu threads are slow and individual expensive
> compilations become the bottleneck, especially in the libjava build.

Did you measure how long insn-attrtab.o generation takes?
See my other email, on a 2.2 GHz barcelona it's about 5mn20s (for each
stage) and it's the main limiting factor (but I assume it's architecture
dependant).

> And the way the testsuite works it can't even come close to using
> even a significant fraction of those cpus.

Ada ACATS testsuite is pretty easy to parallelize, I've never done it
because it's not enabled by default so it doesn't affect many people.
For dejagnu I don't know but since it's never been done I assume it must
be hard to do.

Laurent

Re: Should we remove java from the default bootstrap languages?

[David Miller]

From: Laurent GUERBY <laurent@guerby.net>
Date: Sun, 22 Jun 2008 10:05:26 +0200

> On Sun, 2008-06-22 at 00:45 -0700, David Miller wrote:
> > > How many core does your main development machine have?
> > 
> > 8 cores and 8 cpu threads per core on one, 64 cpus total.
> > 16 cores and 8 cpu threads per core on another, 128 cpus total.
> > 
> > It still takes an hour or so to bootstrap on these machines because
> > the individual cpu threads are slow and individual expensive
> > compilations become the bottleneck, especially in the libjava build.
> 
> Did you measure how long insn-attrtab.o generation takes?
> See my other email, on a 2.2 GHz barcelona it's about 5mn20s (for each
> stage) and it's the main limiting factor (but I assume it's architecture
> dependant).

On Sparc, which is what these systems are and the cpu I target with
gcc, insn-attrtab.o compilation is not expensive.

Re: Should we remove java from the default bootstrap languages?

[Andrew Pinski]

Sent from my iPhone because I no longer have Internet access at home :(.

On Jun 22, 2008, at 0:21, Laurent GUERBY <laurent@guerby.net> wrote:

> On Sun, 2008-06-22 at 00:04 -0700, David Miller wrote:
>> From: Laurent GUERBY <laurent@guerby.net>
>> Date: Sat, 21 Jun 2008 22:12:24 +0200
>>
>>> I'm curious at how many GCC developpers use non x86/_64 as their
>>> main development machine (and how many non x86/_64 core they use).
>>
>> I definitely am one.
>
> How many core does your main development machine have?


One with only 512megs of memory. It does have two hardware thread  
though.

Thanks,
Andrew Pinski

Re: Should we remove java from the default bootstrap languages?

[Andrew Pinski]

Sent from my iPhone

On Jun 22, 2008, at 2:08, Andrew Pinski <pinskia@gmail.com> wrote:

> Sent from my iPhone because I no longer have Internet access at  
> home :(.
>
> On Jun 22, 2008, at 0:21, Laurent GUERBY <laurent@guerby.net> wrote:
>
>> On Sun, 2008-06-22 at 00:04 -0700, David Miller wrote:
>>> From: Laurent GUERBY <laurent@guerby.net>
>>> Date: Sat, 21 Jun 2008 22:12:24 +0200
>>>
>>>> I'm curious at how many GCC developpers use non x86/_64 as their
>>>> main development machine (and how many non x86/_64 core they use).
>>>
>>> I definitely am one.
>>
>> How many core does your main development machine have?
>
>
> One with only 512megs of memory. It does have two hardware thread  
> though.

I forgot to mention that this is a multilib target with three  
different multilibs. This makes building libjava more than 50% of the  
time. It takes around 18 hours to do a bootstrap/test on this machine.

-- Pinski

>
>
> Thanks,
> Andrew Pinski

Re: Should we remove java from the default bootstrap languages?

[NightStrike]

On 6/22/08, Andrew Pinski <pinskia@gmail.com> wrote:
> On Jun 22, 2008, at 2:08, Andrew Pinski <pinskia@gmail.com> wrote:
> > On Jun 22, 2008, at 0:21, Laurent GUERBY <laurent@guerby.net> wrote:
> > > On Sun, 2008-06-22 at 00:04 -0700, David Miller wrote:
> > > > From: Laurent GUERBY <laurent@guerby.net>
> > > > Date: Sat, 21 Jun 2008 22:12:24 +0200
> > > > > I'm curious at how many GCC developpers use non x86/_64 as their
> > > > > main development machine (and how many non x86/_64 core they use).
> > > > I definitely am one.
> > > How many core does your main development machine have?
> > One with only 512megs of memory. It does have two hardware thread though.
> I forgot to mention that this is a multilib target with three different
> multilibs. This makes building libjava more than 50% of the time. It takes
> around 18 hours to do a bootstrap/test on this machine.


You know, you guys don't realize how easy you have it.  For the Win64
testsuite, all I test is c,c++,fortran, and objc (the library
testsuites don't work at all).  It takes over three days to run!  18
hours would be wonderful in comparison.

Re: Should we remove java from the default bootstrap languages?

[Jakub Jelinek]

On Sat, Jun 21, 2008 at 08:58:05PM +0200, Laurent GUERBY wrote:
> On Sat, 2008-06-21 at 08:10 -0400, Diego Novillo wrote:
> > On Sat, Jun 21, 2008 at 05:21, Andrew Haley <aph@redhat.com> wrote:
> And for make -k check:
> 
> -j1 2h18
> -j2 1h11
> -j4 0h55
> -j6 0h44

For make check, it would perhaps help to split the langest runtest.exp
invocations (primarily check-gcc, maybe check-fortran and check-libstdc++-v3)
into two parts, so that make could schedule them concurrently with higher
make -jN -k check (e.g. for check-gcc run dg.exp in check-gcc-dg goal,
all the ohter *.exp's in check-gcc-other and check-gcc: check-gcc-dg check-gcc-other).

	Jakub

Re: Should we remove java from the default bootstrap languages?

[Richard Guenther]

On Sun, Jun 22, 2008 at 4:04 AM, Richard Kenner
<kenner@vlsi1.ultra.nyu.edu> wrote:
>> > An interesting question that I see as relevant here and for which I have no
>> > data is: what percentage of the time does a patch cause an error *only*
>> > in libjava?   I think you have to weigh the cost of the build of that
>> > library against the number of bugs that it finds.
>>
>> Happened to me multiple times.
>
> Is there a way to characterize the kinds of changes that are more likely
> to make this happen?  We see this with Ada, for example: many middle-end
> developers by now know which changes are most likely to potentially affect
> Ada and do an Ada build too for those cases.  Would this sort of approach
> work for Java as well?

libjava adds quite an amount of coverage on the general optimizers.  Most
of the time this was ICEs and miscompiles only happening for libjava but
not in the C/C++ testsuite.

Richard.

RE: Should we remove java from the default bootstrap languages?

[Dave Korn]

NightStrike wrote on 22 June 2008 12:57:

> On 6/22/08, Andrew Pinski <pinskia@gmail.com> wrote:
>> On Jun 22, 2008, at 2:08, Andrew Pinski <pinskia@gmail.com> wrote:
>>> On Jun 22, 2008, at 0:21, Laurent GUERBY <laurent@guerby.net> wrote:
>>>> On Sun, 2008-06-22 at 00:04 -0700, David Miller wrote:
>>>>> From: Laurent GUERBY <laurent@guerby.net>
>>>>> Date: Sat, 21 Jun 2008 22:12:24 +0200
>>>>>> I'm curious at how many GCC developpers use non x86/_64 as their
>>>>>> main development machine (and how many non x86/_64 core they use).
>>>>> I definitely am one.
>>>> How many core does your main development machine have?
>>> One with only 512megs of memory. It does have two hardware thread
>>> though. 
>> I forgot to mention that this is a multilib target with three different
>> multilibs. This makes building libjava more than 50% of the time. It
>> takes around 18 hours to do a bootstrap/test on this machine.
> 
> 
> You know, you guys don't realize how easy you have it.  For the Win64
> testsuite, all I test is c,c++,fortran, and objc (the library
> testsuites don't work at all).  It takes over three days to run!  18
> hours would be wonderful in comparison.


  Until very recently, my one and only main dev box was an 850MHz athlon
with 384Megs of memory on board.  Running full bootstraps and tests for
Cygwin could take a week.

  [  I've recently moved to a dual core athlon 64 with 4g ram.  It still
takes a couple of days but I haven't been using -j (ran into a build failure
the first time I tried and didn't have time to look into it) nor building
ada or libjava, so it's a lot smaller job.  ]

    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: Should we remove java from the default bootstrap languages?

[Ian Lance Taylor]

Jakub Jelinek <jakub@redhat.com> writes:

> On Sat, Jun 21, 2008 at 08:58:05PM +0200, Laurent GUERBY wrote:
>> On Sat, 2008-06-21 at 08:10 -0400, Diego Novillo wrote:
>> > On Sat, Jun 21, 2008 at 05:21, Andrew Haley <aph@redhat.com> wrote:
>> And for make -k check:
>> 
>> -j1 2h18
>> -j2 1h11
>> -j4 0h55
>> -j6 0h44
>
> For make check, it would perhaps help to split the langest runtest.exp
> invocations (primarily check-gcc, maybe check-fortran and check-libstdc++-v3)
> into two parts, so that make could schedule them concurrently with higher
> make -jN -k check (e.g. for check-gcc run dg.exp in check-gcc-dg goal,
> all the ohter *.exp's in check-gcc-other and check-gcc: check-gcc-dg check-gcc-other).

I think it would only be a few days of work for somebody familiar with
Tcl to add -j support to DejaGNU.  I think that would be a very useful
contribution to gcc development.

Ian

Re: Should we remove java from the default bootstrap languages?

[Ian Lance Taylor]

Laurent GUERBY <laurent@guerby.net> writes:

> I noticed insn-attrtab compilation is about 5mn20s and probably
> explain about all of the not so perfect speedup: when I look at top it
> takes more than 1 minute per stage finishing alone. I've seen it up to 3
> minutes alone. There's a comment in the Makefile about trying to
> compile it earlier but it doesn't seem to work all the time
> (likely reason behind c,fortran faster than =c)

First I'll note that insn-attrtab.c is particularly slow for
x86/x86_64, presumably due to the many processor varieties and complex
scheduling.  It is much faster for other targets.

Compiling it earlier than it would otherwise be does work.  The
problem is that insn-attrtab.o depends on the generated file
insn-attr.h.  GNU make works by making a pass over the targets, and it
builds all the ones whose dependencies are already built.  Then it
makes another pass.  The effect is that all the objects which do not
depend upon insn-attr.h wind up getting built before insn-attrtab.o.

We could game the system to force insn-attrtab.o to be compiler even
earlier by adding false dependencies in the Makefile.

A cleaner approach would be to somehow change genattrtab.c to generate
multiple files.  Or, in general, to somehow invert the sense of the
functions to work as a switch on ix86_tune, rather than testing it in
every single conditional.  I'm guessing, but I think that inverting
the function that way should significantly reduce the number of basic
blocks and speed up compilation.

Ian

Re: Should we remove java from the default bootstrap languages?

[NightStrike]

On 6/22/08, Ian Lance Taylor <iant@google.com> wrote:
> Jakub Jelinek <jakub@redhat.com> writes:
>
> > On Sat, Jun 21, 2008 at 08:58:05PM +0200, Laurent GUERBY wrote:
> >> On Sat, 2008-06-21 at 08:10 -0400, Diego Novillo wrote:
> >> > On Sat, Jun 21, 2008 at 05:21, Andrew Haley <aph@redhat.com> wrote:
> >> And for make -k check:
> >>
> >> -j1 2h18
> >> -j2 1h11
> >> -j4 0h55
> >> -j6 0h44
> >
> > For make check, it would perhaps help to split the langest runtest.exp
> > invocations (primarily check-gcc, maybe check-fortran and check-libstdc++-v3)
> > into two parts, so that make could schedule them concurrently with higher
> > make -jN -k check (e.g. for check-gcc run dg.exp in check-gcc-dg goal,
> > all the ohter *.exp's in check-gcc-other and check-gcc: check-gcc-dg check-gcc-other).
>
> I think it would only be a few days of work for somebody familiar with
> Tcl to add -j support to DejaGNU.  I think that would be a very useful
> contribution to gcc development.

I had looked into this when I first was confronted with the insane
amount of time required to test on a Windows platform (mostly due to
the overhead of ssh/sftp).  The biggest downside was that each
invocation of expect writes to a separate log file.

So for instance, if you invoke "make check-gcc" from the top level
makefile, this will spawn: check-gcc, check-fortran, check-objc,
check-c++, and check-libstdc++-v3 for the default Win64 target.  Each
make target has its own sum/log file, and make parallelizes at that
level.

If you were to have make break that up, you would have then an
additional set of sum/log files for every check-* make target.  If you
instead try to have expect break things up or parallelize a single
check-* target, you then have to deal with a sum file that is out of
order compared to any other run, and a log file at best that's just as
out-of-order, and at worst that has all of the output from the various
parallel tasks running into each other.


Now what you could do (though I don't know how "hackey" it is) would
be to run each .exp file separately and run a script to combine the
results after the fact and create a new table of results data and new
sum/log files.  I was working on doing that, and was deciding on awk
or perl as the language of choice to do such a thing.  We can already
specify via RUNTESTFLAGS a single exp file (and actually, a single
test in a single exp file) to run.  We could just do (pseudocode
example) expect file1.exp > out1.txt &; expect file2.exp > out2.txt &;
etc...

That would then allow multiple invocations of expect that would each
write to their own sum/logs.  Then at the end, you have to combine
them all so as to not be so untidy as to have tons of sum/log sets.
Combining them isn't a hugely difficult task.  It's just tedious when
you aren't that advanced at scripting.

Re: Should we remove java from the default bootstrap languages?

[Ralf Wildenhues]

* Ian Lance Taylor wrote on Sun, Jun 22, 2008 at 04:42:19PM CEST:
> 
> First I'll note that insn-attrtab.c is particularly slow for
> x86/x86_64, presumably due to the many processor varieties and complex
> scheduling.  It is much faster for other targets.
> 
> Compiling it earlier than it would otherwise be does work.  The
> problem is that insn-attrtab.o depends on the generated file
> insn-attr.h.  GNU make works by making a pass over the targets, and it
> builds all the ones whose dependencies are already built.  Then it
> makes another pass.  The effect is that all the objects which do not
> depend upon insn-attr.h wind up getting built before insn-attrtab.o.
> 
> We could game the system to force insn-attrtab.o to be compiler even
> earlier by adding false dependencies in the Makefile.

I would be surprised if just dependency ordering could improve things
for insn-attrtab on trunk further than this patch (a variant of which
was committed to trunk on 2008-04-03):
<http://gcc.gnu.org/ml/gcc-patches/2008-03/msg01888.html>

> A cleaner approach would be to somehow change genattrtab.c [...]

That sounds like the next logical step to me.

Cheers,
Ralf

Re: Should we remove java from the default bootstrap languages?

[Ralf Wildenhues]

* NightStrike wrote on Sun, Jun 22, 2008 at 06:14:22PM CEST:
> On 6/22/08, Ian Lance Taylor <iant@google.com> wrote:
> >
> > I think it would only be a few days of work for somebody familiar with
> > Tcl to add -j support to DejaGNU.  I think that would be a very useful
> > contribution to gcc development.

> Now what you could do (though I don't know how "hackey" it is) would
> be to run each .exp file separately and run a script to combine the
> results after the fact and create a new table of results data and new
> sum/log files.

Has anybody ever looked at using threading capabilities of tcl directly?
Parallel DejaGNU could benefit other packages too.  There is a thread
pools package (tpool.html, linked from <http://wiki.tcl.tk/2770>) but I
have no idea how functional it is in practice; likely the GCC testsuite
would need at least a bit of restructuring, too.

Cheers,
Ralf (who'd love to look into it but I doesn't have time ATM)

Re: Should we remove java from the default bootstrap languages?

[Tom Tromey]

>>>>> "Ralf" == Ralf Wildenhues <Ralf.Wildenhues@gmx.de> writes:

Ralf> Has anybody ever looked at using threading capabilities of tcl directly?

At least Fedora compiles the system Tcl without threading enabled.
This has been attempted a few times over the years but apparently
always reverted due to bugs.

Tom

Re: Should we remove java from the default bootstrap languages?

[Tom Tromey]

>>>>> "Richard" == Richard Kenner <kenner@vlsi1.ultra.nyu.edu> writes:

>> IME, bugs found during libjava have been also triggered during
>> libstdc++ and/or C.  Though several folks at the summit mentioned that
>> they had found bugs triggered only by libjava.

Richard> To me, as I said, this is the key issue: how often do we have
Richard> such bugs?

I don't know, but one option open to us is to try this approach and
then revert it if we see "too many" regressions.

Tom

Re: Should we remove java from the default bootstrap languages?

[Jeff Law]

Richard Guenther wrote:
> On Sat, Jun 21, 2008 at 12:50 AM, Richard Kenner
> <kenner@vlsi1.ultra.nyu.edu> wrote:
>>> Fundamentally, our philosophy has been to catch errors *before* they get
>>> into the repository.  Sure one day of breaking the trunk isn't so bad, but
>>> when it breaks it affects hundreds of developers and it adds up.  Everyone
>>> separately either stops and waits, or tracks down which patch it was and
>>> reverts it so they can continue working.
>> An interesting question that I see as relevant here and for which I have no
>> data is: what percentage of the time does a patch cause an error *only*
>> in libjava?   I think you have to weigh the cost of the build of that
>> library against the number of bugs that it finds.
> 
> Happened to me multiple times.
Similarly.

> 
> The thing to tackle is to make libjava build more parallel.
I think this applies across the board, including the testsuites.

jeff

Re: Should we remove java from the default bootstrap languages?

[Daniel Jacobowitz]

On Sun, Jun 22, 2008 at 07:13:03PM +0200, Ralf Wildenhues wrote:
> Has anybody ever looked at using threading capabilities of tcl directly?
> Parallel DejaGNU could benefit other packages too.  There is a thread
> pools package (tpool.html, linked from <http://wiki.tcl.tk/2770>) but I
> have no idea how functional it is in practice; likely the GCC testsuite
> would need at least a bit of restructuring, too.

[Insert QMTest plug here]

I don't know for sure whether the QMTest support in the testsuite is
still good... we use QMTest internally, but not for GCC at the moment.
I'd love to get it working again, at least for native.  That can
parallelize tests as fine-grained as you wish, and present consistent logs.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Should we remove java from the default bootstrap languages?

[Joseph S. Myers]

On Sun, 22 Jun 2008, Daniel Jacobowitz wrote:

> On Sun, Jun 22, 2008 at 07:13:03PM +0200, Ralf Wildenhues wrote:
> > Has anybody ever looked at using threading capabilities of tcl directly?
> > Parallel DejaGNU could benefit other packages too.  There is a thread
> > pools package (tpool.html, linked from <http://wiki.tcl.tk/2770>) but I
> > have no idea how functional it is in practice; likely the GCC testsuite
> > would need at least a bit of restructuring, too.
> 
> [Insert QMTest plug here]
> 
> I don't know for sure whether the QMTest support in the testsuite is
> still good... we use QMTest internally, but not for GCC at the moment.
> I'd love to get it working again, at least for native.  That can
> parallelize tests as fine-grained as you wish, and present consistent logs.

I'm pretty sure it's very bitrotten; it depends on an externally 
maintained emulation of testsuite logic (qmtest_gcc), and the testsuite 
has moved on a *long* way since in terms of custom Tcl code to control 
which tests apply on what systems, with what options (target-supports* 
etc.).  Unless people wish to put in the work to make it operational again 
for 4.4, I think the code in GCC should be removed (not deprecated, it's 
too far bitrotten for that).  (If people do wish to use this code long 
term, qmtest_gcc probably needs to be an integrated part of GCC, not a 
separate package.)

A cleanup I'd like to see, regardless of any move to QMTest, is making all 
testing installed testing (via installing in a temporary directory in the 
build tree).  The testsuite should not need to know how to tell the 
compiler to find bits of itself in a build tree, only "make install" 
should need to know how to put such bits together in an install tree so 
the compiler can find them without special options.  (Worse, core DejaGnu 
itself has hardcoded knowledge about how to locate bits of GCC source and 
build trees, some of it long obsolete.  It's unfortunate that qmtest_gcc 
has such logic as well, as needed when emulating the DejaGnu logic, but at 
least QMTest and qmtc don't.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Should we remove java from the default bootstrap languages?

[Andrew Pinski]

On Fri, Jun 20, 2008 at 1:56 PM, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
> Ugh, I think this is a terrible idea.  It took me all of zero days to find
> an example of libjava breaking when someone didn't test it:
>
> http://gcc.gnu.org/ml/gcc-patches/2008-06/msg01351.html

Actually that patch caused a bootstrap failure even before libjava was
compiled.  See http://gcc.gnu.org/ml/gcc-regression/2008-06/txt00007.txt
for the log.  Though it really depends on memory layout and other
factors when (and if) the build fails since it was an use after a
free.

So in summary that patch is not a good example of breaking libjava at all.

Thanks,
Andrew Pinski

Re: Should we remove java from the default bootstrap languages?

[Andrew Pinski]

On Fri, Jun 20, 2008 at 1:56 PM, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
> Ugh, I think this is a terrible idea.  It took me all of zero days to find
> an example of libjava breaking when someone didn't test it:
>
> http://gcc.gnu.org/ml/gcc-patches/2008-06/msg01351.html

This is a bad example for libjava breakage as it is an use after
freeing.  See http://gcc.gnu.org/ml/gcc-regression/2008-06/msg00013.html
and http://gcc.gnu.org/bugzilla/show_bug.cgi?id=36596.  The segfault
is very dependent on the version of glibc and kernel you are using,
etc.  It is very dependent on the actual memory layout of the
application.  Just it was failing during compiling for libjava, not
earlier like it was for Geoff's regression tester.

Thanks,
Andrew Pinski

Re: Should we remove java from the default bootstrap languages?

[Ben Elliston]

On Fri, 2008-06-20 at 17:05 -0400, Diego Novillo wrote:

> On Fri, Jun 20, 2008 at 16:56, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
> 
> > That aside, our current policy already allows e.g. not testing java if
> > your change is to a part of the compiler that can't possible affect it.
> 
> I didn't make it completely clear, but my suggestion was mostly to
> help us middle/back-end hackers.

One practice I've started using is to use two build trees.  One I use in
my day-to-day work with only the C and C++ front-ends.  I have another
build tree that has all languages enabled that is built and tested
overnight (before I commit).  This means my changes get tested better,
but without me incurring the build delays.

Ben

Re: Should we remove java from the default bootstrap languages?

[Ben Elliston]

On Sat, 2008-06-21 at 10:58 +0200, Ralf Wildenhues wrote:

> IIRC, then objects in libjava were built from lists of source files as a
> means to avoid per-object overhead of libtool and some other stuff, and
> to produce a bit better code[1].  Now, at least libtool compile mode
> overhead should be a fair bit lower than back then (upstream is a bit
> better, if that turns out to be significant, GCC could sync again).

A few years ago, I hacked the libjava Makefiles to eliminate the use of
libtool of Linux systems.  It cut 20 minutes from the build time on my
(at the time) modern hardware.

Ben

Re: Should we remove java from the default bootstrap languages?

[Ben Elliston]

On Sun, 2008-06-22 at 19:13 +0200, Ralf Wildenhues wrote:

> Has anybody ever looked at using threading capabilities of tcl directly?
> Parallel DejaGNU could benefit other packages too.  There is a thread
> pools package (tpool.html, linked from <http://wiki.tcl.tk/2770>) but I
> have no idea how functional it is in practice; likely the GCC testsuite
> would need at least a bit of restructuring, too.

Yes, I tried it about 5 years ago.

I used Tcl safe interpreters to run each .exp script in isolation.  It
immediately raised some problems, namely that you have to decide what
variables and procs should be available to the interpreter.  This is
nice, in that it makes you decide what the public interface between
DejaGnu and a test script should be.  Its failing is that DejaGnu does
not have well-defined interfaces.  ;-)

In principle, it should work.  It was just a lot more work than I
bargained for.  Keep in mind that GCC does not have a huge number
of .exp scripts.  Some (eg. execute.exp) run many individual tests and,
if one wanted to, these scripts could be parallelised.  This just means
that there won't be wins for other scripts or other packages using
DejaGnu.

Ben

Re: Should we remove java from the default bootstrap languages?

[Ben Elliston]

On Sun, 2008-06-22 at 07:32 -0700, Ian Lance Taylor wrote:

> I think it would only be a few days of work for somebody familiar with
> Tcl to add -j support to DejaGNU.  I think that would be a very useful
> contribution to gcc development.

What did you have in mind, Ian?  That DejaGnu would run .exps in
parallel?  Perhaps what I tried is more complicated than what you are
proposing.

Cheers, Ben

Re: Should we remove java from the default bootstrap languages?

[Kaveh R. GHAZI]

On Sat, 21 Jun 2008, Andrew Haley wrote:

> Steven Bosscher wrote:
> > On Sat, Jun 21, 2008 at 12:41 AM, Kaveh R. Ghazi <ghazi@caip.rutgers.edu> wrote:
> >> Fundamentally, our philosophy has been to catch errors *before* they get
> >> into the repository.  Sure one day of breaking the trunk isn't so bad, but
> >> when it breaks it affects hundreds of developers and it adds up.
> >
> > But, for languages that are not enabled by default, no-one is directly
> > affected except the people who might have caused the breakage, and the
> > people who are working on the broken part of the compiler.
>
> They are directly affected by these errors, but they don't know: the middle-end
> bugs revealed by the libjava build and testing are real bugs in other languages
> too, they're just not detected by bootstrap & test.
> Andrew.

Andrew,

I think you hit the heart of this argument.  The current bootstrap and
testsuite doesn't ensure complete code coverage testing for GCC.  Having
more languages, libraries and their associated testsuites uncovers more
bugs that would otherwise go undetected and remain hidden in the core
middle-end.

These hidden bugs could possible be triggered in C, C++ and/or fortran,
but our current testsuite may not trigger them by chance whereas java or
objc++ or whatever else we include might expose them during bootstrap.

The argument for switching off java (that it takes too long) could be
applied to C++ as well.  The libstdc++ testsuite takes longer for me than
the java one.  But I don't recommend eliminating that either.  How
widespread a language is used is not the determining factor in whether we
should activate it by default.  Many people have vouched for the fact that
java exposes middle-end bugs not uncovered by the other languages.  That
is where the worth of including it is found.

IMHO, having more languages and testsuites in the bootstrap process
enhances the quality of GCC.  I sympathize with the problems of regtest
duration.  I believe some of this could be addressed through making things
run in parallel better.

		--Kaveh
--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Should we remove java from the default bootstrap languages?

[NightStrike]

On 6/22/08, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
> IMHO, having more languages and testsuites in the bootstrap process
> enhances the quality of GCC.  I sympathize with the problems of regtest
> duration.  I believe some of this could be addressed through making things
> run in parallel better.

I think it could also be addressed with the gcc compile farm.  I
thought that there was some place where we could put patches, and they
would be automatically picked up and tested by some sort of automatic
scripts....  am I dreaming about that?

Re: Should we remove java from the default bootstrap languages?

[Laurent GUERBY]

On Sun, 2008-06-22 at 21:57 -0400, NightStrike wrote:
> On 6/22/08, Kaveh R. GHAZI <ghazi@caip.rutgers.edu> wrote:
> > IMHO, having more languages and testsuites in the bootstrap process
> > enhances the quality of GCC.  I sympathize with the problems of regtest
> > duration.  I believe some of this could be addressed through making things
> > run in parallel better.
> 
> I think it could also be addressed with the gcc compile farm.  I
> thought that there was some place where we could put patches, and they
> would be automatically picked up and tested by some sort of automatic
> scripts....  am I dreaming about that?

Sebastian already write a script to do that, see "Automatic bootstrap
and regression testing" on http://gcc.gnu.org/wiki/CompileFarm
(might not be up to date, Sebastian ?)

BTW testsuite timings data on compile farm 8 core barcelona:

-j8 check =c
1679.16user 568.21system 32:10.07elapsed 116%CPU

-j8 =c,ada check
5748.11user 224.91system 16:11.46elapsed 614%CPU

-j8 =c,c++ check
4445.12user 915.01system 33:12.87elapsed 268%CPU

-j8 =c,java (,c++ implied) check
4921.19user 1068.66system 34:20.50elapsed 290%CPU

Laurent

Re: Should we remove java from the default bootstrap languages?

[Ralf Wildenhues]

* Ben Elliston wrote on Mon, Jun 23, 2008 at 01:58:38AM CEST:
> On Sat, 2008-06-21 at 10:58 +0200, Ralf Wildenhues wrote:
> 
> > IIRC, then objects in libjava were built from lists of source files as a
> > means to avoid per-object overhead of libtool and some other stuff, and
> > to produce a bit better code[1].  Now, at least libtool compile mode
> > overhead should be a fair bit lower than back then (upstream is a bit
> > better, if that turns out to be significant, GCC could sync again).
> 
> A few years ago, I hacked the libjava Makefiles to eliminate the use of
> libtool of Linux systems.  It cut 20 minutes from the build time on my
> (at the time) modern hardware.

Current git Libtool compile mode is a lot faster than 1.5.x was.
Anyway its overhead in libjava is noise ATM, so that's not an issue.

Cheers,
Ralf

Re: Should we remove java from the default bootstrap languages?

[NightStrike]

On 6/23/08, Laurent GUERBY <laurent@guerby.net> wrote:
> > I think it could also be addressed with the gcc compile farm.  I
> > thought that there was some place where we could put patches, and they
> > would be automatically picked up and tested by some sort of automatic
> > scripts....  am I dreaming about that?
>
> Sebastian already write a script to do that, see "Automatic bootstrap
> and regression testing" on http://gcc.gnu.org/wiki/CompileFarm
> (might not be up to date, Sebastian ?)

Ok, maybe that can at least for the time being alleviate some of the concerns?

Re: Should we remove java from the default bootstrap languages?

[Ian Lance Taylor]

Ben Elliston <bje@au1.ibm.com> writes:

> On Sun, 2008-06-22 at 07:32 -0700, Ian Lance Taylor wrote:
>
>> I think it would only be a few days of work for somebody familiar with
>> Tcl to add -j support to DejaGNU.  I think that would be a very useful
>> contribution to gcc development.
>
> What did you have in mind, Ian?  That DejaGnu would run .exps in
> parallel?  Perhaps what I tried is more complicated than what you are
> proposing.

Running .exps in parallel wouldn't help much with the gcc testsuite.
What would help is for functions like c-torture-execute and dg-runtest
to run tests in parallel.  It seems to me that that could be done
fairly easily by maintaining a fixed number of concurrent executions,
and using system to execute them all in the background.  As one
execution finished, the script would start another one.  Each
execution would write to its own temporary log file.  The master
script would gather the log files and write them out in the
appropriate order.  This assumes that executing a single test does not
change any DejaGNU global variable; it might be necessary to change
code to ensure that.  This feature could not be used when executing
programs on a target board or when using DejaGNU to drive a compiler
running on a different host; however, that is not the common case.

Ian

Re: Should we remove java from the default bootstrap languages?

[Ian Lance Taylor]

Ian Lance Taylor <iant@google.com> writes:

> This feature could not be used when executing
> programs on a target board or when using DejaGNU to drive a compiler
> running on a different host; however, that is not the common case.

To be clear, when using an embedded target board, the compilations
could be parallelized, the executions (generally) could not be.  That
is not so bad, as the compilations are what take the most time; most
of the tests execute quite quickly.

Ian

Re: Should we remove java from the default bootstrap languages?

[Jonathan Wakely]

2008/6/23 NightStrike:
> On 6/23/08, Laurent GUERBY wrote:
>> > I think it could also be addressed with the gcc compile farm.  I
>> > thought that there was some place where we could put patches, and they
>> > would be automatically picked up and tested by some sort of automatic
>> > scripts....  am I dreaming about that?
>>
>> Sebastian already write a script to do that, see "Automatic bootstrap
>> and regression testing" on http://gcc.gnu.org/wiki/CompileFarm
>> (might not be up to date, Sebastian ?)
>
> Ok, maybe that can at least for the time being alleviate some of the concerns?
>

I tried using it last week and didn't have permission to write to
Sebastian's dir.

Jonathan

Re: Should we remove java from the default bootstrap languages?

[Mark Mitchell]

Andrew Haley wrote:

>>> But, I am actually ok with having it be disabled by default, provided
>>> that regressions affect gcj are treated seriously: fixed in a timely
>>> way by the person causing the regression, or, if not, letting gcj
>>> maintainers start the patch-reversion clock.
>>>
>>> If we make this change I'll set up an auto-tester on the compile farm
>>> that builds gcj along with everything else.  I think this would
>>> provide a pretty reasonable compromise. 

I agree.  I also agree that if someone breaks Java, they should be 
required to fix the problem.  In fact, we could have the rule that the 
Java maintainers get to revert a patch summarily based merely on the 
fact that there exists a Java post-patch failure that does not occur 
pre-patch.  That does shift the testing burden to the Java maintainers, 
but it also means that there is no risk that Java is completely hosed.

I am a huge fan of testing, but I do think that right now we're running 
too much testing for not enough return.  It's not that the testing is 
bad, or that more testing doesn't prevent bugs; it's that the marginal 
cost of bug-prevention from the Java testing seems too high to me, given 
that after-the-fact auto-testing can delivery much of the same value.

-- 
Mark Mitchell
CodeSourcery
mark@codesourcery.com
(650) 331-3385 x713

Re: Should we remove java from the default bootstrap languages?

[Diego Novillo]

On 6/26/08 12:06 AM, Mark Mitchell wrote:

> I am a huge fan of testing, but I do think that right now we're running 
> too much testing for not enough return.  It's not that the testing is 
> bad, or that more testing doesn't prevent bugs; it's that the marginal 
> cost of bug-prevention from the Java testing seems too high to me, given 
> that after-the-fact auto-testing can delivery much of the same value.

Agreed.


Diego.

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Mark Mitchell wrote:
> Andrew Haley wrote:
> 
>>>> But, I am actually ok with having it be disabled by default, provided
>>>> that regressions affect gcj are treated seriously: fixed in a timely
>>>> way by the person causing the regression, or, if not, letting gcj
>>>> maintainers start the patch-reversion clock.
>>>>
>>>> If we make this change I'll set up an auto-tester on the compile farm
>>>> that builds gcj along with everything else.  I think this would
>>>> provide a pretty reasonable compromise. 
> 
> I agree.  I also agree that if someone breaks Java, they should be
> required to fix the problem.  In fact, we could have the rule that the
> Java maintainers get to revert a patch summarily based merely on the
> fact that there exists a Java post-patch failure that does not occur
> pre-patch.

OK.  I'm hoping that the java mainatiners won't have _all_ the burden, though.

We should have a trial phase where java build breakage on the autobuilders
is mailed to the maintainers who checked in patches and to the java
maintainers, and we'll see how it goes.

I'm open-minded about this, but if it doesn't work we should be prepared to
revert the policy.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Mark Mitchell]

Andrew Haley wrote:

>> I agree.  I also agree that if someone breaks Java, they should be
>> required to fix the problem.  In fact, we could have the rule that the
>> Java maintainers get to revert a patch summarily based merely on the
>> fact that there exists a Java post-patch failure that does not occur
>> pre-patch.
> 
> OK.  I'm hoping that the java mainatiners won't have _all_ the burden, though.
> 
> We should have a trial phase where java build breakage on the autobuilders
> is mailed to the maintainers who checked in patches and to the java
> maintainers, and we'll see how it goes.
> 
> I'm open-minded about this, but if it doesn't work we should be prepared to
> revert the policy.

I think that's reasonable.  Perhaps a 30-day trial period, after the 
autobuilder is set up?  Then if we're seeing that the Java maintainers 
have had to beat people up a lot -- and particularly if that isn't 
yielding results -- then we revert?

To be clear, I have no special decision-making power here.  I'm hoping 
we can build a consensus to move in this direction, but it has to be a 
consensus decision.

-- 
Mark Mitchell
CodeSourcery
mark@codesourcery.com
(650) 331-3385 x713

Re: Should we remove java from the default bootstrap languages?

[Andrew Haley]

Mark Mitchell wrote:
> Andrew Haley wrote:
> 
>>> I agree.  I also agree that if someone breaks Java, they should be
>>> required to fix the problem.  In fact, we could have the rule that the
>>> Java maintainers get to revert a patch summarily based merely on the
>>> fact that there exists a Java post-patch failure that does not occur
>>> pre-patch.
>>
>> OK.  I'm hoping that the java mainatiners won't have _all_ the burden,
>> though.
>>
>> We should have a trial phase where java build breakage on the
>> autobuilders
>> is mailed to the maintainers who checked in patches and to the java
>> maintainers, and we'll see how it goes.
>>
>> I'm open-minded about this, but if it doesn't work we should be
>> prepared to
>> revert the policy.
> 
> I think that's reasonable.  Perhaps a 30-day trial period, after the
> autobuilder is set up?  Then if we're seeing that the Java maintainers
> have had to beat people up a lot -- and particularly if that isn't
> yielding results -- then we revert?

OK, but perhaps with a slightly longer trial period.  I'm not hung up on
that though.

> To be clear, I have no special decision-making power here.  I'm hoping
> we can build a consensus to move in this direction, but it has to be a
> consensus decision.

Understood.

Andrew.

Re: Should we remove java from the default bootstrap languages?

[Graham Stott]

All,

--- Andrew Haley <aph@redhat.com> wrote:

> Mark Mitchell wrote:
> > Andrew Haley wrote:
> > 
> >>> I agree.  I also agree that if someone breaks Java, they should be
> >>> required to fix the problem.  In fact, we could have the rule that the
> >>> Java maintainers get to revert a patch summarily based merely on the
> >>> fact that there exists a Java post-patch failure that does not occur
> >>> pre-patch.
> >>
> >> OK.  I'm hoping that the java mainatiners won't have _all_ the burden,
> >> though.
> >>
> >> We should have a trial phase where java build breakage on the
> >> autobuilders
> >> is mailed to the maintainers who checked in patches and to the java
> >> maintainers, and we'll see how it goes.
> >>
> >> I'm open-minded about this, but if it doesn't work we should be
> >> prepared to
> >> revert the policy.
> > 
> > I think that's reasonable.  Perhaps a 30-day trial period, after the
> > autobuilder is set up?  Then if we're seeing that the Java maintainers
> > have had to beat people up a lot -- and particularly if that isn't
> > yielding results -- then we revert?
> 
> OK, but perhaps with a slightly longer trial period.  I'm not hung up on
> that though.
> 
> > To be clear, I have no special decision-making power here.  I'm hoping
> > we can build a consensus to move in this direction, but it has to be a
> > consensus decision.
> 
> Understood.
> 
> Andrew.
> 

Maybe we could have the default depend on which stage of development the tree
is currently in. For example during stages 1 and 2 the default could be
disabled and during stage 3 it could be enabled. 

Cheers
Graham

Re: Should we remove java from the default bootstrap languages?

[Matthias Klose]

Andrew Haley writes:
> Mark Mitchell wrote:
> > Andrew Haley wrote:
> > 
> >>>> But, I am actually ok with having it be disabled by default, provided
> >>>> that regressions affect gcj are treated seriously: fixed in a timely
> >>>> way by the person causing the regression, or, if not, letting gcj
> >>>> maintainers start the patch-reversion clock.
> >>>>
> >>>> If we make this change I'll set up an auto-tester on the compile farm
> >>>> that builds gcj along with everything else.  I think this would
> >>>> provide a pretty reasonable compromise. 
> > 
> > I agree.  I also agree that if someone breaks Java, they should be
> > required to fix the problem.  In fact, we could have the rule that the
> > Java maintainers get to revert a patch summarily based merely on the
> > fact that there exists a Java post-patch failure that does not occur
> > pre-patch.
> 
> OK.  I'm hoping that the java mainatiners won't have _all_ the burden, though.
> 
> We should have a trial phase where java build breakage on the autobuilders
> is mailed to the maintainers who checked in patches and to the java
> maintainers, and we'll see how it goes.
> 
> I'm open-minded about this, but if it doesn't work we should be prepared to
> revert the policy.

would it be a possibility to reduce the cost of having java built by
default and keep java in the default bootstrap languages?

 - currently libjava is built as a static *and* as shared library, while
   the static library is not that useful. make it the default of not
   building the static library by default? Doesn't cut the build time
   of libjava by 50%, but should speed up the build considerably.

 - on platforms where multilib is the default, libjava is built as
   biarch as well; unfortunately the build will fail on machines
   which don't have all the build dependencies available for all build
   dependencies (e.g. gtk). Disable the multilib build by default to
   save 50% build time on those platforms.

Sorry for getting that late into the discussion. gcj is still
important for some os/architectures, where OpenJDK isn't yet an
alternative.  Could we delay this decision until after the 4.4 branch
is created?

  Matthias