Re: V3: SPARC bug and tree freeze

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

<<glibc is too big to use as a check-in criteria.  However, it should be
part of the release testing process, and nightly jobs to test
substantial programs (like glibc) are a Good Thing.
>>

Is that really true with modern fast computers. In the GNAT world, the
checkin criterion for even the tiniest "can't-possibly-affect-anything"
modification is to run the entire internal regression suite (about 7000
test cases, about 7 million lines of code). The way we do it is with
a robot that accepts the patch as a message and runs the suite 
automatically. It takes a few hours of running time on a fast machine.
Certainly testing glibc sounds like something that could be done quite
quickly.

Re: V3: SPARC bug and tree freeze

[Mark Mitchell]

>>>>> "Robert" == Robert Dewar <dewar@gnat.com> writes:

    Robert> <<glibc is too big to use as a check-in criteria.
    Robert> However, it should be part of the release testing process,
    Robert> and nightly jobs to test substantial programs (like glibc)
    Robert> are a Good Thing.
    >>>

    Robert> Is that really true with modern fast computers. In the

Yes.

We already run thousands of tests before checkin, including C, C++,
Fortran, and library tests.  That already takes hours for many people;
glibc takes about that same amount of time to build, even on a fast
machine.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: V3: SPARC bug and tree freeze

[Alexandre Oliva]

On Nov 12, 2000, dewar@gnat.com (Robert Dewar) wrote:

> The way we do it is with a robot that accepts the patch as a message
> and runs the suite automatically.

Is the technology you use for this available for public use?  I'd
certainly love to have something like this for some projects I happen
to be a maintainer of.  And I don't think it would be a bad idea to
have to in GCC either, even though we might need a very fast machine
to keep up with the volume of patches.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: V3: SPARC bug and tree freeze

[Rod Stewart]

On Sun, 12 Nov 2000, Robert Lipe wrote:
> Mark Mitchell wrote:
> 
> > system, but we can't ask people to build glibc locally before every
> > checkin; that's crippling.
> 
> Especially since glibc only meaningfully builds on a relatively small
> percentage of the targets represented by GCC...  Note that I didn't say
> for "small percentage of users", so please don't flame me [again] for
> reminding the crowd that the world has systems where glibc just doesn't
> exist.

And not all machines where glibc does build are in the gigaherz range.  I
autobuild four GNU projects nightly, on i386-linux, armv4l-linux, and
armv3l-linux.  The longest builds are the arm ones.  A full GCC, with
'make -k chek', takes just under 10 hours, and glibc takes just over 4
hours.

Not all of GNU/Linux is highend x86s, alphas, and sparcs.

Not to detract from the idea of more automated testing, but sometimes you
have to build on these 'slower' architectures as they have been known to
show up bugs which affect the the highend systems, but which currently do
not show up.

http://www.netwinder.org/build/stats/gcc.html

-Rms

Re: V3: SPARC bug and tree freeze

[Alexandre Oliva]

On Nov 12, 2000, Andreas Jaeger <aj@suse.de> wrote:

> What should I do in case of failures?  Send automatically an email to
> gcc-bugs?

gcc-regression would probably be the appropriate place.  You may want
to contact Geoff Keating, he's already got a set of scripts that does
this.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: V3: SPARC bug and tree freeze

[Andreas Jaeger]

>>>>> Mark Mitchell writes:

>>>>> "Andreas" == Andreas Jaeger <aj@suse.de> writes:
Andreas> I would really appreciate if testing glibc would be a
Andreas> checkin criteria - or done on a regular basis.

 > Everyone wants their favorite program to be a checkin criteria. :-)

 > glibc is too big to use as a check-in criteria.  However, it should be
 > part of the release testing process, and nightly jobs to test
 > substantial programs (like glibc) are a Good Thing.

I'll try tomorrow to setup a nightly (German time = CET) compilation
of GCC and build with it Glibc on GNU/Linux running on ia32 and alpha
platforms if that would be considered helpfull.

What should I do in case of failures?  Send automatically an email to
gcc-bugs?  I won't have time to check the results on a daily basis.

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj

Re: V3: SPARC bug and tree freeze

[Robert Lipe]

Mark Mitchell wrote:

> system, but we can't ask people to build glibc locally before every
> checkin; that's crippling.

Especially since glibc only meaningfully builds on a relatively small
percentage of the targets represented by GCC...  Note that I didn't say
for "small percentage of users", so please don't flame me [again] for
reminding the crowd that the world has systems where glibc just doesn't
exist.

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

<<No, the issue is about different architectures and that's a major difference
between GNAT and GCC.  For GNAT, it's relatively rare for a change to
cause failures on only one target, but for GCC it's quite common.  So if
we standardize on GNU/Linux on x86 as the test target and a test fails,
how can a developer whose environment is another target going to be able to
debug it?  That's why it's required that the test suite be run on the
developer's machine, so he can easily debug any failures.
>>

That's of course perfectly reasonable, and there is no reason to change
that policy. All I am suggesting is the capability for *additional* 
testing, that can't possibly make development more difficult! I am
certainly not suggesting that the remote testing on GNU/Linux would
substitute for local testing now. 

But note that running the test suite on your own machine does not in
anyway solve the problem of your patches breaking other machines!

What would seem reasonable is an additional gating rule that you
must not break GNU/Linux. Since this is the primary target of 
interest for the GNU community, that's a useful additional rule.

Re: V3: SPARC bug and tree freeze

[Richard Kenner]

    That's a completely independent question. Obviously anyone doing
    development must have the resources to debug problems, that's true
    now, and adding additional testing capability does not change the
    situation. The facilities for testing are simply to do testing. It is
    presumably the case now that people are not supposed to checkin
    patches that break things. All the effort of automating more extensive
    testing would be about is making sure that it is easier to follow this
    rule!

No, the issue is about different architectures and that's a major difference
between GNAT and GCC.  For GNAT, it's relatively rare for a change to
cause failures on only one target, but for GCC it's quite common.  So if
we standardize on GNU/Linux on x86 as the test target and a test fails,
how can a developer whose environment is another target going to be able to
debug it?  That's why it's required that the test suite be run on the
developer's machine, so he can easily debug any failures.

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

(I am resending the message because somehow headers got messed up)

<<What, with such facilities, is the procedure for developers without such a
machine themselves to debug problems automated testing shows up?  (My
understanding is that the existing regression checker deliberately uses a
simulator target, so that any GCC developer can build for that environment
to investigate problems.)
>>

That's a completely independent question. Obviously anyone doing development
must have the resources to debug problems, that's true now, and adding
additional testing capability does not change the situation. The facilities
for testing are simply to do testing. It is presumably the case now that
people are not supposed to checkin patches that break things. All the
effort of automating more extensive testing would be about is making sure
that it is easier to follow this rule!

Re: V3: SPARC bug and tree freeze

[Joseph S. Myers]

On Sun, 12 Nov 2000, Robert Dewar wrote:

> 2. Running a complex set of tests is not such an easy task and it is
> easy to make a mistake doing it, or to fail to notice a discrepancy.

Running the GCC testsuite is trivial.  I do something along the following
lines (with a couple of trivial scripts, and optimizing the first step
away if I already have the test logs for unmodified GCC and haven't
updated my GCC tree since running those tests):

# Build unmodified GCC and run make -k check
mv `find . -name '*.sum'` ..
# Build patched GCC and run make -k check
diff ../libstdc++.sum i686-pc-linux-gnu/libstdc++/testsuite/libstdc++.sum
diff ../libio.sum i686-pc-linux-gnu/libio/testsuite/libio.sum
diff ../g++.sum gcc/testsuite/g++.sum
diff ../g77.sum gcc/testsuite/g77.sum
diff ../objc.sum gcc/testsuite/objc.sum
diff ../gcc.sum gcc/testsuite/gcc.sum

If adding or changing testcases as part of the patch, I compare gcc.log as
well to make sure that the output is exactly as intended.

I hope that any Ada testsuites that get included in GCC will similarly
provide .sum or similar summary files that can meaningfully be compared
with diff (with any regressions obvious, and with any non-regression
differences minimal).

> As for "who will provide the machines", I will immediately make a
> commitment that if something like this is set up, ACT will contribute
> a top of the line PC for the task (we may be able to do more, but I
> have to consult with other folks before offering too much on my own
> since we have limited resources :-)

What, with such facilities, is the procedure for developers without such a
machine themselves to debug problems automated testing shows up?  (My
understanding is that the existing regression checker deliberately uses a
simulator target, so that any GCC developer can build for that environment
to investigate problems.)

-- 
Joseph S. Myers
jsm28@cam.ac.uk

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

<<I doesn't seem important to me to have these machines in a single
location.  In fact, having them spread over the world (or at least
over the U.S.) is probably a good thing.  So we shouldn't need to
worry about a single company taking over the whole patch validation
process.  If each company manages each machine in-house, we don't even
have to worry about connectivity costs: that's up to the contributor
of the machine to work out.
>>

I see why you feel this is an advantage, but in practice I think it would
make the administration of the operation unmanagably complex.

Re: V3: SPARC bug and tree freeze

[Alexandre Oliva]

On Nov 12, 2000, dewar@gnat.com (Robert Dewar) wrote:

> As for "who will provide the machines", I will immediately make a
> commitment that if something like this is set up, ACT will contribute
> a top of the line PC for the task

I think we can get enough machines from companies involved in
supporting GCC.  Red Hat has already kind-of contributed the machine
that runs the regression tester, and I believe other companies may
also want to get into the loop.

I doesn't seem important to me to have these machines in a single
location.  In fact, having them spread over the world (or at least
over the U.S.) is probably a good thing.  So we shouldn't need to
worry about a single company taking over the whole patch validation
process.  If each company manages each machine in-house, we don't even
have to worry about connectivity costs: that's up to the contributor
of the machine to work out.

In fact, if we're lucky, we may even get contributions from hardware
suppliers of machines with various different architectures, so that
GCC can be automatically tested on them.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

By the way, in our environment at ACT, one of the major tasks is keeping
the testing machines up and running. We run the test suite frequently
(it has been run nearly ten thousand times in the last couple of years
at ACT), and the frequency of mysterious gremlins that cause machines
to conk out for one reason or another is definitely nonzero. 

One thing that would seem reasonable to me here is to set up the testing
just for GNU/Linux. A huge amount of the complexity we face at ACT comes
from maintaining a heterogenous network with all kinds of different
machines on it, a clean farm containing only identical GNU/Linux based
machines could be much more reliable (for example one of our major
sources of difficulties is NFS reliability between machines of greatly
different speeds and architectures). In practice if the tree is kept
reliable on GNU/Linux then

a) in terms of the GNU project that is the main task in any case

b) if it is clean and realiable for GNU/Linux, then there is a good
starting point for cleanups on other targets.

(in particular, we have had a heck of a time keeping automated builds
on NT working well, and have not so far managed to set up an NT machine
for checkin testing that worked reliably enough).

We are definitely willing to contribute code, experience and material
resources to such a project.

Robert Dewar

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

<<Sure, we could do the mozilla-like server farm automated
test-before-checkin thing.  Who's going to provide those machines?
How do we avoid the political issues that are arising out of
gcc.gnu.org's current affiliation?  I have no objection to such a
system, but we can't ask people to build glibc locally before every
checkin; that's crippling.
>>

I absolutely agree, furthermore, relying on people to run tests locally
just isn't reliable enough for two reasons:

1. Unless you really automate the procedures, people will still operate
in the "this can't possibly affect anything mode" and skip the testing.

2. Running a complex set of tests is not such an easy task and it is
easy to make a mistake doing it, or to fail to notice a discrepancy.

As for "who will provide the machines", I will immediately make a
commitment that if something like this is set up, ACT will contribute
a top of the line PC for the task (we may be able to do more, but I
have to consult with other folks before offering too much on my own
since we have limited resources :-)

Setting the whole thing up is by far from trivial, but I think it would
be worth discussing what might be involved.

As far as "political issues arising out of ..." surely this is the
sort of thing that the steering committee can discuss and deal with?

Robert Dewar

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

Note incidentally that we definitely want to set something like this up
for the ACATS tests for GNU Ada. We are also doing two other things to
help the testing situation with GNU Ada once it is in the tree:

1. We are contributing a suite of purpose written tests that we wrote
to test various components of the GNAT system, particularly library
units that we have added.

2. We are implementing a new policy that bug reports from unsupported
users of the public version must come with a disclaimer that allows
the code to be added to the public test suite. That seems only fair!

Robert Dewar

P.S. We are working away on getting to the point of releasing the tree.
We have to get things to a reasonably stable state so that the further
maintenance of the tree can be done without any massive patches. We are
just about to do a code freeze internally for our next release (3.14)
of GNAT, so that's an ideal opportunity to move the tree out.

Re: V3: SPARC bug and tree freeze

[Mark Mitchell]

>>>>> "Robert" == Robert Dewar <dewar@gnat.com> writes:

    Robert> <<We already run thousands of tests before checkin,
    Robert> including C, C++, Fortran, and library tests.  That
    Robert> already takes hours for many people; glibc takes about
    Robert> that same amount of time to build, even on a fast machine.
    >>>

    Robert> But couldn't the needed testing be distributed on several
    Robert> machines with appropriate robots, and run in parallel. I
    Robert> know I am coming late to this discussion, but it does not
    Robert> seem workable for the tree to get this broken to me.

Sure, we could do the mozilla-like server farm automated
test-before-checkin thing.  Who's going to provide those machines?
How do we avoid the political issues that are arising out of
gcc.gnu.org's current affiliation?  I have no objection to such a
system, but we can't ask people to build glibc locally before every
checkin; that's crippling.

Also, I don't think the tree is that badly broken -- failing to
compile glibc could indicate one failure, or many.  I suspect a very
few.

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

<<Is the technology you use for this available for public use?  I'd
certainly love to have something like this for some projects I happen
to be a maintainer of.  And I don't think it would be a bad idea to
have to in GCC either, even though we might need a very fast machine
to keep up with the volume of patches.
>>

Potentially yes, it is a bunch of scripts that would need quite a bit
of massaging for general use. This might be something that Laurent
Guerby could help with. What we do at ACT is to have about ten machines
with various architectures available to run the tests. Most patches
are tested on just one machine (submitters choice), but occasionally
where you know there are target dependent issues, you can run on several
machines simultaneously. Then every night, we run on all machines to
make sure that

a) there is no case of two patches conflicting (very rare)

b) there is no case of a patch causing some kind of target dependent
regression (not so rare, happens perhaps one out of ten nights).

"a very fast machine"

First, you can get a fully configured gigaherz machine these days for
under $2000 that can chew things pretty fast.

Second, the "a" here can definitely be changed to "several". 

I do not want to minimize the effort in setting up something like this.
The scripts involved are complex, and it is hard work to maintain the
test suite in a form suitable for the automatic scripts, but we have
found this approach invaluable in avoiding regresssions. 

Robert

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

<<We already run thousands of tests before checkin, including C, C++,
Fortran, and library tests.  That already takes hours for many people;
glibc takes about that same amount of time to build, even on a fast
machine.
>>

But couldn't the needed testing be distributed on several machines
with appropriate robots, and run in parallel. I know I am coming
late to this discussion, but it does not seem workable for the tree
to get this broken to me.

I must say I am a bit surprised that glibc should take hours to build
on a fast machine -- is that really the case -- gigaherz processors
chew through stuff pretty quickly :-)

Re: V3: SPARC bug and tree freeze

[Mark Mitchell]

>>>>> "Andreas" == Andreas Jaeger <aj@suse.de> writes:

    Andreas> I would really appreciate if testing glibc would be a
    Andreas> checkin criteria - or done on a regular basis.

Everyone wants their favorite program to be a checkin criteria. :-)

glibc is too big to use as a check-in criteria.  However, it should be
part of the release testing process, and nightly jobs to test
substantial programs (like glibc) are a Good Thing.

I have already made good progress tracking down the bug, and expect to
post a patch soon.  I believe the problem to be a latent bug in
delete_computation, exposed by Bernd turning on JUMP_NOOP_MOVES after
reload.  More soon...

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Re: V3: SPARC bug and tree freeze

[Andreas Jaeger]

Mark, just for your information: The current GCC CVS tree is since
around four weeks broken, it doesn't compile glibc at all.  Even if I
work around PR 771 (remove -g from CFLAGS)[1], glibc doesn't
bootstrap.  It looks like glibc gots miscompiled.  So far I haven't
been able to track this down.

I would really appreciate if testing glibc would be a checkin criteria
- or done on a regular basis.

Andreas


Footnotes: 
[1]  but this might be fixed by a patch Neil just sent me

-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj

V3: SPARC bug and tree freeze

[Mark Mitchell]

[ NOTE: TREE FROZEN -- READ BELOW. ]

I returned to find the SPARC bootstrap stuck while building V3.  This
must be a recently introduced bug, since it worked a day or two back.

The code that cause the problem can be boiled down to:

  int main ()
  {
    long long i = 1;
    long long j = 0;

    while (i > j) {
      j = i;
      i = i * 2 + 1;
    }
  }

This code, when compiled with the newly build C compiler, will not
terminate.  Here is the body of the loop:

	ldd	[%fp-24], %i0
	std	%i0, [%fp-32]
	ldd	[%fp-24], %i2
	srl	%i3, 31, %i5
	sll	%i2, 1, %i4
	addcc	%i1, 1, %i1
	addx	%i0, 0, %i0
	std	%i0, [%fp-24]
	b	.LL3

The first two instructions assign i to j.

Then, we try to do the multiply and add.  The computation places
values into %i5 and %i4, but those values are then ignored.

I am going to try to track this down, but I would appreciate help from
anyone who might recognize the cause of this.

I don't want to switch to V3 until we can verify Solaris, but I also
don't want to risk instability that makes it harder to switch.  We
need to make the switch -- it's a very important milestone.

Therefore, until this bug is fixed, please consider the tree
completely frozen, except for changes to the Java, Objective-C, and
Fortran front-ends, and back-ends other than SPARC, MIPS, and x86.  In
other words, let's have no changes to code that could affect the MIPS
IRIX, SPARC Solaris, or x86 GNU/Linux C or C++ front-ends, or
associated libraries.

I expect this to be a very brief freeze, as I think we can track this
bug down quickly.

Thank you for your patience,

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com