Re: V3: SPARC bug and tree freeze

Re: V3: SPARC bug and tree freeze

[Robert Dewar]

<<Is the technology you use for this available for public use?  I'd
certainly love to have something like this for some projects I happen
to be a maintainer of.  And I don't think it would be a bad idea to
have to in GCC either, even though we might need a very fast machine
to keep up with the volume of patches.
>>

Potentially yes, it is a bunch of scripts that would need quite a bit
of massaging for general use. This might be something that Laurent
Guerby could help with. What we do at ACT is to have about ten machines
with various architectures available to run the tests. Most patches
are tested on just one machine (submitters choice), but occasionally
where you know there are target dependent issues, you can run on several
machines simultaneously. Then every night, we run on all machines to
make sure that

a) there is no case of two patches conflicting (very rare)

b) there is no case of a patch causing some kind of target dependent
regression (not so rare, happens perhaps one out of ten nights).

"a very fast machine"

First, you can get a fully configured gigaherz machine these days for
under $2000 that can chew things pretty fast.

Second, the "a" here can definitely be changed to "several". 

I do not want to minimize the effort in setting up something like this.
The scripts involved are complex, and it is hard work to maintain the
test suite in a form suitable for the automatic scripts, but we have
found this approach invaluable in avoiding regresssions. 

Robert

Automated testing framework (was: V3: SPARC bug and tree freeze)

[Laurent Guerby]

Robert Dewar wrote:
> Potentially yes, it is a bunch of scripts that would need quite a bit
> of massaging for general use. This might be something that Laurent
> Guerby could help with.

I would be happy to do so. 

For the curious, the ACT scripts do roughly as follows (well at least
2 years ago when I worked on them ;-):

1. Checkout and package the nigthly source tarball (at some fixed
hour) on one machine.

2. On various architectures, completely build and user install GNAT
from the source tarball and using a stable compiler).

3. Run ACATS and internal test suite to provide test result baseline
and send a detailed report of nightly changes and failures.

4. On various architectures a server wait for patches, apply the
patches, rebuild and install the compiler, build and run the internal
test suite, produces a report of changes against the nightly baseline,
and goes back to clean state. The server interface is email based, and
requests are queued.

Note that since the system starts out from fresh sources, it usually
is able to catch build stuff regression (missing files, wrong
Makefile dependancies, etc...).

I didn't look too closely at the current GCC testing framework (seems
like dejagnus is a huge beast, but embedded build and run annotations
for it look simple to grok anyway - good design ;-), and I don't know
what C/C++/other language packages are reasonable candidates for
filling the test suite (ie they have lots of testing executables).

All this stuff is not rocket science, but is definitely a big plus to
have and even more so if a project reaches a point when 100% pass is
required before commit.

News on the Ada testing front:

- Thanks to ACT contribution of configuration files, I'm now able to
build and run 2361 executable ACATS tests with the latest public GNAT
release (GCC 2.8.1 based), 16 executable tests are missing, mostly
distributed tests that have exotic run requirements (total build and
run time is 45 minutes on a P2 350MHz).

- The ACES performance testing suite packaging effort is frozen
pending copyright investigation (the government agency that contracted
the thing no longer exists).

- ACT is working on selecting and fixing the copyright/license on a
subset of their internal test, and will provide them to me for
packaging.

This means that the GNAT testing framework should be there in time
when the GNAT sources commit happens. I hope it can be used by people
to build and test GNAT on platforms ACT doesn't support yet like
GNU/Linux on Alpha, SPARC and PowerPC.

BTW, could someone send me the legal form(s) for GCC contribution
copyright assignement? I'm in working in France for a bank, have no
obnoxious clause in my contract, and work on the GNU project on my
free time with my home machine and email (both completely separated
from my day work envrionment).

-- 
Laurent Guerby <guerby@acm.org>

Re: Automated testing framework (was: V3: SPARC bug and tree freeze)

[Alexandre Oliva]

On Nov 12, 2000, Laurent Guerby <guerby@acm.org> wrote:

> BTW, could someone send me the legal form(s) for GCC contribution
> copyright assignement? I'm in working in France for a bank, have no
> obnoxious clause in my contract, and work on the GNU project on my
> free time with my home machine and email (both completely separated
> from my day work envrionment).

Please send this paragraph to rms@gnu.org and he'll send you the
appropriate forms.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: Automated testing framework (was: V3: SPARC bug and tree freeze)

[Laurent Guerby]

> Please send this paragraph to rms@gnu.org and he'll send you the
> appropriate forms.

Okay, done. BTW < http://gcc.gnu.org/contribute.html > says:

> Copyright Assignment
> 
> Before we can accept code contributions from you, we need a copyright
> assignment form filled out and filed with the FSF.
> 
> See some documentation by the FSF for details and contact us (either
> via the gcc@gcc.gnu.org list or the GCC maintainer that is taking care
> of your contributions) to obtain the relevant forms.

And the FSF documentation talks about files available on GNU machines,
and to request these forms to people on the project. I do not mind
playing email ping pong, but may be some information needs updating
;-).

-- 
Laurent Guerby <guerby@acm.org>

Re: Automated testing framework (was: V3: SPARC bug and tree freeze)

[Alexandre Oliva]

On Nov 12, 2000, Laurent Guerby <guerby@acm.org> wrote:

>> Please send this paragraph to rms@gnu.org and he'll send you the
>> appropriate forms.

> Okay, done.

Thanks.

> BTW < http://gcc.gnu.org/contribute.html > says:

We used to have the forms on-line, but RMS asked us to remove them.  A
few days ago, someone posted a message asking for them here, and
someone told him to ask RMS, so I though I could do the same :-)

> I do not mind
> playing email ping pong

:-)

> but may be some information needs updating ;-).

I don't understand why RMS doesn't want the forms on-line.  I thought
that was a very good thing, and I often pointed people who wanted to
contribute to GNU projects other than GCC at the forms available at
the GCC web-site.

Richard, would you mind enlightening us on the rationale for not
having the forms on-line?  Should I direct other GCC contributors
directly to you, or wait for someone from the GCC Steering Committee
to do so?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: Automated testing framework (was: V3: SPARC bug and tree freeze)

[Richard Stallman]

    >> Please send this paragraph to rms@gnu.org and he'll send you the
    >> appropriate forms.

Please don't ask me, ask the GCC maintainers.  Telling contributors
what forms to use is part of a package maintainer's job.

What forms to use (and whether papers are needed at all) depends on
what the contribution consists of.  Since the package maintainers have
to read and study the contribution anyway to decide whether they want
to use it, once they know the criteria for papers they can very
efficiently also tell the contributor what we need as regards forms.

Posting the criteria proved to be a bad idea, because a fair number of
people don't fully understand them at first reading.  Asking each
contributor to individually apply them to his own case means that most
people only do this once.  That has led to frequent mistakes.  They
were posted in the GCC web pages, but sometimes people working on
other projects tried to use them, so the mistakes affected other
packages too.

It is much more reliable for the maintainer to do this, because the
number of GNU package maintainers is much smaller; once a maintainer
understands the criteria (by asking me questions if necessary), he can
do the job easily and reliably.

To have me to this job for all GNU packages would be even more
reliable--but I don't have time to do it.  (And I would have to look
at each contribution in order to decide.)  When this task is divided
among the various package maintainers, it is a small job for each of
them, compared with the work of maintaining the package.  It takes
much less time to write a message saying what papers we need for a
change than to study the change and understand it.

Re: Automated testing framework (was: V3: SPARC bug and tree freeze)

[law]

  In message < 200011140722.AAA22852@wijiji.santafe.edu >you write:
  >     >> Please send this paragraph to rms@gnu.org and he'll send you the
  >     >> appropriate forms.
  > 
  > Please don't ask me, ask the GCC maintainers.  Telling contributors
  > what forms to use is part of a package maintainer's job.
Actually, since you refuse to publish this kind of stuff, I think it
is quite reasonable to send this stuff to you.  You might then realize
that it does take a significant amount of time and that most of the
work can be eliminated by simply publishing the forms and instructions.

If people were making mistakes with the previous online instructions or
forms, then that means those instructions need to be clarified.

jeff

Re: Automated testing framework (was: V3: SPARC bug and tree freeze)

[Joseph S. Myers]

On Tue, 14 Nov 2000, Richard Stallman wrote:

>     >> Please send this paragraph to rms@gnu.org and he'll send you the
>     >> appropriate forms.
>
> Please don't ask me, ask the GCC maintainers.  Telling contributors
> what forms to use is part of a package maintainer's job.

There still needs to be a system for all 89 GCC maintainers (the count of
distinct people in MAINTAINERS, including those marked as being people who
ought to have accounts for CVS write access but don't yet) to have the
forms for sending to contributors, better than everyone individually
getting them from assign@gnu.org.

> It is much more reliable for the maintainer to do this, because the
> number of GNU package maintainers is much smaller; once a maintainer
> understands the criteria (by asking me questions if necessary), he can
> do the job easily and reliably.

Here are some questions:

Does assign.future also cover changes to the GCC manual?  I've been told
in July that

	> I have on file an assign.future for "GNU CC".
	>
	> If further assignements are needed for changes to manuals / web pages /
	> testsuite / ..., then let me know what to put on what assignment forms
	> (and add some explanation to contribute.html).

	It's definitely sufficient for the manuals and also the web pages; I am
	not completely sure about the testsuite -- Jeff is our assignment expert,
	I believe...

but the GNU maintainers instructions say that

   You do not need to ask for separate papers for a manual that is
   distributed only in the software package it describes. But if we
   sometimes distribute the manual separately (for instance, if we
   publish it as a book), then we need separate legal papers for changes
   in the manual. For smaller changes, use
   `/gd/gnuorg/disclaim.changes.manual'; for larger ones, use
   `/gd/gnuorg/assign.changes.manual'. To cover both past and future
   changes to a manual, you can use `/gd/gnuorg/assign.future.manual'.

and I thought that the FSF do publish the GCC manual as a book.

Does assign.future cover the web pages?  Does it cover the testsuite?
What exactly is the position on whether an assign.future for "GNU CC" or
"GCC" (assigning "... Developer's copyright in changes and/or enhancements
to the program <name of program> ...") covers all programs, libraries,
manuals and other files in the GNU Compiler Collection (GCC)?  If the
testsuite is covered, should all testcases created afresh for the GCC
testsuite (rather than code fragments from bug reports from the net) have
copyright notices included?  Should they have a licence notice included?

The instructions for maintainers say that

   The list of year numbers should include each year in which you
   finished preparing a version which was actually released, and which
   was an ancestor of the current version.

How does this apply with anoncvs access to current development?  The GCC
practice is that the first change made to any file in a year requires the
copyright dates to be updated (whether or not there is a release finished
that year).  Is that correct?

If file F1 has copyright dates D1, and file F2 has copyright dates D2,
and, in year Y, I copy or move code from F1 to F2, should the copyright
dates on F1 be added to those on F2, if not already there?  That is,
should the new dates on F2 be (D2 union {Y}) or (D1 union D2 union {Y}),
or something else?

Could the official version of the guidelines from Prof. Moglen on what can
go in the testsuite be put on the GCC web pages (rather than just the
summary at http://gcc.gnu.org/ml/gcc/2000-01/msg00593.html )?

-- 
Joseph S. Myers
jsm28@cam.ac.uk

Re: Automated testing framework (was: V3: SPARC bug and tree freeze)

[Richard Stallman]

    Actually, since you refuse to publish this kind of stuff, I think it
    is quite reasonable to send this stuff to you.  You might then realize
    that it does take a significant amount of time and that most of the
    work can be eliminated by simply publishing the forms and instructions.

I know all about this.  As the maintainer of Emacs at some times, and
of GCC at other times, I had to take care of legal papers like every
other maintainer.  In most cases it was quick and easy, once I had
read the contribution in question (which I had to do anyway).

The hard cases were (1) in peculiar situations where it wasn't obvious
what to do, and (2) when someone had trouble getting disclaimers
signed or wanted to change the wording.  Nothing can make those hard
cases easy.  However, you can (and should) ask for help with #1, and
with the new system, Brian Youmans will often take care of #2.

Every other GNU maintainer of an FSF-copyrighted package does this
job.  Please try doing it, and you will find it not so hard.

I say that "you", the GCC maintainers should do it.  But that doesn't
mean Jeff Law in particular has to do it.

    If people were making mistakes with the previous online instructions or
    forms, then that means those instructions need to be clarified.

That is easy to say, but no matter how much work we put into trying to
make them clear, people will continue to misunderstand them.  Many of
our contributors do not even read English very well.  Making a
foolproof description that people will essentially never misunderstand
is a very hard job, and it is a foolish risk.

Fortunately, the complexity of these rules is small compared with what
you have to know to maintain GCC, and a few smart people who decide to
learn these rules, ask questions as needed to assure they understand
right, and then carry them out, will not find it tiring.

Re: Automated testing framework (was: V3: SPARC bug and tree freeze)

[Richard Stallman]

    There still needs to be a system for all 89 GCC maintainers (the count of
    distinct people in MAINTAINERS, including those marked as being people who
    ought to have accounts for CVS write access but don't yet) to have the
    forms for sending to contributors, better than everyone individually
    getting them from assign@gnu.org.

Properly speaking, the maintainers of GCC are the people in the
steering committee--they are the ones who have accepted the post of
GCC maintenance.  Other people participate, but these people have the
responsibility to make sure it is done right.  The file MAINTAINERS,
despite its name, doesn't define who the maintainers are.

The 89 people listed there probably do various different kinds of
work.  Some of them look at changes written by other contributors and
install them; those people need to check for papers, and normally
ought to know what to say to the other contributors to tell them how
to sign papers.  I hope the number of these people is much smaller
than 89, though.  It would be unreliable to have 89 people doing this
job--someone is sure to misunderstand.

As for the people who write and install their own changes, but don't
install other people's changes, we need papers *from them* but they
don't ever have to direct other people how to deal with the papers.

I hope this results in a number of people who tell others what papers
to sign which is large enough that there isn't too much work for each,
but small enough that teaching all of them is feasible and reliable.
That way the job can get done easily and reliably.

    Does assign.future also cover changes to the GCC manual?

I asked our lawyer about this a month ago, and that is when I wrote
the text in maintain.texi about the subject.  In the case of the GCC
manual, since it is published separately as a book, we should get
separate papers for the manual.

It would have been a good idea for me to have asked about this in the
past.  I wish I had asked him earlier.

    What exactly is the position on whether an assign.future for "GNU CC" or
    "GCC" (assigning "... Developer's copyright in changes and/or enhancements
    to the program <name of program> ...") covers all programs, libraries,
    manuals and other files in the GNU Compiler Collection (GCC)?

Yes, but the test suite is not part of GCC.  (If that is not entirely
clear in the distribution, the web site, etc, we need to make it
clearer!)

      The GCC
    practice is that the first change made to any file in a year requires the
    copyright dates to be updated (whether or not there is a release finished
    that year).  Is that correct?

It is reasonable.  I myself would not update the year if there has
been only a trivial change since the beginning of the year, but
it does no harm to do so.

    If file F1 has copyright dates D1, and file F2 has copyright dates D2,
    and, in year Y, I copy or move code from F1 to F2, should the copyright
    dates on F1 be added to those on F2, if not already there?

Yes, it should be.

    Could the official version of the guidelines from Prof. Moglen on what can
    go in the testsuite be put on the GCC web pages (rather than just the
    summary at http://gcc.gnu.org/ml/gcc/2000-01/msg00593.html )?

I don't know; it depends what is in that message.  (I am not sure that
the term "official version" really describes it properly; I would have
to see it again to judge that.)

Copyright forms (was: Automated testing framework)

[Gerald Pfeifer]

On Thu, 16 Nov 2000, Richard Stallman wrote:
>> [gcc/MAINTAINERS]
> The 89 people listed there probably do various different kinds of
> work.  Some of them look at changes written by other contributors and
> install them; those people need to check for papers, and normally
> ought to know what to say to the other contributors to tell them how
> to sign papers.  I hope the number of these people is much smaller
> than 89, though.

It is 46.

These are volunteers who maintain front-ends, ports, i18n, documentation,
web pages... who may install changes of their own and approve changes of
others in those areas they maintain.

Not all of them install/approve other people's changes regularily, and
often these other contributors are well-known, such as maintainers of
other parts of the compiler.

>> Does assign.future also cover changes to the GCC manual?
> I asked our lawyer about this a month ago, and that is when I wrote
> the text in maintain.texi about the subject.  In the case of the GCC
> manual, since it is published separately as a book, we should get
> separate papers for the manual.

Ouch. Does that mean that before the next edition of the book someone
(FSF staff?) will have to contact all who have contributed to the manual?

If so, this will be hardly possible, as -- according to the GNU Coding
Standards -- ChangeLogs for documentation are not required and we, based
on a suggestion by myself, switched to requiring such ChangeLogs only two
years ago.

Have you considered the possibility to provide a "unified"  copyright
assignment that applies to both GCC and the book that we could use in
the future?

Gerald

Re: Copyright forms (was: Automated testing framework)

[Richard Stallman]

    It is 46.

    These are volunteers who maintain front-ends, ports, i18n, documentation,
    web pages... who may install changes of their own and approve changes of
    others in those areas they maintain.

46 is a lot of people to advise contributors on signing papers.  And it
sounds like most of them will do this only very rarely--maybe
a handful of times a year.

Perhaps it is best for most of them simply to ask someone else what to
do on such occasions.  I would suggest choosing approximately 8 people
who do this often as the ones who will do all of this work, and
inviting the others who rarely have occasion to forward it to those 8.
Of course, 8 is just a suggestion--a smaller number would be ok if
that's what the maintainers prefer, and a somewhat larger number would
be ok too.

    Ouch. Does that mean that before the next edition of the book someone
    will have to contact all who have contributed to the manual?

It would be a good idea to do this, yes.  Isn't it safe to assume,
though, that only contributors to GCC's code have contributed to the
manual?  That would make a finite set of people to ask, and they could
be asked easily enough by sending the same message text to each of
them.

    Have you considered the possibility to provide a "unified"  copyright
    assignment that applies to both GCC and the book that we could use in
    the future?

That is a good idea; I will get something like that drawn up.

Re: Automated testing framework (was: V3: SPARC bug and tree freeze)

[Mike Stump]

> Date: Tue, 14 Nov 2000 00:22:02 -0700 (MST)
> From: Richard Stallman <rms@gnu.org>
> To: aoliva@redhat.com

> Posting the criteria proved to be a bad idea, because a fair number of
> people don't fully understand them at first reading.

Posting the source to gcc is bad because a fair number of people don't
fully understand it at first reading.  :-( I do not understand your
point.  If people are not understanding what they read, that clearly
shows a lack of proper documentation.  Eliminating the documentation
while it helps them to not misunderstand it, doesn't help them
understand it better.

> To: rms@gnu.org
> From: law@redhat.com
> Date: Tue, 14 Nov 2000 09:29:17 -0700

> If people were making mistakes with the previous online instructions
> or forms, then that means those instructions need to be clarified.

I agree.

> Date: Tue, 14 Nov 2000 19:11:55 +0000 (GMT)
> From: "Joseph S. Myers" <jsm28@cam.ac.uk>
> To: Richard Stallman <rms@gnu.org>
> cc: <aoliva@redhat.com>, <guerby@acm.org>, <gcc@gcc.gnu.org>

> There still needs to be a system for all 89 GCC maintainers (the count of
> distinct people in MAINTAINERS, including those marked as being people who
> ought to have accounts for CVS write access but don't yet) to have the
> forms for sending to contributors, better than everyone individually
> getting them from assign@gnu.org.

I think it is easier for us to have a comprehensive web site that
includes all the subtle details that we learn about over time, that
can be reviewed by the FSF, to ensure that our understanding is
correct, and to have a place for updates to appear so that all of the
gcc maintainers can see the changes, instead of the one on one
conversations with various parties in the know.  I think this also has
the added benefit of getting people to actually file a disclaimer or
assignment.  If people have to ask for directions from someone, and if
that someone doesn't respond directly and in a timely fashion, we can
loose out on having another contributor.  The egcs project tried to
address the idea of loosing out on contributors, and I think they were
successful.  One aspect of this that was addressed was the inclusion
of the forms on the web site.  If we include example scenarios of
correct applications, and examples of what not to do, we can increase
the usefulness and accuracy of the web site, and hopefully the
accuracy of how people use the various forms.

Asking all the maintainers to be versed in all the detail is I think
asking too much.  Better to have it written down.  The fact that other
folks from other projects were getting advice from the gcc web site in
these matters shows that there is an existing need for such a web
site.

I think if we are to be tasked with solving the problem, then the
exact form of the solution to the problem ought to rest squarely on
our shoulders.  If we can improve the documentation so that it doesn't
confuse people from other projects, then I think we should.  It we can
improve it to be more understandable, then we should.  If we can alert
people to the 30 most common problems in the application of the
advice, well, even I would be educated by it.

I would think that a written system is better from a management
perspective, as then management can know what advice is being given
out and have a chance to correct it, update it and review it.  Also,
if people give out advice that is contrary to the documentation, it
offers the ability to have that corrected in a more timely fashion.
From a point of view of someone _giving advice_, I think it would be
better, as we can know that the advice we are giving is in fact the
best advice we can give.

Now, if we want to hide it down on a maintainers part of the web site
instead of having it on page 1 of the web site, that would be a
reasonable compromise I think.  This way, random folks are less likely
to stumble across is, yet people in the know can still find it.  Would
this compromise solution be acceptable to you?