Re: [maint] [maint] Michael Chastain for testsuite

Re: [maint] [maint] Michael Chastain for testsuite

[Michael Elizabeth Chastain]

Peter Barada writes:
> In fact, I'd like to also require for each testcase information in the
> testcase about what PR it is submitted for so if a regression occurs
> an automates tester can point it out as a way to stat to figure out
> *why* it failed.

If there is a PR for a test, then the test can connect to it with
the "kfail" number.

Michael C

Re: [maint] [maint] Michael Chastain for testsuite

[Michael Elizabeth Chastain]

cgf> Does that translate into every bug reported eventually gets a test?

Hmmm, to put that in process terms: it means that a bug can't be closed
until there is a test for that bug in the test suite.

That would be good, but I'm not ready to go as far as to make it
a requirement for closing a PR.  We've already got several problems
with PR's: the number of open PR's increases with time; and most PR's
go for a long time before getting any attention.  I like to be a little
conservative about closing PR's but I'm not willing to add a requirement
that a PR has a test.

cgf> Would it make sense to add the test for a reported bug before a patch to
cgf> fix it is submitted?

I don't think so.  It might be sensible to have these requirements
in parallel:

  test must be committed before closing PR
  code fix must be committed before closing PR

But I don't think it helps to require "test must be committed before
code fix is committed".

Michael C

Re: [maint] [maint] Michael Chastain for testsuite

[Michael Elizabeth Chastain]

Chris Faylor writes:
cgf> If we have someone as a maintainer then it seems like their area of
cgf> maintainership should reflect their vision, i.e.  you can't just give
cgf> carte blanche to a group people to do whatever they want in that arena.

I think it might be good for me to state my vision.  This is all
stuff that y'all know about me already.

(1) Copyright -- is the most important thing to me.  I want every file
    in testsuite/ to bear a valid FSF copyright notice.

(2) Testing submissions -- I mean to require that every patch says
    how it was tested (or not tested).  I don't think it's useful
    to formalize or standardize how a patch is tested beyond that.

cgf> I know that this goes without saying but I thought I'd say it anyway.
cgf> :-)

Me too.

Michael C

Re: [maint] [maint] Michael Chastain for testsuite

[Christopher Faylor]

On Fri, Jun 04, 2004 at 07:37:26PM -0400, Michael Elizabeth Chastain wrote:
>Chris Faylor writes:
>cgf> If we have someone as a maintainer then it seems like their area of
>cgf> maintainership should reflect their vision, i.e.  you can't just give
>cgf> carte blanche to a group people to do whatever they want in that arena.
>
>I think it might be good for me to state my vision.  This is all
>stuff that y'all know about me already.
>
>(1) Copyright -- is the most important thing to me.  I want every file
>    in testsuite/ to bear a valid FSF copyright notice.
>
>(2) Testing submissions -- I mean to require that every patch says
>    how it was tested (or not tested).  I don't think it's useful
>    to formalize or standardize how a patch is tested beyond that.

Does that translate into every bug reported eventually gets a test?
Would it make sense to add the test for a reported bug before a patch to
fix it is submitted?

I don't know who would be tasked with doing something like that but it
would sort of make sure that no bug was lost as the test suite FAILs
steadily rise...

cgf

Re: [maint] [maint] Michael Chastain for testsuite

[Peter Barada]

>>(2) Testing submissions -- I mean to require that every patch says
>>    how it was tested (or not tested).  I don't think it's useful
>>    to formalize or standardize how a patch is tested beyond that.
>
>Does that translate into every bug reported eventually gets a test?
>Would it make sense to add the test for a reported bug before a patch to
>fix it is submitted?

I'd like to see a testcase being *required* to be added that shows a
current failure *before* a patch for its fix is accepted.  This would
expand the regession capability quite immensely.  Then we'd have a
much more palatable problem of having too many testcases that stress
various areas of the code, a problem that I'd much rather see :)

Of course testcases may overlap so a failure for one problem may be
fixed by a different patch, but the submitted testcase adds to the
arsenal we could use for regressions, and if we're smart enough, we
could link the testcases together.

In fact, I'd like to also require for each testcase information in the
testcase about what PR it is submitted for so if a regression occurs
an automates tester can point it out as a way to stat to figure out
*why* it failed.

-- 
Peter Barada
peter@the-baradas.com

Re: [maint] [maint] Michael Chastain for testsuite

[Eli Zaretskii]

> From: Peter Barada <peter@the-baradas.com>
> Date: Fri,  4 Jun 2004 20:17:06 -0400 (EDT)
> 
> I'd like to see a testcase being *required* to be added that shows a
> current failure *before* a patch for its fix is accepted.

That's a noble goal, but what do we do in cases where it's
impractical?  For example, a particular bug could only be raising its
ugly head in a very large program.

Re: [maint] [maint] Michael Chastain for testsuite

[Bob Rossi]

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

On Sat, Jun 05, 2004 at 01:02:24PM +0200, Eli Zaretskii wrote:
> > From: Peter Barada <peter@the-baradas.com>
> > Date: Fri,  4 Jun 2004 20:17:06 -0400 (EDT)
> > 
> > I'd like to see a testcase being *required* to be added that shows a
> > current failure *before* a patch for its fix is accepted.
> 
> That's a noble goal, but what do we do in cases where it's
> impractical?  For example, a particular bug could only be raising its
> ugly head in a very large program.

Well, it's a tough decision. Obviously it's impractical to run many
large programs to prove the behavior of GDB is correct. However, once
the bug fix is committed with out a testcase, you can consider it broken
already. What can break, will.

BTW, over the years, I have had a lot of experience with finding bugs in
large programs. It can take hours to find the bug, however, once it is
found, I typically find that it can be reproduced with a very small
segment of the original code. I doubt you would need to run a test on a
large program in almost all cases, you will probably have to create a
subset of the original code, and use that as the testcase.

Bob Rossi

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

[maint] [maint] Michael Chastain for testsuite

[Andrew Cagney]

Hello,

Michael Chastain has been nominated for the role of testsuite 
maintainer, taking over responsibility for all the open areas of the 
testsuite (config, lib, gdb.base etc.).  In addition, it has been 
proposed that the `global maintainers' serve as backup to Michael in 
these areas (Michael has already indicated that he is ok with this).

Thoughts?  I'll table this for a week.

Andrew

Re: [maint] [maint] Michael Chastain for testsuite

[Christopher Faylor]

On Fri, Jun 04, 2004 at 03:52:04PM -0400, Andrew Cagney wrote:
>Michael Chastain has been nominated for the role of testsuite
>maintainer, taking over responsibility for all the open areas of the
>testsuite (config, lib, gdb.base etc.).  In addition, it has been
>proposed that the `global maintainers' serve as backup to Michael in
>these areas (Michael has already indicated that he is ok with this).
>
>Thoughts?  I'll table this for a week.

FWIW, I know that Michael will be active in improving the test suite
and that is a *good thing* (tm).  I think this is a great idea.

Having the global maintainers serve as backup seems like an ok idea as
long as Michael agrees to exercise some oversight.  If we have someone
as a maintainer then it seems like their area of maintainership should
reflect their vision, i.e.  you can't just give carte blanche to a group
people to do whatever they want in that arena.  So, the global
maintainers would, I hope, show some restraint in making changes to the
test suite and always solicit Michael's opinion, deferring to him when
he has objections, or giving him adequate time to voice objections
before making changes.

I know that this goes without saying but I thought I'd say it anyway.
:-)

cgf